Add info for systemd based distros

[stratakis]

Provide examples for managing the buildbot-worker
service through systemd unit files for systemd based
distributions.

[Mariatta]

Note I have just removed the Netlify integration, since we're using readthedocs preview build now.

[tiran]

Wants=network-online.target
After=network-online.target local-fs.target

[stratakis]

Will amend that

[tiran]

ausearch -m AVC

[stratakis]

I usually use sealert -a /var/log/audit/audit.log which provides messages with the warning and how to fix them, but I thought it could be too specific for the devguide.

[tiran]

Is this really going to be an issue? Unless you define a policy the buildbot daemon should run in unconfined_t and not be affected by SELinux policy restrictions.

[stratakis]

It was always affected on all the buildbots I've set up (Fedora and RHEL). Basically selinux prevents systemd's open and read access for the twistd.pid file.

[stratakis]

So after following the instructions on the buildbot worker page to set it up and creating the systemd unit files, when trying to start the service, initially this error message is show (date/time is trimmed out):

`
Following twistd.log until startup finished..
AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

AVC avc: denied { read } for pid=1 comm="systemd" name="twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012
If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012

localhost setroubleshoot[1489084]: SELinux is preventing systemd from read access on the file twistd.pid. For complete SELinux messages run: sealert -l 57d3c874-be75-46f6-82c9-3680ac54002c

localhost python3[1489084]: SELinux is preventing systemd from read access on the file twistd.pid.#12#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed read access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012
`
After adjusting the policy, another error comes up:

`
Following twistd.log until startup finished..

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost systemd[1]: buildbot-worker.service: Can't convert PID files /home/buildbot/buildarea/twistd.pid O_PATH file descriptor to proper file descriptor: Permission denied

localhost audit[1]: AVC avc: denied { open } for pid=1 comm="systemd" path="/home/buildbot/buildarea/twistd.pid" dev="dm-0" ino=25387713 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

localhost setroubleshoot[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid. For complete SELinux messages run: sealert -l 3d54c639-fea4-4a18-be47-c5fe1d57a02a

localhost python3[8714]: SELinux is preventing systemd from open access on the file /home/buildbot/buildarea/twistd.pid.#012#012***** Plugin catchall (100. confidence) suggests **************************#12#012If you believe that systemd should be allowed open access on the twistd.pid file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#12# ausearch -c 'systemd' --raw | audit2allow -M my-systemd#012# semodule -X 300 -i my-systemd.pp#012`

After allowing that as well, the service starts normally.

[tiran]

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem

• move buildbot out of /home. For example deploy buildbot code to /opt/buildbot and use systemd RuntimeDirectory, StateDirectory, and LogsDirectory (/run, /var/log, ... see man systemd.exec)
• run buildbot as systemd user service. You have to put the service file to /home/buildbot/.config/systemd/user/buildbot.service, enable lingering with loginctl enable-linger buildbot, and start the service from a buildbot login shell (not su/sudo!) as systemctl --user enable --now buildbot.service.
• create a custom SELinux policy, types, and file contexts for buildbot.
• make the init_t SELinux type permissive semanage permissive -a init_t

[stratakis]

I see. SELinux and systemd both treat /home as a privileged and restricted area. System services are generally not allowed to access /home tos prevents daemons from stealing ssh private keys, personal emails, or your cat pictures.

I know of four options to deal with this problem

* move buildbot out of `/home`. For example deploy buildbot code to `/opt/buildbot` and use systemd `RuntimeDirectory`, `StateDirectory`, and `LogsDirectory` (/run, /var/log, ... see man systemd.exec)

That would go against the current guidelines of setting up a buildbot worker (or requiring a bigger overhaul) so I wouldn't go with that option.

* run buildbot as systemd user service. You have to put the service file to `/home/buildbot/.config/systemd/user/buildbot.service`, enable lingering with `loginctl enable-linger buildbot`, and start the service from a buildbot login shell (not su/sudo!) as `systemctl --user enable --now buildbot.service`.

This solution I liked the most, however the systemctl --user option is not available in RHEL7.

* create a custom SELinux policy, types, and file contexts for buildbot.

Maybe that would be the best way then. Any pointers on how to work with that?

* make the init_t SELinux type permissive `semanage permissive -a init_t`

Not sure I would like to change init_t

[ezio-melotti]

If this PR is still relevant, it should be updated after the devguide reorganization, reviewed, and merged.

[stratakis]

If this PR is still relevant, it should be updated after the devguide reorganization, reviewed, and merged.

Thanks for the reminder. Is there an ETA for the devguide reorganization?

[ezio-melotti]

The devguide reorganization is done already, and there is now a conflict on the PR that must be resolved.

[ezio-melotti]

@stratakis can you merge main into your branch and resolve the conflict?

[stratakis]

Rebased.

[ezio-melotti]

Thanks!
@methane / @zware, can you (re-)review?

[zware]

This dropped off my radar and I forgot about it for two years :(. I'm good with merging this as is after conflicts are fixed, and we can iterate on it after merge if needed.

[stratakis]

I'll take another look as well, I've completely forgot about it as well.