Puppet fails to complete run, tries to complete multi-user target by itself

[jorhett]

Overview

In GCP some node types don't come with cloud-init so the only thing you can do to automate Puppet startup is to use a startup script https://cloud.google.com/compute/docs/instances/startup-scripts/linux. This startup script runs prior to the full system initialization, thus multi-user target has not been reached.

In our startup script we install the Puppet repo, the puppet-agent package, and then connect to our puppet server for the remaining setup. We're finding that Puppet is hanging for an hour during the initial puppet run, apparently attempting to manage multi-user.target (likely from the changes by @jcpunk in #270)

Jan 20 18:56:01 example puppet-agent[3957]: (/Stage[main]/Stdlib::Manage/User[root]/password) changed [redacted] to [redacted]
Jan 20 18:56:01 example puppet-agent[3957]: (/Stage[main]/Stdlib::Manage/Sshkey[github.com@ssh-ed25519]/ensure) created
Jan 20 18:56:01 example puppet-agent[3957]: (/Stage[main]/Stdlib::Manage/Sshkey[github.com@ecdsa-sha2-nistp256]/ensure) created
Jan 20 18:56:01 example puppet-agent[3957]: (/Stage[main]/Stdlib::Manage/Sshkey[github.com@ssh-rsa]/ensure) created

...30 minutes of silence from puppet...

Jan 20 19:26:02 example puppet-agent[87837]: Run of Puppet configuration client already in progress; skipping  (/opt/puppetlabs/puppet/cache/state/agent_catalog_run.lock exists)

...almost 30 more minutes of silence from puppet...

Jan 20 19:51:16 example puppet-agent[3957]: (/Stage[main]/Systemd/Service[multi-user.target]/ensure) change from 'stopped' to 'running' failed: execution expired
Jan 20 19:51:16 example puppet-agent[3957]: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Jan 20 19:51:17 example puppet-agent[3957]: Execution of Puppet configuration client did not complete within 3600 seconds and was terminated.

Now I can guess where you are going, and yes we do set systemd::default_target: multi-user.target and yes I'll remove that. My question is more about whether Puppet should be in charge of making the system move to this target? It doesn't seem likely to be something that Puppet can or should complete on its own. I would think that Puppet should define the target, but not be in charge of trying to take the entire system to that target 😕

In particular, I think that this line is the problem https://github.com/voxpupuli/puppet-systemd/blob/master/manifests/init.pp#L332 and we should consider removing it? I find it hard to imagine a negative impact from the removal, as systemd will ensure that the node reaches the default target will it not? Even if we do need to prompt systemd, I don't think using the service mechanism to try and start this state within the puppet convergence is the appropriate action

[jorhett]

Note that specifically Puppet is running this command, which I believe to be somewhat naive way to achieve the target state:

   2300    3957    2273    2273 ?     -1 Sl       0   0:12 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
   3957   87493   87493   87493 ?     -1 Ss       0   0:00 /usr/bin/systemctl start -- multi-user.target