Handle AMIs with buildbot and its dependencies pre-installed.

[dinatale2]

We should be able to support AMIs that have all dependencies, including buildbot, pre installed. This means that dependency installation should be skipped and only buildbot is started at boot strapping.

We already support disabling dependency installation, this issue should handle the changes to the script provided as user data for cloud init.

[morrone]

Why do we want to stick different workflows into the same build sequence, just to have to then special case out the parts that don't apply to that sequence? Perhaps it will all be simpler to set up and maintain if we just have different build steps for the different major builder types? For instance, all of the static AMI builders that just build and package lustre would have one share set of build steps. The buildbot builders that do testing would have a separate set of build steps. Likewise future builders that dynamically install all dependencies could have their own build step sequence as well.

[dinatale2]

This issue has nothing to do with the build steps a build slave follows. Currently, bb-bootstrap.sh is installing all dependencies necessary to install buildbot during cloudinit when an instance is spun up. I recently created static AMIs which have all the lustre build dependencies pre-installed, but they don't have buildbot pre-installed. If buildbot is pre-installed, then the bootstrap process simply needs to set some variables and start buildbot. The cloudinit script is what this issue seeks to improve.

[morrone]

Ah, right. So you are going to make a new cloudinit script that is specific to the static AMI slaves, right? No need for bb-bootstrap.sh for those slaves.

[dinatale2]

I'm actually leaning towards a check in bb-bootstrap.sh to determine if buildbot is already installed so buildbot dependency & buildbot installation can be skipped. This will keep all bootstrapping logic consolidated and it should be a really straight forward change.

[morrone]

Why is consolidating the configuration in a home-grown system better than consolidating the configuration in the buildbot configuration file as buildbot intends? I'm concerned that the static AMI builder configuration path already contains a large number of local inventions and extraneous moving parts that make this overly complicated and difficult to understand. Keep in mind that the configuration complexity is just going to continue growing.

We are going to keep adding new configurations and demands on this system, and if we always take the route of consolidation and keep trying to cram more and more different models into the same code path, the code will continue grow in complexity. If we just list each part of the configuration in the buildbot configuration style individually, yes the configuration file will be larger in size, but it will remain simple and understandable.

[dinatale2]

The bb-bootstrap.sh script's entire role is to get buildbot running on a slave. It is not part of any build step and is downloaded and called using runurl during cloudinit. The script doesn't impact builder configuration and has no impact on supporting other future configurations. The addition I am planning to make to bb-bootstrap.sh is an if statement that says, if buildbot is not installed, then install buildbot. So, I'm a bit confused about your above comment.