WIP: Action series

[turtle0x1]

This isn't yet practical (but it does work, you have to create the structure through manual SQL statements right now let me know if you want an example) & when it is dont think of this as a tool to setup hosts - its more of an update and re-configure tool for hosts you don't control with cloud-init or images

Action series - Tree's are magical!

(Instances = the agnostic term for containers & vms not the host themselves)

Ever wanted to run a bunch of commands (bash or what ever? - this doesn't care as long as your instance will process it) on lots off different LXD instances in a structured manor ? Ever wanted to control exactly what happens after each command and take a different path (on every instance) ? Well this is that (plus it records the history against the instance - so if you dont re-name the instance gives audit history as-well)

What you do is create "Action Series" which is a chain off commands, you can branch depending on the return result of the command

(Red represents the path it will take if an error occurs - this is the overview of the "series")

Capture the result of each command at each step on each instance on every host, means you can see how every host processed the chain I.E

(A solid black line indicates a path that wasn't taken because the step didn't reach the required status)

Because this is tree's it will support inf deep chains (what if you error handling fails ? Clearly you call another error handling function! And if that fails ? You N+1 that bad boi) depending on the return code you can do what ever you want!

I dont think at this point I will support multi parent nodes as this creates horrible graphs - I may create some kind of labelled error handler system to reduce duplication of commands

This also records this stdout and stderr log locations but doesn't yet give access to them, but it will - so you can inspect every host every step of the way!

Right now this is blocking in the browser - It waits on each step on each instance to complete & offer's 0 visual feedback as to where the process is (so not very practical), we need to be able to run background jobs which push to the node server which updates the web interface for the user (you could theoretically - right now - run long jobs (apt install big-package) open another tab and keep refreshing the run to see its current progress)

And we should include a "patches" export so if you need help someone debug something but dont want them to wait for to run a bunch of commands you can send them a "patch" (an action series) they can import and run

[bmullan]

@turtle0x1
I think this is a good concept but may take a while to implement & document well.

Couple ideas (maybe you've done something like these already though):

1. I for each "run" of an update I would suggest perhaps not just adding to one big central log but give each "run" some kind of ID number and then create a Numbered List of all Nodes that succeeded in processing all of the update and another Numbered List of all Nodes that failed to process all of the update and where they may have failed.
   This make it easier to immediately address failures without doing searches thru a Large central log file.
2. For updates that require a change to a text-type config file I would take a snapshot/backup of the original file so it can be restored in the case of a successful update that may upon retroflection may not be what you really wanted or need.

I'll add more comments as I learn more about what you are doing.

However, what tool(s), programming approach are you implementing this with? JSON, REST, ansible, etc?

One tool I looked at in the past that I thought "could" be a great fit for this is Rundeck:

https://www.rundeck.com/new-to-rundeck

Rundeck is extensible by other tools like Ansible, Jenkins, PagerDuty:

https://www.rundeck.com/key-integrations

Rundeck also is designed to support different "User Role Based Access Controls" that dictate what updates a particular "user" is permitted to execute/perform.

There is also a good collection of already existing Rundeck "plugins":

https://resources.rundeck.com/plugins/

Finally, regarding Rundeck here is a Blog that a Rundeck user wrote to help new Rundeck developers get started:

https://tech.davidfield.co.uk/rundeck-3-install-setup-and-an-example-project/

The Blog does focus on RPM based installation of Rundeck instead of APT-GET but I think most of it is probably generic in nature.

Another software tool I have used and liked is "Fabric" which is a Python based tool that was fairly simple to learn/use also

**http://www.fabfile.org/**

One GUI for Ansible (ansible is now owned by Redhat) is Ansible Tower. However, the Upstream Open Source GUI is called "Ansible AWX":

https://computingforgeeks.com/how-to-install-ansible-awx-on-ubuntu-debian/

https://github.com/ansible/awx - AWX provides a web-based user interface, REST API, and task engine built on top of Ansible.

Both Rundeck & Fabric use SSH for fundamental access to nodes.

But as I don't program any more today I'm not a good judge :-)

Brian

[turtle0x1]

Thanks for the feeback!

Ill be sure to add the lists of instances that failed onto the action series overview!

for each "run" of an update I would suggest perhaps not just adding to one big central log

Each command run on each instance (container / VM) is recorded individually in its own log files (stdout and stderr output)

The second picture in my original post shows the result of each instance (not host) for a particular "run"

For updates that require a change to a text-type config file I would take a snapshot/backup of the original file so it can be restored

So your "execution plan" (which is the series of commands to run on each host) might look something like this (read left to right)

Right now this just using PHP & "pure LXD" and Id like to try to keep it that way (as soon as you start using SSH / another protocol you move into a different class of products its no longer compatible with just LXD),

I do need to look at software to run background jobs but this is probably somewhat of a "replacement" for Rundeck in the sense you program scripts to be called - this could be used to control ansible scripts (or playbooks or whatever) running and perform custom actions based on the results of each action