orderly [<SUPERVISOR-FLAGS>] [ -- <PROCESS-FLAGS> ]+orderly Provides ordered starting, supervision and stopping of a collection of processes. orderly starts a list of processes in order, then monitors them with provided health check scripts.
On failure, processes are stopped and restarted in a well specified order. Scripts can also be provided to cleanup after a failed process (unmounting filesystems, deleting files, etc.).
Arrangement of orderly invocations in a tree hierarchy allows the creation of fault tolerant process supervision trees.
The size of the start pool, when this pool less than one and a (re)start is required orderly aborts. Each (re)start decreases the pool size by one. Note that the initial start is counted towards the quota, so the pool size must be at least 1 for a successful startup.
The rate at which restarts are added into the (re)start pool.
An alias for the rate at which restarts are added into the (re)start pool.
An alias for the rate at which restarts are added into the (re)start pool.
If specified, a file to be written containing the current status of orderly. The file will contain either "STARTING", "RUNNING". orderly transitions from starting, to running after all procesess it is controlling have started successfully at least one time. The main use for this file is for creating nested orderly supervision trees that start in order.
An optional hook to run when the first startup completes successfully,
An optional hook to run before each restart that is triggered by a command failure.
An optional hook to run when orderly encounters an unrecoverable error, and must abort operation.
An optional hook to run just before orderly exits after a clean shutdown.
Shorthand for setting all lifecycle hooks to the same script, in this case env variables can disambiguate the action to take.
The number of seconds to wait for a given hook before giving up and triggering a restart. A negative value means no timeout.
The amount of time in seconds to wait between health check loops.
If set, health checks that pass do not cause any logging output.
The name of the service, passed to all callbacks under the env variable as 'ORDERLY_SERVICE_NAME'.
The command invoked by orderly to launch a supervised process. If this program exits, it will trigger a restart.
An optional command invoked concurrently with the service, it should exit with a 0 exit code when this process is ready and the next process can be started.
An optional command invoked periodically as a health check. If this commands times out or returns an unsuccessful exit code, a restart will be triggered. This check is in addition to ensuring the run process has not exited.
An optional command to shutdown the supervised process. If not specified, orderly will send SIGTERM to terminate the supervised process.
The shutdown command may be run if a command start times out, a sibling process dies and the server needs to restart, or orderly is shutting down.
An optional command to cleanup any resources the running process may have left. If it exits with an unsuccessful exit code, a restart will be triggered. Process cleaning should be idempotent, and always happens in reverse order to process startup.
Shorthand for setting all commands to the same script, in this case env variables can disambiguate the action to take.
The number of seconds to wait for a given command before giving up. A negative value means no timeout.
The amount of time to wait after a shutdown command before terminating the child with a SIGKILL if it does not exit on it's own.
The following env variables are passed to any specified process scripts.
The name of the process being managed.
One of START_COMPLETE, RESTART, FAILURE, RUN, WAIT_STARTED, CHECK, SHUTDOWN, CLEANUP depending on which action orderly is requesting.
The pid of the orderly process.
The pid of the supervised process, if it is running.
orderly shuts all processes down with the provided or default shutdown commands in reverse order.
orderly exits with a zero exit code only if shutdown after a SIGINT or SIGTERM occured with no errors.
Given the executable service script 'sv':
#! /usr/bin/env bash
set -eu
p () {
echo "$ORDERLY_SERVICE_NAME $ORDERLY_ACTION"
}
case $ORDERLY_ACTION in
RUN)
p
exec sleep 9999
;;
WAIT_STARTED)
sleep 0.1
p
;;
CHECK)
p
;;
SHUTDOWN)
p
kill -9 $ORDERLY_RUN_PID
;;
CLEANUP)
p
;;
*)
echo "unknown action: $ORDERLY_ACTION"
exit 1
;;
esacAnd the invocation:
orderly -- \
-name sv1 -all-commands ./sv \
-- \
-name sv2 -all-commands ./sv \
-- \
-name sv3 -all-commands ./sv &
pid="$!"
sleep 1
kill -SIGINT "$pid"
waitYou will see output like:
2019-03-28 12:23:10 INFO [orderly] (re)starting all procs.
2019-03-28 12:23:10 INFO [orderly] running sv3 cleanup.
sv3 CLEANUP
2019-03-28 12:23:10 INFO [orderly] running sv2 cleanup.
sv2 CLEANUP
2019-03-28 12:23:10 INFO [orderly] running sv1 cleanup.
sv1 CLEANUP
2019-03-28 12:23:10 INFO [orderly] starting sv1.
sv1 RUN
sv1 WAIT_STARTED
2019-03-28 12:23:10 INFO [orderly] starting sv2.
sv2 RUN
sv2 WAIT_STARTED
2019-03-28 12:23:10 INFO [orderly] starting sv3.
sv3 RUN
sv3 WAIT_STARTED
2019-03-28 12:23:10 INFO [orderly] checking sv1.
sv1 CHECK
2019-03-28 12:23:10 INFO [orderly] checking sv2.
sv2 CHECK
2019-03-28 12:23:10 INFO [orderly] checking sv3.
sv3 CHECK
2019-03-28 12:23:11 INFO [orderly] supervisor shutting down gracefully.
2019-03-28 12:23:11 INFO [orderly] shutting down sv3.
sv3 SHUTDOWN
2019-03-28 12:23:11 INFO [orderly] running sv3 cleanup.
sv3 CLEANUP
2019-03-28 12:23:11 INFO [orderly] shutting down sv2.
sv2 SHUTDOWN
2019-03-28 12:23:11 INFO [orderly] running sv2 cleanup.
sv2 CLEANUP
2019-03-28 12:23:11 INFO [orderly] shutting down sv1.
sv1 SHUTDOWN
2019-03-28 12:23:11 INFO [orderly] running sv1 cleanup.
sv1 CLEANUPLogging facilities may be added in the future, though currently a logging process can simply be part of the process list, and can be sent input via named pipes or any other mechanism.
orderly is Copyright (C) 2019 Andrew Chambers https://acha.ninja/