Lunchpail pipelines

[starpit]

it occurred to me that maybe this whole dispatcher vs worker thing can be orthogonalized? dispatcher vs worker? isn’t this just a shittier way of expressing pipelines?

versus

dispatcher | worker

i.e. just two separate applications that you can combine more flexibly

and: in a pipeline, the second stage doesn’t need its own inbox… its inbox is the outbox of the prior stage. it can be told this on stdin from the prior stage

and: if you want to dispatch without consuming, …. why don’t we have the semantics be that the workstealer and minio only self-destruct… not when all tasks have been assigned… and not after all tasks have been completed… but after all outbox tasks have been consumed.

so if i just say dispatcher… it’ll stay up because there are no consumers

if you want just to test the dispatch, then dispatcher > /dev/null and lunchpail can detect that stdout is a file descriptor, and auto-drain the outbox to that fd

and: if you want to attach a second worker to a dispatcher… then maybe something like dispatcher & worker --attach <run_id_of_dispatcher> which should have the equivalent effect of dispatcher | worker

if you want to attach and drain, then something like

lunchpail cat --attach runid > /dev/null

etc.

[starpit]

Oh, re: termination conditions, above i discussed that a stage stays alive while its output remains unconsumed. A second way a stage can die if it is sent a SIGPIPE. i.e. we can leverage normal unix pipeline semantics here.

e.g. in a normal bash world, if i d

cat giantfile | grep somethinginfirstline

the cat terminates even though there is more to cat, because the grep found what it is looking for...