Resume pipeline after breaks/crashes

[sky-2002]

Is your feature request related to a problem? Please describe.
I have a haystack pipeline that has certain components, most of which interacts with LLMs, and have custom processing logic.
At times, if a component in the later stages fails(there could be a lot of reasons), I have to re-run the pipeline for the specific input.

Reference - deepset-ai/haystack-experimental#281 (comment)

Describe the solution you'd like
What I would think of is a way to maintain stage for every component, and then whenever a pipeline breaks/crashes at a component, a way to resume from last state.

Benefits:

• Avoids running computationally heavy workloads(like GPU inference for embeddings), or cost heavy workloads(LLM calls) repeatedly, if there are any in the previous stages of the pipeline.

A discussion on this is preferred, around these points roughly:

• Whether this is a common problem or not, if it is only me, I would rather not make changes to haystack code
• What other benefits this could have, if at all we implement this
• Can we reuse(or borrow) ideas from breakpoints feature?
• Any workarounds?
• Implementation

Describe alternatives you've considered
I would appreciate a discussion around this.

Additional context
I would be happy to contribute this via a PR if we refine this idea and decide to go ahead with it.

fyi @davidsbatista

[sky-2002]

@davidsbatista @sjrl Hey guys, just pinging in case this got missed, any thoughts on this?

[davidsbatista]

Hi @sky-2002! I'm working on it on having a PoC - I will keep you posted here as soon as there's something to try out/test

[sky-2002]

Amazing. Feel free to pull me in if you need any help with the tasks.

[davidsbatista]

Hi again @sky-2002!

So, I have a simple PoC - which reuses as much code as possible from Breakpoints logic to create a persistance state as the Pipeline runs each component.

The idea is to pass two variables at run time, i.e.: state_persistence and state_persistence_path, i.e.:

results = pipeline.run(
    data=test_data, 
    state_persistence=True, 
    state_persistence_path=snapshots_dir
)

This will save a PipelineState in the path set at state_persistence_path before each component is run.

I included two simple pipelines for you test, check it at:

• haystack/test/core/pipeline
• test_pipeline_persistance_state_complex.py - an hybrid retrieval pipeline
• test_pipeline_persistence_state_simple.py - a Pipeline that simples queries an LLM

All the code is on this branch: https://github.com/deepset-ai/haystack/tree/poc/resume-state-for-crashes - you can clone it or if you prefer you can install it as a pip package

pip install git+https://github.com/deepset-ai/haystack.git@poc/resume-state-for-crashes

Let me know what you think, feedback is very welcome!

[sky-2002]

That's great. I will look at it and give it a try.

Let me know what you think, feedback is very welcome!

yeah sure