Perform namespace renaming for confirms in modular pipelines

[antoinebon]

Kedro version: 0.18.0

It seems that at the moment, confirms nodes in modular pipelines with namespace are not renaming their references to IncrementalDataSet inputs/outputs in the confirms list, which makes it hard to use the confirm functionality within modular pipelines.

In _copy_node, only inputs and outputs are renamed, but not confirms:

    def _copy_node(node: Node) -> Node:
        new_namespace = node.namespace
        if namespace:
            new_namespace = (
                f"{namespace}.{node.namespace}" if node.namespace else namespace
            )

        return node._copy(
            inputs=_process_dataset_names(node._inputs),
            outputs=_process_dataset_names(node._outputs),
            namespace=new_namespace,
        )

For example the following code will not work because it will look for a dataset named incremental_dataset in the catalog instead of a dataset named my_namespace.incremental_dataset

pipeline(
    pipe=Pipeline([
        node(
            func=partition,
            input='input'
            output='incremental_dataset'
        ),
        node(
            func=process_fn,
            input='incremental_dataset',
            output='processed_input',
            confirms='incremental_dataset'
        )
    ]),
    namespace='my_namespace'
)

[antonymilne]

Hmmm, this is a tricky one. What you say definitely makes sense, but I think this is probably deliberate, and changing it would overall not improve the situation.

If we were to automatically namespace confirms in the same way that we do input and output, we'd also need to let pipeline take that as an argument. Since confirms datasets can also come from any pipeline, we'd need to enable all the same options that we allow for inputs and outputs (i.e. Union[str, Set[str], Dict[str, str]]). This seems like quite a complication for the benefit of a feature that to be honest is quite specialist and not well used (confirms). I also think it's less clear that namespacing by default is the "correct" behaviour for confirms (vs. inputs/outputs/parameters, where it does seem to be the best default). On the other hand, this would make the behaviour for confirms consistent compared to inputs/outputs/parameters. Hmmm...

P.S. small simplification - as of 0.17.7 you no longer need to nest pipeline and Pipeline calls like that. It's best just to use pipeline:

pipeline(
   [
        node(
            func=partition,
            input='input'
            output='incremental_dataset'
        ),
        node(
            func=process_fn,
            input='incremental_dataset',
            output='processed_input',
            confirms='incremental_dataset'
        )
    ],
    namespace='my_namespace'
)

[antoinebon]

Thanks for the detailed explanation Antony! Indeed if the intent is that confirms works across pipelines then it's probably not a good idea to apply the namespace to it. One solution is to manually namespace the confirms, which is not that much effort.

[FlorianGD]

Hello, I just want to chime in to say that I'd also like this feature. I have a (custom) datasets that has a confirms. I had the perfect use case for a modular pipeline where I have the exact same pipeline, but different inputs and outputs. Sadly, it seems that due to confirms not being namespaced, I have to manually duplicate the pipeline so that each pipeline confirms the correct dataset

[astrojuanlu]

Could you folks open this as an issue? I think it's going to receive more attention 🙏🏽

[adrienpl]

Same for me I have the same issue

[astrojuanlu]

Hi @adrienpl could you go to https://github.com/kedro-org/kedro/issues/new/choose and open a new issue describing your setup?