How should credentials.yml properly interact with env variables?

[ianwhale]

Howdy all!

I'm opening a new discussion from this thread on slack.

The larger question @noklam and I were discussing is: "How should credentials.yml properly interact with environment variables?"

This was specifically in the context of calling LLM APIs in a Kedro pipeline. The two approaches are:

• Datasets that accept credentials and dynamically set credentials when they are constructed.
• Setting global environment variables from with a dotenv hook

Using datasets

This is the approach used here: https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets_experimental/langchain/_openai.py

You'd have your credentials.yml

openai:
     openai_api_base: <openai-api-base>
     openai_api_key: <openai-api-key>

and then your catalog.yml

text_embedding_ada_002:
    type: langchain.OpenAIEmbeddingsDataset
    kwargs:
        model: "text-embedding-ada-002"
    credentials: openai

Pros:

• This feels more kedronic
• Swapping between kedrro environments is easier perhaps?

Cons:

• Requires fiddling with internals in dataset classes

Using dotenv

(Or using dynaconf's load_dot_env).

This would ignore all the bits we have to do above to set keys programmatically. Instead we'd use a hook like:

import logging

from dotenv import load_dotenv
from kedro.framework.hooks import hook_impl

logger = logging.getLogger(__name__)

class DotenvHook:
    @hook_impl
    def after_context_created(self, *args, **kwargs) -> None:
        loaded = load_dotenv()

        if loaded:
            logger.info("Dotenv loaded successfully.")

        else:
            logger.warning("Dotenv not found!")


HOOKS = (DotenvHook(),)

Then just leave it to our libaries (e.g. lanchain) to access environment variables.

Pros:

• Feels a little easier.

Cons:

• Credentials are in two places now credentials.yml and .env used by dotenv.

Wondering what the team thinks is the right way.

@noklam let me know if this gets at the question we were discussing!

Related: #3877

[ianwhale]

Potentially related: #3915

[astrojuanlu]

Hi @ianwhale, thanks for opening this discussion. I found it while looking for some material on credentials myself 😄

Recently I went ahead and implemented my own BigQuery dataset with manual credential handling on credentials.yml files #1621 (comment)

But it was a burden, because essentially I had to rewrite the JSON key file given by BigQuery into a YAML file, for very little gain.

Later I asked around and figured that nobody actually cared about that because exporting the GOOGLE_APPLICATION_CREDENTIALS variable on the terminal already fixes everything.

And then there are more complex examples, like this one we documented on fetching credentials from Azure KeyVault using hooks https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials

#3811 did a good job at suggesting a way forward and collecting some of the issues that have been active over the years about credentials, but we haven't prioritized it.

Long story short, I'm in the same page as you and I don't think our current credentials management is fit for purpose.