Make pip and venv installation instructions more visible

[astrojuanlu]

Description

Currently we're favouring conda in our installation instructions as the "recommended approach" https://docs.kedro.org/en/stable/get_started/install.html

But the Python packaging ecosystem is very fragmented, and conda usage for environment isolation seems to be around 22 % +- 1 % of the overall userbase:

• https://www.jetbrains.com/lp/python-developers-survey-2020/#PythonVersions
• https://lp.jetbrains.com/python-developers-survey-2021/#PythonVersions
• https://lp.jetbrains.com/python-developers-survey-2022/#PythonPackaging

Closer analysis reveals that conda usage is declining or stagnating around 41 % of data scientists, with 73 % reporting that they use virtualenv, Pipenv, or Poetry. From the raw data, these are the numbers I came up with:

Year	None	virtualenv + Pipenv + Poetry	conda	Other
2020	14.9 %	72.0 %	44.6 %	35.2 %
2021	16.2 %	70.0 %	40.9 %	37.0 %
2022	16.1 %	73.1 %	40.8 %	37.1 %

Among non-data scientists, conda usage lies between 13 % and 15 %.

In addition, an informal survey in Slack surfaced what I've been suspecting for long: that users have a "global" Kedro, plus a project-specific Kedro. For the "global" one, users have mentioned a variety of solutions: conda, pyenv, pipx, rtx, and more https://linen-slack.kedro.org/t/16040768/u05bdslpj72-finally-gave-the-steps-in-https-kedro-org-slack-#db2c34ed-43bf-4479-839c-5a4fb4154a10

Possible Implementation

In summary, I think we should

• Acknowledge the current reality that kedro new requires a "global Kedro"
• Favour virtualenv as the preferred/recommended solution, and mention conda in second place
• Avoid hiding commands behind collapsible menus (makes it impossible to Cmd+F)
• Remove mention of Pipenv completely, given that we don't even have good Poetry support Poetry Support for Kedro Projects #1722 and that Pipenv is a first-generation workflow tool that has been superseded by 2nd generation tools (Poetry), 3rd generation (PDM, Hatch) and now even 4th generation (rye) Reword, reorder or rework tool recommendations for application dependencies pypa/packaging.python.org#912 (comment)

[stichbury]

Let's do this. I think a new page specifically about virtual environments may be the best place to do this so it doesn't clutter the get started instructions. It means it's a separate click away but then it's all self-contained. Some decent keyword optimization and we could even start ranking if we do a good job.

[astrojuanlu]

Is there a way to tweak the actual installation instructions at https://docs.kedro.org/en/stable/get_started/install.html to have the pip install kedro on top (possibly coupled with python -m venv .venv for the actual environment creation), and then add extra info in a virtual environments page as you mention?

[stichbury]

Yes definitely. I won't be able to do that today but if you want to make a start and put a PR together, go ahead, otherwise I'll pick it up as soon as I can get to it.

[antonymilne]

Possible text for this that's used on vizro: 8e222fe.

[DimedS]

@astrojuanlu , during my onboarding at McKinsey, I learned that the company recommends using VSCode + conda environments, supported by analytics indicating it's a globally popular choice. I recall the same recommendation from the company course "Introduction to Software Engineering Principles for Data Scientists and Data Engineers." It appears you and @stichbury were responsible for that course. Does this mean you set this standard across the company? Should we consider revising it as well?

[astrojuanlu]

About the internal recommendations, let's talk about those in private.

About my own preference and what was included in the ISWE4DX course (you can see more about that here kedro-org/kedro-devrel#12 and also in https://github.com/kedro-org/kedro-academy/tree/main/iswe4dx/environment-dependencies), it's rooted on the fact that sometimes installing old packages on old Python versions and non-Linux architectures is easier with conda than with pip. However, this is evolving a lot, and in fact recently we've solved the last issue that was preventing me from completely ditching conda kedro-org/kedro-plugins#402

So I still think conda is a nice tool and ecosystem, but

• It's still in the minority (see my comments above)
• The official conda is still slower than micromamba or Prefix
• Usually folks are unaware that the defaults channel from https://anaconda.org is subject to the Anaconda Inc terms of service https://nbviewer.org/github/astrojuanlu/ie-mbd-advanced-python/blob/main/sessions/03-04_pip-vs-conda.ipynb
• Not all packages are in conda-forge, sometimes you end up having to use pip inside conda environments - and this can create lots of problems for folks that are not experts in packaging

So, with all these in mind, it's in our best interest to align ourselves with the broader Python ecosystem.

[DimedS]

• Not all packages are in conda-forge, sometimes you end up having to use pip inside conda environments - and this can create lots of problems for folks that are not experts in packaging

You right, the strange part of that recommendation was: VsCode + conda env + pip, I agree that better to change conda to virtualenv.