Python & Jupyter extension fail to detect upgraded Python

[paul-rogers]

Type: Bug

Behaviour

After an OS upgrade replaced an old Python with a new version, VS Code fails to detect the new version.

Steps to reproduce

I use Linux Mint. Version 21 included Python 3.10.12 with symlinks in /bin/python3 and /usr/bin/python3. VS code detected these and things worked just fine.

I recently upgraded to Mint version 22, which removed the Python 3.10.12 and replace it with Python 3.12.3. In a shell:

> which python3
/usr/bin/python3
> python3 --version
Python 3.12.3

However, if I use Python: Select Interpreter I get the following:

Notice that VS Code still believes that the system Python is 3.10.12. Just to be clear: I have restarted VS Code multiple times since the OS upgrade.

I reviewed the documentation. This document says that Python will detect the Python versions automatically. Clearly, it did not.

Python 3.10.12 no longer exists on the system. Yet, the Python extension seems to believe that the Python in /bin/python3 is that version. Perhaps the version information is cached. However, the documentation provides no hints (which I could find) to tell VS Code to flush its cache and to go out and look to see which version is actually installed.

At the same time, Jupyter fails to detect this version also. Create a new notebook. Select "Select Interpreter." Only the non-existent 3.10.12 interpreter is listed. Now, it could be that Jupyter fails to find the system Python because that version is managed by the OS and I've not installed the required Python modules, though the documentation says that I'll be prompted to install them.

My workaround was to create a virtual environment (in a terminal). In a terminal, with the virtual environment added to the path:

> which python
/home/paul/pyenv/3.12/bin/python
> python --version
Python 3.12.3

The documentation says that this virtual environment will be detected automatically, since it is in ~/pyenv. However, it was not. I was able to configure it via Python: Select Interpreter/Enter Interpreter Path.

Then, I had to install IPyKernel by hand after which I could finally select the correct interpreter in a new Juypter notebook.

Specific Issues

1. Neither the Python nor Jupyter extensions have noticed that Python was upgraded.
2. It may be that the displayed version number is just cosmetic and does not affect operation. Still, it would be good for it to be correct.
3. The Python extension did not find my virtual environment as promised: I had to configure it manually.
4. The Jupyter extension did not find my virtual environment and install the IPyKernel module as promised in the documentation.
5. Either provide information on how to diagnose such problems (perhaps turn on logging somewhere), or if there is no current mechanism, please provide one that tells me exactly where the Python and Jupyter extensions are searching, why, and why they choose to not find my interpreters.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

Extension version: 2024.16.1
VS Code version: Code 1.95.0 (912bb683695358a54ae0c670461738984cbb5b95, 2024-10-28T20:16:24.561Z)
OS version: Linux x64 6.8.0-47-generic
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.3
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

User Settings

languageServer: "Pylance"

testing
• unittestArgs: "<placeholder>"
• unittestEnabled: true

Installed Extensions

Extension Name	Extension Id	Version
Code Spell Checker	streetsidesoftware.code-spell-checker	3.0.1
Debugger for Java	vscjava.vscode-java-debug	0.58.0
Extension Pack for Java	vscjava.vscode-java-pack	0.29.0
GitLens — Git supercharged	eamodio.gitlens	15.6.2
Gradle for Java	vscjava.vscode-gradle	3.16.4
IntelliCode	VisualStudioExptTeam.vscodeintellicode	1.3.2
IntelliCode API Usage Examples	VisualStudioExptTeam.intellicode-api-usage-examples	0.2.9
JavaScript Debugger	ms-vscode.js-debug	1.95.1
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Jupyter	ms-toolsai.jupyter	2024.10.0
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.20
Language Support for Java(TM) by Red Hat	redhat.java	1.35.1
markdownlint	DavidAnson.vscode-markdownlint	0.56.0
Maven for Java	vscjava.vscode-maven	0.44.0
Project Manager for Java	vscjava.vscode-java-dependency	0.24.0
Pylance	ms-python.vscode-pylance	2024.10.1
Python	ms-python.python	2024.16.1
Python Debugger	ms-python.debugpy	2024.12.0
Rainbow CSV	mechatroner.rainbow-csv	3.12.0
Rewrap	stkb.rewrap	1.16.3
Scientific Terms - Code Spell Checker	streetsidesoftware.code-spell-checker-scientific-terms	0.2.2
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
Test Runner for Java	vscjava.vscode-java-test	0.42.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 x 4429)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off webnn: disabled_off
Load (avg)	3, 2, 2
Memory (System)	31.31GB (10.72GB free)
Process Argv	--crash-reporter-id 5e232e7b-355a-468c-ac4a-02c300f898bd
Screen Reader	no
VM	0%
DESKTOP_SESSION	cinnamon
XDG_CURRENT_DESKTOP	X-Cinnamon
XDG_SESSION_DESKTOP	cinnamon
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
a9j8j154:30646983
962ge761:30959799
pythongtdpath:30769146
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
g316j359:31013175
dvdeprecation:31068756
dwnewjupytercf:31046870
2f103344:31071589
impr_priority:31102340
nativerepl1:31139838
refactort:31108082
pythonrstrctxt:31112756
wkspc-onlycs-t:31132770
nativeloc1:31134641
wkspc-ranged-t:31151552
cf971741:31144450
iacca1:31156133
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31170013

[karthiknadig]

We typically don't re-run detection is some of these cases. A workaround is to run Python: Clear Cache and Reload command.

[paul-rogers]

@karthiknadig, thanks for the explanation. I ran that command and it did, indeed "forget about" the old Python versions and found the updated version. Unfortunately, if also forgot about my virtual environment and did not find it when I selected Python: Choose Interpreter. I had to configure that by hand with Enter interpreter path. Recall that I placed my virtual environment in a path location that the documentation says that VS Code will search: ~pyenv/3.12. Any reason that VS Code would not find that virtual environment?

It would be helpful to add to the documentation the steps needed when the system Python is upgraded:

1. Update the system Python.
2. Python: Clear Cache and Reload
3. Python: Select Interpreter and create/configure a new virtual environment (if needed)
4. In each notebook, Select Interpreter to pick one of the newly available interpreters.

Step 3 above is needed in systems like Linux Mint (Ubuntu base) where the system interpreter is "managed" and does not easily allow installing user-specific packages.

[karthiknadig]

You should be able to see in the logs for Output > Python and Output > Python Locator to see what it is looking for.

[paul-rogers]

@karthiknadig, thanks for the suggestion. It seems that the internals found the virtual environment, but not the Python: Select Interpreter dialog nor the Jupyter Select Kernel dialog. Any idea why this might be?

I ran another Python: Clear Cache and Reload. Here are the relevant log lines:

2024-10-30 12:07:11.021 [info] Starting Python Locator /home/paul/.vscode/extensions/ms-python.python-2024.16.1-linux-x64/python-env-tools/bin/pet server
2024-10-30 12:07:11.536 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:07:11.537 [info] Discovered env: /home/paul/anaconda3/bin/python
2024-10-30 12:07:11.577 [info] Discovered env: /bin/python3
2024-10-30 12:07:11.636 [info] Discovered env: /usr/bin/python3
...
2024-10-30 12:07:11.665 [info] Locator PyEnv took 2.126647ms
...
2024-10-30 12:07:11.705 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:09:15.754 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:09:15.794 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:09:15.794 [info] Discovered env: /home/paul/anaconda3/bin/python
2024-10-30 12:09:15.797 [info] Discovered env: /usr/bin/python3
2024-10-30 12:09:15.801 [info] Discovered env: /bin/python3
...
2024-10-30 12:09:15.853 [info] Locator PyEnv took 363.697µs
...
2024-10-30 12:13:07.849 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:19:29.004 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
2024-10-30 12:19:29.044 [info] Discovered manager: (Conda) /home/paul/anaconda3/bin/conda
2024-10-30 12:19:29.045 [info] Discovered env: /home/paul/anaconda3/bin/python
...
2024-10-30 12:37:33.179 [info] Resolved Python Environment /home/paul/pyenv/3.12/bin/python
(Above repeated about 40 times)

The whole process seems to have been repeated a couple more times though I triggered a refresh only once. The above suggests that the code did find my environment. In fact, the code did so a total of about 50 times. (Why so many?) However, if I then do Python: Select Interpreter, this is what I see:

Notice that the virtual environment does not appear on the list though it is marked as the selected interpreter.

The expected behavior is that, if the code did discover my virtual environment, that it would be available as one of the interpreters so I can select it. Any reason that it does not appear in the UI interpreter list?

A subtle point is that the first time I did the above, my virtual environment was not selected. The second time, the virtual environment selection appeared to "survive" the Python: Clear Cache and Reload, so that, even though it does not appear in the list, it is still marked as the current interpreter.

All of this would just be an annoying UI glitch if it were not then for Jupyter. Using Python: Clear Cache and Reload causes a Jupyter
notebook to "forget about" my virtual environment, which was the selected kernel. As noted above, the Python: Select Interpreter says my virtual environment is the selected one. But Jupyter knows nothing about it. Click Select Kernel and this is what I see:

Now, to be complete, let's select another interpreter using Python: Select Interpreter: the recommended one. Again run Python: Select Interpreter. The system interpreter is shown selected, but the rest of the list looks identical to the above screenshot: the selected interpreter is still in the list and my virtual environment does not appear.

Let's try one more trick. Again run Python: Select Interpreter and select \usr\bin\python3. (This is just a symlink to the other one, probably for backwards compatibility with older Linux setups.) Now, yet again click Select Kernel in the Jupyter notebook. This selected kernel does not appear, I see the same list as in the image above. Why would the selected interpreter fail to appear? Since it is just a symlink, it is identical to the interpreter which does appear. Moreover, why does the Jupyter list not contain the Anaconda entry in the Python: Select Interpreter list?

Finally, I used Python: Select Interpreter/Enter interpreter path and manually selected my environment. That environment still does not appear in the Python: Select Interpreter list. Why? This means that if I were to switch to another interpreter for some reason, VS Code will again "forget" about my virtual environment.

Fortunately, after I manually configure the virtual environment, it does then show up in the above Jupyter Select Kernel list. Why does this environment show only in Juypter, but not in Python?

In fact, once I configure my environment manually, the notebook then automatically selects that kernel, likely because the kernel name is stored in the notebook itself. Thus, the Python: Clear Cache and Reload command doesn't appear to clear the kernel associated with notebooks, it just makes those kernels unavailable to Jupyter if the kernel configured in Jupter is not in the refreshed list.

Bottom line: this is all rather fiddly and could, perhaps, be streamlined a bit to avoid the need to ask for help for such a simple task. To be clear, the following would be helpful:

1. Fix the bug that prevents the discovered virtual environment from showing in the list of interpreters.
2. Fix the bug that prevents Jupyter from displaying the full set of interpreters (kernels) in the Python interpreter list. (If the interpreter is not usable, just show it a "misconfigured" or some such to help us users figure things out.)
3. Document the process as suggested earlier.

[karthiknadig]

@paul-rogers Can you check what python.locator is set to? try setting it to native, and see if that shows up in the list of interpreters. That might also improve the Jupyter case as it uses the same set just bit more filtered.

the locator has two options js and native. The js is the legacy locator, and native is a new locator we are experimenting with. the native locator should handle things better than js, but it could have gaps in it. With native it should also handle the updates, and changes to python installs better.

[paul-rogers]

@karthiknadig, the python.locator was already set to native. Thus, the behavior described above is associated with the newer native locator.

[karthiknadig]

@DonJayamanne Any idea what might be going on here? I am not fully familiar with how the kernel picker finds/filters things.

[paul-rogers]

FWIW, while debugging Jupyter Issue #16185 , I enabled trace logging for Jupyter. I see the following in the resulting logs:

Visual Studio Code (1.95.0, undefined, desktop)
Jupyter Extension Version: 2024.10.0.
Python Extension Version: 2024.16.1.
Pylance Extension Version: 2024.10.1.
Platform: linux (x64).
Temp Storage folder ~/.config/Code/User/globalStorage/ms-toolsai.jupyter/version-2024.10.0
Workspace folder ~/omitted, Home = /home/paul
21:45:41.154 [debug] Start refreshing Kernel Picker (1730436341154)
21:45:41.751 [trace] No controller, hence notebook communications cannot be initialized for editor ~/omitted.ipynb
21:45:41.752 [trace] No controller, hence notebook communications cannot be initialized for editor ~/omitted.ipynb
21:45:41.752 [debug] Start refreshing Interpreter Kernel Picker
21:45:43.395 [trace] Kernel Spec for 'Python 3 (ipykernel)' (~/.local/share/jupyter/kernels/python3/kernel.json) is hidden. (isDefaultKernelSpec = true, language = python, registrationInfo = undefined)
21:45:43.396 [debug] Get Custom Env Variables, Class name = Em, completed in 2243ms, has a truthy return value, Arg 1: undefined, Arg 2: "RunPythonCode"
21:45:43.396 [debug] Jupyter Paths /kernels: 
21:45:43.397 [debug] Kernel Spec Root Paths, /usr/share/jupyter/kernels, /usr/local/share/jupyter/kernels, ~/.local/share/jupyter/kernels
21:45:45.116 [trace] Python API env change detected, add => '/bin/python3'
21:45:45.117 [trace] Search for KernelSpecs in Interpreter /bin/python3
21:45:45.171 [trace] Python API env change detected, add => '/usr/bin/python3'
21:45:45.172 [trace] Search for KernelSpecs in Interpreter /usr/bin/python3
21:45:45.211 [trace] Python API env change detected, add => '/home/~/anaconda3/bin/python'
21:45:45.211 [trace] Search for KernelSpecs in Interpreter ~/anaconda3/bin/python
21:45:45.248 [trace] Hiding default KernelSpec ~/anaconda3/bin/python for interpreter ~/anaconda3/bin/python (KernelSpec file ~/anaconda3/share/jupyter/kernels/python3/kernel.json)
21:45:45.252 [debug] Refreshed Environments
21:45:45.257 [trace] Python API env change detected, add => '/home/~/pyenv/3.12/bin/python'
21:45:45.257 [trace] Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python
21:45:45.364 [trace] Hiding default KernelSpec /python for interpreter ~/pyenv/3.12/bin/python (KernelSpec file ~/pyenv/3.12/share/jupyter/kernels/python3/kernel.json)
21:45:45.381 [debug] KernelProvider switched kernel to id = .jvsc74a57bd049d54976ddc812de6cd3e61a814bfa938ec6d96e5bed8701f21d85baa25c5f8d./home/~/pyenv/3.12/python./home/~/pyenv/3.12/python.-m#ipykernel_launcher

At which point the kernel starts the kernel for my notebook, etc.

To my naive eyes, it seems that the search did not find my virtual environment. It looks like it then does Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python, which is the virtual environment configured in Python and used for this notebook. Then, it says that it is Search for KernelSpecs in Interpreter ~/pyenv/3.12/bin/python.

This raises two questions: why does it not locate the virtual environment during the various searches? Why does it want to "hide" the virtual environment once it finds it?

[DonJayamanne]

@paul-rogers Lets try to focus on one thing at a time.
Looks like you are running into a few issues and we're trying to solve all here.

1. My workaround was to create a virtual environment (in a terminal). In a terminal, with the virtual environment added to the path:

What is this virtual environment?
Is this what you are referring to as the virtual env ~/pyenv/3.12/bin/python
@karthiknadig this is returned by pet but not listed in the python quick pick.

2. You do not seem to have the latest version of the Python extension.
   Please can you install the latest version, reload and try again.

3. To my naive eyes, it seems that the search did not find my virtual environment. It looks
   Yes it did.
   This is further evidence of that

KernelProvider switched kernel to id = .jvsc74a57bd049d54976ddc812de6cd3e61a814bfa938ec6d96e5bed8701f21d85baa25c5f8d./home//pyenv/3.12/python./home//pyenv/3.12/python.-m#ipykernel_launcher

Why does it want to "hide" the virtual environment once it finds it?
Hiding default KernelSpec /python for interpreter ~/pyenv/3.12/bin/python (KernelSpec file ~/pyenv/3.12/share/jupyter/kernels/python3/kernel.json)

We're not hiding Envirolnments, we're hiding a kernel spec, (the file hiddent is python3/kernel.json) thats because it does't contain anything specia. and it points to the same environment. This is not an issue.

For now lets ignore Jupyter, and deal with Python extension and not seeing the environments you expect to see
Once you have resolved this, we can then focus on why things do not appear in Jupyter kernel picker.

[paul-rogers]

@DonJayamanne, sure, let's leave Jupyter out of the discussion.

I upgraded the Python extension as you suggested and restarted VS Code. I'm now on version v2024.18.0. The interpreter menu looks the same as the image shown above. Sometime after that upgrade and restart, while writing this comment, I got an error message in the lower right of my VS Code window:

The terminal process failed to launch: Starting directory (cwd) 
"/home/paul/.vscode/extensions/ms-python.python-2024.16.1-linux-x64/python_files/deactivate/bash" 
does not exist.

Sorry that I was not clear about the virtual environment. I have just one virtual environment, the one in ~/pyenv/3.12/bin/python. I had mentioned that I created this virtual environment in a terminal to make clear that I created it using Python directly: I did not use the VS Code command to create it. I mentioned that in case it was relevant.

I created the environment in ~/pyenv/3.12 based on two bits of information. First, the [documentation]:

The extension automatically looks for interpreters in the following locations, in no particular order:

...

• Virtual environments located in the folder identified by the python.venvPath setting (see General Python settings), which can contain multiple virtual environments. The extension looks for virtual environments in the first-level subfolders of venvPath.

Going to that the python.venvPath setting, I found that pyenv was one of the defaults, which is why I chose that location:

Python: Venv Path
Path to folder with a list of Virtual Environments (e.g. ~/.pyenv, ~/Envs, ~/.virtualenvs).
[                         ]

One cool feature I stumbled upon that is worth mentioning: One can find the python.venvPath setting. Then, select Python: Select Interpreter and leave the list open. Now, fiddle with the setting. The list updates immediately. Very handy for troubleshooting.

Out of curiosity, I went ahead and set python.venvPath to ~\pyenv. Now, my virtual environment does appear in the Python: Select Interpreter list. So, this is one solution.

Just to nit pick, the top comment could also be clearer, suggestions in bold.

Folder that contains virtual environments. (e.g. ~/.pyenv, ~/Envs, or ~/.virtualenvs).

That is, the folder does not contain "a list", it contains the virtual environments directly. Also, the setting is not "a list", it must be a single directory (it seems). To check this, I changed the setting to include my original ~/pyenv folder and the symlink ~\.pyenv: ~/pyenv, ~\.pyenv, following the wording in the setting description. The result is that neither virtual environment shows in the Python: Select Interpreter. From this one is lead to believe that the setting is not a list, as shown in the example, but just a single directory.

The settings search for venvPa also found a second setting, one not mentioned in the documentation linked above:

Python: Venv Folders
Folders in your home directory to look into for virtual environments (supports pyenv, direnv and 
virtualenvwrapper by default).

The comment "supports pyenv, ... by default". Is why I placed my virtual environment in ~\pyenv. However, it seems that the comment is wrong: the code does not search in that location by default. One thing to mention is that I'm on Linux, which is case sensitive. If the code is looking for, say, ~/PyEnv, then it won't find ~/pyenv.

OK, so perhaps the default is really the name ~\.pyenv listed in the comment for python.venvPath. I tested that case by creating a symlink ~/.pyenv -> ~/pyenv, but, after creating that link, and clearing python.venvPath back to empty, my virtual environment no longer appears in the Python: Select Interpreter list. Thus, I suspect that the comment may be wrong: perhaps there are no defaults.

To test this, while leaving python.venvPath blank, I added pyenv using the Add Item button. Sure enough, my virtual environment again appears in the Python: Select Interpreter list.

So, perhaps the easiest fix is to change the last line of the setting description. Possible change in bold:

Folders in your home directory to look into for virtual environments. No default.

This leaves us with three possible bugs/changes/clarifications:

• In the documentation quoted above, include mention of the Venv Folders option.
• Clarify the comment for python.venvPath to make it clear that the entry is a single directory.
• Fix the code to match the defaults listed in the Venv Folders option (best). Or, revise the comment as suggested above. (Quick and dirty.)

I hope this gives you the information you need to resolve this issue.

[DonJayamanne]

Python: Venv Folders
Folders in your home directory to look into for virtual environments (supports pyenv, direnv and
virtualenvwrapper by default).

I think this comment is referring to the environment managers pyenv, direnv, and virtualenvwrapper, and not the directory.
I.e supports the directoryes for pyenv, direnv and virtualenvwrapper environment managers by default
E.g. for pyenv, the default directory is .pyenv, similarly for direnv it is .direnv and for virtualenvwrapper its virtualenvs or the like.

Thanks for highlighting these, we'll update the comments/descriptions in the settings
@karthiknadig /cc

[paul-rogers]

I was about to suggest that this is just a documentation improvement request. But, further exploration showed that there are bugs when using the interpreter as a kernel in Jupyter.

Once I manually set Venv Path to ~/pyenv directory, everything works fine. The specific documentation suggestions are:

• Include a section for what to do when the interpreter changes: the Python: Clear Cache and Reload command.
• In the section that describes creating a virtual environment manually, add a note that the user should set Venv Path. The docs currently promise that VS code will detect it automatically, which it won't unless I happen to choose one of the supported directory names.

Had that wording existed, I would have not had to try to sort things out myself.

It turns out that Venv Folders is broken. Don't use it until the team can fix it. See below.

Thanks for the help.

[paul-rogers]

Here's a description why Venv Folders is broken, and the time I wasted learning that fact.

I cleaned up the extra virtual environments left over from my attempts to set things up. I now have only ~/pyenv/3.12. I added pyenv to the list of Venv Folders as pyenv (since the setting says these are folders in my home directory. Now, when I select the interpreter, I see two entries for this same virtual environment:

The behavior of these two entries differs. If I select the first one, with the leading tilde, ~/pyenv/..., then again use Python: Select Interpreter, I see the second one highlighted: the one without the tilde:

Note how the text for the selected interpreter has the tilde, the highlighted one does not.

Now, if I select the second one, the one without the tilde, and again use Python: Select Interpreter, neither of the entries are highlighted:

Note that now the selected path is ./pyenv: that is, relevant to whatever VS Code's working directory happens to be. This can't be a good thing.

Just to test, I tried removing my pyenv entry from Venv Folders and added ~\pyenv instead. Now, my virtual environment does not appear in the Python: Select Interpreter` list.

So, the user has to know to select the entry with the tilde (to get an absolute path), but understand that the UI will mark the one without the tilde. I think this is just a bit confusing.

[paul-rogers]

I tried to work around the above behavior by renaming my virtual environment's folder from ~/pyenv to ~/.pyenv which earlier comments suggested is supported by default: it is evidently the one created by pyenv. I then cleared my Venv Path entries. But, now, the virtual environment does not appear in the Python: Select Interpreter list. This suggests that VS Code either does not support ~/.pyenv by default, or that a manually-created folder is not in the form that is expected for ~/.pyenv.

To double-check, I restarted VS Code, and ran Python: Clear Cache and Reload. Still nothing.

[paul-rogers]

So, the one thing which does work is to ignore the Venv Folders setting. Instead, set Venv Path (python.venvPath) to ~/pyenv.

With this setting, my virtual environment appears (once) in the Python: Select Interpreter list. If I select it, and use Python: Select Interpreter a second time, that entry is highlighted as expected.

I also found (as described in the other issue), that configuring Venv Folders causes many errors in the Jupyter exension. However, using Venv Path works smoothly.

So, the bottom line is that Venv Folders is broken: don't use it. Use Venv Path instead.

[karthiknadig]

@paul-rogers There are lots of concepts here that I think are getting mixed.

About ~/.pyenv

That is a special folder used by a tool called pyenv (https://github.com/pyenv/pyenv?tab=readme-ov-file#installation). Python extension specifically checks that folder only if you have the pyenv tool installed. The way pyenv tool creates environments is very different from how venv creates environments. Another thing, pyenv uses different layout pattern for where and how it stored files. Basically, putting regular virtual environments inside ~/.pyenv does not work. Because it is location specific to a tool. Same is true for other directories like direnv.

The broken experience here might be because the interplay between the expectations python extension has for that location (i.e, layout specific to pyenv), vs it being a regular virtual environment. the way path is getting modified says that.
 

About venvFolders

This only uses folders that are created using venv. It expects a layout like this:
"python.venvFolders" : ["/home/user/.virtualenvs", ".virtualenvs"]

Both patterns are supported. Note that it is using either absolute expanded path or a home relative path.

/home/usr/.virtualenvs
    |- env1
    |- env2
    |- env3

About venvPath

This setting is an older setting that will be deprecated. Basically, allows you to point to a single environment.

I do think that we need to improve documentation to clarify that the locations searched for are specific to certain tools, and that if those tools are not used and such locations exists then it can break experience. Note, in python, folder names and the layout of contents are used for identifying different environment types. Unless there is very well known marker file, we use heuristics like this to handle it. So, this seems like a case of using pyenv folder name in a way that it wasn't expected.

[paul-rogers]

@karthiknadig, thanks for the explanation.

Do note the bizarre behavior described above when I added my virtual environment using venvFolders. If venvPath is to be deprecated, it would be handy to have the venvFolders setting work correctly.

In particular, see the description of the duplicate entries and the use of venvFolders causing issues in Jupyter notebooks.

Thanks for your help.

[karthiknadig]

That is a bug we will investigate. I will create a tracking bug for that using your comment. (#24376)