jupyterconfig error while running on server

[joshqsumner]

Describe the bug
A user is having trouble with jupyterconfig on the v5 branch. jupcon.run() seems to find all the images, write the python file, and write the config but the workflow runs unrealistically fast. They are trying to save out masks for use in another software so they are not touching args.result which should not matter since checkpointing is not merged to v5 so far.
There are also difficulties with this workflow running in other settings it sounds like. Could be conda env related as well, not sure that the getenv: true flag is doing enough through jupyterconfig especially with clusters other than LocalCluster.
May also have been issues with saving the python script, not clear immediately if that is from the jupyter notebook not having had edits saved before calling jupcon.run() though.

To Reproduce
Talk to Noah/Josh for stargate path to jupyter notebook, running that notebook should reproduce error.

Expected behavior
Notebook should run in parallel saving a directory of masks.

Local environment (please complete the following information):

• OS: Linux
• Environment: Conda
• PlantCV Version: 5

Additional context
This is the first time running jupyterconfig on the new stargate cluster, not entirely sure how much of this is a problem with that feature but probably a good bit.

[joshqsumner]

Trying running this on the jupyter server and I got a bunch of errors about not being able to import PlantCV suggesting it's a problem with getting the conda environment.
The jupcon.run call does what it is supposed to but it does not give you an error (per normal behavior in parallel) if workers fail. The workers all fail with error logs like:

2026-01-21 11:28:25,855 - distributed.core - INFO - Event loop was unresponsive in Worker for 18.05s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
Traceback (most recent call last):
  File "/data/datascience/users/jsumner/khein/root-segmentation.py", line 18, in <module>
    from plantcv import plantcv as pcv
ModuleNotFoundError: No module named 'plantcv'

I think if that could be imported it would work, not immediately sure how to tell dask to get the environment corresponding to the active jupyter kernel though, that seems tricky. Might say that it is limited to kernels made from conda environments or global?

Initializing from /data/datascience/users/jsumner/khein/root-segmentation.ipynb Notebook
Saved /data/datascience/users/jsumner/khein/root-segmentation.json
Starting parallel workflow from notebook kernel
Starting run 2026-01-21_11-27-52

Reading image metadata...
Reading image metadata took 0.19377446174621582 seconds.
Building job list... 
Task list includes 33 workflows
Building job list took 0.10391831398010254 seconds.
Processing images... 
Done!
Processing images took 33.2482271194458 seconds.
Processing results... 
Processing results took 0.04070711135864258 seconds.
Converting json to csv... 
Processing results took 0.06242108345031738 seconds.

Checking environment variables with os.environ I am seeing that on the remote (ddpsc jupyter server) I am not in a conda env despite having activated it on the terminal. Maybe the whole jupyter server is running without conda on a machine and my conda activation is only useful for making another kernel that I can then call in the notebook. That would be one way to explain why this feature works on my LocalCluster but not on the datascience jupyter server with HTCondorCluster. That would also explain why setting the job_extra_directives to getenv: true doesn't do anything on the jupyter server. Going to postpone digging more into this until @nfahlgren can provide some more direction on how we'd want this to work.

Seems possible to use job_script_prologue in the dask scheduler to activate a conda env for each job, maybe you could be prompted to do that if there is no environment detected?

Edit:

Pretty sure this confirms the problem since I got it to work by just adding these to the jupcon object

jupcon.cluster_config["submit_command_extra"] = ["request_memory = 10GB"] # Hungry for memory
jupcon.cluster_config["job_script_prologue"] = ["source /data/datascience/users/jsumner/miniforge3/bin/activate", "conda activate plantcv"] # start a conda env with plantcv since the server relies on the kernel instead

Yields this and populates the expected output folders:

Initializing from /data/datascience/users/jsumner/khein/root-segmentation.ipynb Notebook
Saved /data/datascience/users/jsumner/khein/root-segmentation.json
Starting parallel workflow from notebook kernel
Starting run 2026-01-21_14-08-55

Reading image metadata...
Reading image metadata took 0.0206453800201416 seconds.
Building job list... 
Task list includes 33 workflows
Building job list took 0.07767415046691895 seconds.
Processing images... 
Done!
Processing images took 292.95569920539856 seconds.
Processing results... 
Processing results took 0.04395794868469238 seconds.
Converting json to csv... 
Processing results took 0.04329562187194824 seconds.

[joshqsumner]

Removing bug labels since I think this is expected behavior given how the jupyter server is set up.

[joshqsumner]

could find path to conda env with sys.executable. That might actually work more robustly than the getenv: true flag in the config (which does not work on PBS). On AWS Noah is basically doing the same thing that I'm doing on the jupyter server.

Might be worth automating, could happen in jupyterconfig or in run_parallel when it makes the workers (same thing really, but for both types of config). So if job_script_prologue is none then edit to add the conda env.