Pigeons installation issue via octofitterpy

[sefffal]

Raise by @Bhavesh012

Hey William, thanks for fixing the issue, and I'm sorry for getting back to you so late (deadlines after deadlines :') ).

So, the issue of instrument names is now solved. But now, a new issue has arisen:

JuliaError: ArgumentError: Package Pigeons not found in current path. Run "import Pkg; Pkg.add("Pigeons")" to install the Pigeons package.

This also makes me wonder how octofitterpy parses the current Julia environment. Does it create a new one even if one exists? Or is there a way to initialise octofitterpy only with a specified Julia environment?

My current Julia environment has everything installed.

The full error output is here:

---

JuliaError Traceback (most recent call last)
Cell In[17], line 1
----> 1 chain, pt = octo.octofit_pigeons(model, n_rounds=2)

File ~/coding/MPIA/APEX/octofitterpy/octofitterpy/python_interface.py:130, in octofit_pigeons(*args, **kwargs)
129 def octofit_pigeons(*args, **kwargs):
--> 130 jl.seval("using Pigeons")
131 return Octofitter.octofit_pigeons(*args, **kwargs)

File ~/.julia/packages/PythonCall/Nr75f/src/JlWrap/module.jl:27, in seval(self, expr)
25 return ValueBase.dir(self) + self._jl_callmethod($(pyjl_methodnum(pyjlmodule_dir)))
26 def seval(self, expr):
---> 27 return self._jl_callmethod($(pyjl_methodnum(pyjlmodule_seval)), expr)
28 """,
29 @FILE(),

JuliaError: ArgumentError: Package Pigeons not found in current path.

• Run import Pkg; Pkg.add("Pigeons") to install the Pigeons package.
  Stacktrace:
  [1] macro expansion
  @ ./loading.jl:1772 [inlined]
  [2] macro expansion
  @ ./lock.jl:267 [inlined]
  [3] _require(into::Module, mod::Symbol)
  @ Base ./loading.jl:1753
  [4] #invoke_in_world#3
  @ ./essentials.jl:926 [inlined]
  [5] invoke_in_world
  @ ./essentials.jl:923 [inlined]
  [6] require(into::Module, mod::Symbol)
  @ Base ./loading.jl:1746
  [7] eval
  @ ./boot.jl:385 [inlined]
  [8] eval
  @ ./Base.jl:88 [inlined]
  [9] pyjlmodule_seval(self::Module, expr::Py)
  @ PythonCall.JlWrap ~/.julia/packages/PythonCall/Nr75f/src/JlWrap/module.jl:13
  [10] pyjl_callmethod(f::Any, self::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject}, nargs::Int64)
  @ PythonCall.JlWrap ~/.julia/packages/PythonCall/Nr75f/src/JlWrap/base.jl:67
  [11] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
  @ PythonCall.JlWrap.Cjl ~/.julia/packages/PythonCall/Nr75f/src/JlWrap/C.jl:63

[sefffal]

Thanks for the bug report! Pigeons is currently marked as one of the Julia dependencies of octofitterpy via juliacall.py. My understanding is that juliacall creates its own global Julia environment with the Julia dependencies of all installed python packages. I'm interested to understand why that isn't working as expected here.

[Bhavesh012]

I see. Is it possible to include the functionality to initiate or link a juliacall to an existing Julia environment? Is this something that can be set manually before? I believe it's much easier to manage the Julia environment from the Julia terminal than the juliacall's own global environment.

[sefffal]

Hi @Bhavesh012 , yes I believe it is. I think you can set some environment variables before starting python to control this: https://juliapy.github.io/PythonCall.jl/stable/juliacall/#julia-config

This environment variable lets you point to a particular Julia binary:
PYTHON_JULIACALL_BINDIR=<path-to-julia-binary>

This one lets you, I think, specify the Julia package environment to use:
PYTHON_JULIAPKG_PROJECT=...

And BTW you probably always want to set JULIA_NUM_THREADS=auto as well.

[Bhavesh012]

Okay, so I tried setting the Julia binary and package environment as environment variables on my jupyter notebook running on VSCode. And then I imported octofitterpy. This did 2 things:

1. Selected the correct project.toml file and updated it. Technically it wiped the whole project.toml file and just put pythoncall as a dependency.
2. Made the kernel crash. Probably due to now corrupted project.toml file?

Here's the complete Jupyter output when I ran import octofitterpy as octo:

Visual Studio Code (1.95.1, undefined, desktop)
Jupyter Extension Version: 2024.10.0.
Python Extension Version: 2024.18.0.
Pylance Extension Version: 2024.11.1.
Platform: darwin (arm64).
Temp Storage folder ~/Library/Application Support/Code/User/globalStorage/ms-toolsai.jupyter/version-2024.10.0
Workspace folder ~/coding/MPIA/APEX, Home = /Users/bhavesh
20:11:53.354 [info] Starting Kernel (Python Path: ~/opt/anaconda3/envs/exo_orbit/bin/python, Conda, 3.10.14) for '~/coding/MPIA/APEX/predoc_work/Computing/GJ676/octofitter_runs/octo_GJ676.ipynb' (disableUI=true)
20:11:55.946 [info] Process Execution: ~/opt/anaconda3/envs/exo_orbit/bin/python -m pip list
20:11:55.995 [info] Process Execution: ~/opt/anaconda3/envs/exo_orbit/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
20:11:55.996 [info] Process Execution: ~/opt/anaconda3/envs/exo_orbit/bin/python -m ipykernel_launcher --f=/Users/~/Library/Jupyter/runtime/kernel-v395434fb43cf95a3489b3f3916912110aa3a56c5a.json
    > cwd: //Users/~/coding/MPIA/APEX/predoc_work/Computing/GJ676/octofitter_runs
20:11:56.902 [info] Kernel successfully started
20:11:56.906 [info] Process Execution: ~/opt/anaconda3/envs/exo_orbit/bin/python /Users/~/.vscode/extensions/ms-toolsai.jupyter-2024.10.0-darwin-arm64/pythonFiles/printJupyterDataDir.py
20:12:28.644 [error] Disposing session as kernel process died ExitCode: 1, Reason:     Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/coding/MPIA/APEX/Project.toml`
  [6099a3de] + PythonCall v0.9.23
    Updating `~/coding/MPIA/APEX/Manifest.toml`
  [992eb4ea] + CondaPkg v0.2.23
  [9a962f9c] + DataAPI v1.16.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.6.1
  [0f8b85d8] + JSON3 v1.14.1
  [1914dd2f] + MacroTools v0.5.13
  [0b3b1443] + MicroMamba v0.1.14
  [bac558e1] + OrderedCollections v1.6.3
  [69de0a69] + Parsers v2.8.1
  [fa939f87] + Pidfile v1.3.0
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [6099a3de] + PythonCall v0.9.23
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [856f2bd8] + StructTypes v1.11.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [e17b2a0c] + UnsafePointers v1.0.0
  [f8abcde7] + micromamba_jll v1.5.8+0
  [0dad84c5] + ArgTools v1.1.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching v1.11.0
  [b77e0a4c] + InteractiveUtils v1.11.0
  [4af54fe1] + LazyArtifacts v1.11.0
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] + Libdl v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [a63ad114] + Mmap v1.11.0
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.11.0
  [de0858da] + Printf v1.11.0
  [3fa0cd96] + REPL v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization v1.11.0
  [6462fe0b] + Sockets v1.11.0
  [f489334b] + StyledStrings v1.11.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test v1.11.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [deac9b47] + LibCURL_jll v8.6.0+0
  [e37daf67] + LibGit2_jll v1.7.2+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.6+0
  [14a3606d] + MozillaCACerts_jll v2023.12.12
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.59.0+0
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
┌ Warning: attempting to remove probably stale pidfile
│   path = "/Users/~/.julia/compiled/v1.11/Tables/Z804B_luop5.ji.pidfile"
└ @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:249
┌ Warning: attempting to remove probably stale pidfile
│   path = "/Users/~/.julia/compiled/v1.11/Parsers/833b9_luop5.ji.pidfile"
└ @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:249
┌ Warning: attempting to remove probably stale pidfile
│   path = "/Users/~/.julia/compiled/v1.11/micromamba_jll/c71yJ_luop5.ji.pidfile"
└ @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:249
   1203.6 ms  ✓ Tables
    800.4 ms  ✓ micromamba_jll
   1123.0 ms  ✓ MicroMamba
   9981.4 ms  ✓ Parsers
   5449.9 ms  ✓ JSON3
   1788.7 ms  ✓ CondaPkg
   7707.3 ms  ✓ PythonCall
  7 dependencies successfully precompiled in 26 seconds. 45 already precompiled.
  No Changes to `~/coding/MPIA/APEX/Project.toml`
  No Changes to `~/coding/MPIA/APEX/Manifest.toml`

20:13:14.644 [error] Error in notebook cell execution Error: The kernel 'exo_orbit (Python 3.10.14)' died. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.
    > Kernel Id = .jvsc74a57bd0528b2443dcb05adc1c028905571d9886313a59eb430b8de1d21a21c2f1813909./Users/~/opt/anaconda3/envs/exo_orbit/python./Users/~/opt/anaconda3/envs/exo_orbit/python.-m#ipykernel_launcher
    > Interpreter Id = ~/opt/anaconda3/envs/exo_orbit/bin/python
    > at Function.verifyKernelState (/Users/~/.vscode/extensions/ms-toolsai.jupyter-2024.10.0-darwin-arm64/dist/extension.node.js:304:80754)
    > originalException = undefined
20:13:14.645 [error] Error in execution (get message for cell) Error: The kernel 'exo_orbit (Python 3.10.14)' died. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.
    > Kernel Id = .jvsc74a57bd0528b2443dcb05adc1c028905571d9886313a59eb430b8de1d21a21c2f1813909./Users/~/opt/anaconda3/envs/exo_orbit/python./Users/~/opt/anaconda3/envs/exo_orbit/python.-m#ipykernel_launcher
    > Interpreter Id = ~/opt/anaconda3/envs/exo_orbit/bin/python
    > at Function.verifyKernelState (/Users/~/.vscode/extensions/ms-toolsai.jupyter-2024.10.0-darwin-arm64/dist/extension.node.js:304:80754)
    > originalException = undefined

[sefffal]

Hi @Bhavesh012 , sorry to hear that. Can you remind me, is it possibly for you use octofitterpy without pointing it to your custom Julia project and installation?

It might be worth just doing pip uninstall juliacall octofitterpy and then pip install octofitterpy , maybe in a fresh visual environment or condo environment.

[Bhavesh012]

Yeah, that was not expected. ':D

Yes, I can still use octofitterpy without pointing it to my custom Julia project. When I do that, it just goes on the default way of making a global Julia environment. And then, we are back on the Pigeons not found error.

What I wonder is that when I pointed the juliacall to my custom Julia project, it went on and added pythoncall but didn't proceed to add octofitter and other dependencies. What could be interrupting this and also killing the Python kernel?

[sefffal]

@Bhavesh012 here is the juliapkg.json file that specifies what dependencies to install:

https://github.com/sefffal/octofitterpy/blob/master/octofitterpy/juliapkg.json

This is how both the Octofitter package and Pigeons are included, so it's weird to me that some packages were installed and some weren't.

Re: killing the python kernel
I think that could happen if python is linking against an incompatible version of Julia, but I don't have a lot more insight.

The pysr.py package uses the same setup, so I gathered up a list of possibly relevant issues from their:

• [BUG]: Julia interface fails on conda environments MilesCranmer/PySR#448
• [BUG]: Compatibility Issue with Julia Version When Using PySR MilesCranmer/PySR#600
• Error Running PySR with Julia in Windows 10 (Anaconda Environment) MilesCranmer/PySR#586
• [BUG]: PySR currently incompatible with Python 3.12 MilesCranmer/PySR#451

[Bhavesh012]

Hey @sefffal,

Yeah, that's definitely weird. I also looked at the PySR issues, but they didn't make a lot of sense to me as I am still pretty new to Julia. ':D