Running fits on HTCondor

[ssaha1234]

Dear pyhf users,
I would like to submit pyhf jobs on HTCondor for doing mle fits. I can run the fits locally by doing the following steps -

$ source setup.sh
$ python Fit.py

• setup.sh: https://gist.github.com/ssaha1234/d504ce501bbc1e126f47dbb7233c51b2
• req.txt: https://gist.github.com/ssaha1234/eace318069d44f0ed9de041ed937b1d9
• Fit.py: https://gist.github.com/ssaha1234/0e84c80cd22fb36126c7302462170ae6
• Condor submission script: https://gist.github.com/ssaha1234/36823f9620b11e91c9292d2b0b15580b

The error file, makes sense to me, since I am not setting up pyhf on the machine, the job is submitted to -

Traceback (most recent call last):
File "condor_exec.exe", line 2, in <module>
   import pyhf
ModuleNotFoundError: No module named 'pyhf'

Is there a way to setup pyhf on condor and submit the jobs ? I am thinking if we have to write a shell script wrapper for it ?
Thank you !
Best,
Shreya

[matthewfeickert]

Is there a way to setup pyhf on condor and submit the jobs ? I am thinking if we have to write a shell script wrapper for it?

There's nothing specific to pyhf here. If you want to load a particular environment then you need to ensure that environment gets loaded in your HTCondor submit file's Executable. So if you make your Executable a shell script (e.g. example.sh) you would already have generated your Python virtual environment in advance in a location that is accessible by HTCondor (so you're not creating it over and over each time you run) and then source that Python virtual environment before you do anything.

For example, if you were on a CVMFS machine and had a Python virtual environment built from a requirements.txt and create_cvmfs_venv.sh

create_cvmfs_venv.sh:

#!/bin/bash

# load LCG view to get Python
. /cvmfs/sft.cern.ch/lcg/views/LCG_101/x86_64-centos7-gcc11-opt/setup.sh
# Create Python virtual environment
python -m venv venv
. venv/bin/activate
# Ensure that the Python virtual environment is at head of PYTHONPATH
# c.f. https://github.com/matthewfeickert/cvmfs-venv for more info and alternatives to deal with this
export PYTHONPATH="$(readlink -f $(find venv -type d -iname site-packages)):${PYTHONPATH}"

python -m pip install --upgrade --ignore-installed pip setuptools wheel
python -m pip install --upgrade --ignore-installed pip-tools

# Generate environment lock file
pip-compile \
    --generate-hashes \
    --output-file requirements.lock \
    requirements.txt

# Use Brett Cannon's recommendations for pip-secure-install to ensure environment
# is reproducible and installed as securely as possible.
# c.f. https://www.python.org/dev/peps/pep-0665/#secure-by-design
# c.f. https://github.com/brettcannon/pip-secure-install
# c.f. https://twitter.com/brettsky/status/1486137764315688961
# As working in CVMFS LCG views also add --ignore-installed to avoid CVMFS packages
python -m pip install \
    --ignore-installed \
    --no-deps \
    --require-hashes \
    --only-binary :all: \
    --requirement requirements.lock

python -m pip list --local

then if you had your executable be something like example.sh which calls example.py

example.sh:

#!/bin/bash

# Load environment
. /cvmfs/sft.cern.ch/lcg/views/LCG_101/x86_64-centos7-gcc11-opt/setup.sh  # load LCG view to get Python
. /afs/cern.ch/user/f/feickert/htcondor_example/venv/bin/activate  # load Python virtual environment
# Ensure that the Python virtual environment is at head of PYTHONPATH
# c.f. https://github.com/matthewfeickert/cvmfs-venv for more info and alternatives to deal with this
# This should be the equivalent of: export PYTHONPATH="$(readlink -f $(find venv -type d -iname site-packages)):${PYTHONPATH}"
export PYTHONPATH="/afs/cern.ch/user/f/feickert/htcondor_example/venv/lib/python3.9/site-packages:${PYTHONPATH}"

# Run script
python example.py

you should be able to submit it in your HTCondor submission script and have things run.

Here's a GitHub Gist with a full example.

(Note that I wrote this without running it and I'm more of a SLURM user, so I might have made a mistake.)

[kratsg]

Using cvmfs-venv from @matthewfeickert, this works for me just fine:

curl -sLO https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/atlas_setup.sh
. atlas_setup.sh testssha1234
python -m pip install -r requirements.txt

The issue is just with how lxplus deals with python virtual environments (which is not very well)

[ssaha1234]

Hi @kratsg ,
Thank you very much for the check, I am not using lxplus in this case, I should have mentioned it before, when I asked the question.
I am using the ATLAS analysis facility at UChicago. Hence, I think there might be some new changes in above example ?
Best,
Shreya

[kratsg]

This also works on any machine with CVMFS. So try it out.

[ssaha1234]

Hi @kratsg,
Thanks for letting me know, I will try it out from scratch, from the github gist :)
Best,
Shreya

[ssaha1234]

Hi @kratsg,
It all worked for me now ! Thank you very much !
Best,
Shreya