Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow optional files in the transfer plugin #1650

Open
bbockelm opened this issue Oct 11, 2024 · 0 comments
Open

Allow optional files in the transfer plugin #1650

bbockelm opened this issue Oct 11, 2024 · 0 comments
Assignees
Labels
client Issue affecting the OSDF client enhancement New feature or request
Milestone

Comments

@bbockelm
Copy link
Collaborator

IceCube would like to switch from GridFTP and staging out within their job wrapper scripts to Pelican and having HTCSS manage the stageout.

One feature of their existing stageout framework is the ability to have output files be marked as optional: that is, depending on the physics process, the job may-or-may-not produce a given output file. They (a) do not want to rewrite how their physics software works to transition to Pelican and (b) do not want to do the old trick of creating zero-length files to make HTCSS happy.

Accordingly, to smooth the transition, we'd like to make some file transfers optional. That is (only when run in the file transfer plugin mode):

  1. Check to see if the PelicanOptionalTransfers attribute exists in the job ClassAd for the transfer; if so, parse it as a string of comma-separated filenames.
  2. For each file in the list, if the file does not exist locally, do not attempt its transfer as part of the stageout (simply don't add it to the list of things to do).
@bbockelm bbockelm added enhancement New feature or request client Issue affecting the OSDF client labels Oct 11, 2024
@bbockelm bbockelm added this to the v7.12.0 milestone Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client Issue affecting the OSDF client enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants