datalad containers-run integration: wrap battle #170
Comments
asmacdo
changed the title
|
@yarikoptic thoughts? |
|
Heres the best integration so far. I tried to take this to discovery to integrate with slurm also, but they are mid-migration this week (the interactive nodes I'd been using are down.) probably best to wait for the new setup. |
This option (used on the datalad blog) is probably the best choice, but unfortunately it cant work with containers-run without either altering containers-run or the container its running. Just add duct to the actual invocation in for all but a test container. So here you would change to and
yeap, that is what I also thought about -- should be doable and provide a solution for docker style container platforms . But may be we find some other / better ways. For now let's pretty much concentrate on "2" . In the longer run I think we should just augment For repronim/containers we could even just add handling via making that |
|
Something isnt working as expected with (side note, datalad containers-run{
"exit_code": 0,
"wall_clock_time": 396.50900530815125,
"peak_rss": 3977216,
"average_rss": 3977194.61096606,
"peak_vsz": 7761920,
"average_vsz": 7761920,
"peak_pmem": 0,
"average_pmem": 0,
"peak_pcpu": 14.6,
"average_pcpu": 5.629242819843339
}datalad run duct+sing{
"exit_code": 0,
"wall_clock_time": 370.62019300460815,
"peak_rss": 40427520,
"average_rss": 24678171.810584974,
"peak_vsz": 1603104768,
"average_vsz": 1274291887.4206126,
"peak_pmem": 0.5,
"average_pmem": 0.03899721448467966,
"peak_pcpu": 12.3,
"average_pcpu": 6.814763231197769
}Here we see more reasonable values for memory usage. Both runs saved to different output dirs on the master branch for easy comparison. |
IIRC it is from the top of the dataset, happen you run command in a subfolder -- you would find it super helpful ;) So check the stats logs -- likely it didn't track kid processes correctly, so rerun and see what is up with the session id etc, or may be even run with higher log level for duct (via env var?) to see what it sees while running |
|
try to submit SLURM job (without datalad but using that container you download using datalad) going through all subjects of the original dataset ds000003 and most recent mriqc. even without |
|
Original DS https://github.com/OpenNeuroDatasets/ds000003 |
Goal: Integrate with datalad and singularity (and SLURM) to execute MRIQC
Option 1 duct wraps datalad
duct datalad containers-run ...This doesn't work as one would hope, because prior to execution, duct creates new files which means datalad starts from a dirty state and fails. We can get around this with
datalad run --explicit, but it doesn't quite work.Which seems like its working... but not really. Datalad commits the duct files prior to duct's exit.
(Another workaround is to pass an output prefix outside of the dataset, but thats not great, since we would want that data committed.)
Option 2 datalad wraps duct
This option (used on the datalad blog) is probably the best choice, but unfortunately it cant work with containers-run without either altering containers-run or the container its running.
It does work with
datalad runby simply wrapping the command to execute the container. But this loses all the benefits of native datalad container integration.Option 3 datalad wraps container wraps duct
We could also try to set up a clever container with duct bind-mounted in, and override the entrypoint to wrap duct around it. This option feels complex and will be opaque to users. However this pattern would work with OCI containers, all other options are intended for apptainer/singularity.f
The text was updated successfully, but these errors were encountered: