Cache metadata in container images

[vsoch]

I tried out the demo this weekend: https://eessi.github.io/docs/pilot/ and I'm worried that the slowness of, for example, running a simple command like "ipython" will be a deterrent to users. The suggestion (after talking with @boegel) is that we could cache metadata in the container image to improve startup speed.

The reason it's important is because a new developer user will be turned away from this initial sluggishness.

[boegel]

Thanks a lot for the feedback @vsoch!

Do you remember the details of the system you tried this on?
Was this a local system, in a US network? Was it in the cloud somewhere?

That's a big factor w.r.t. initial sluggishness...

The good news here is that there are plans in CernVM-FS to significantly improve client performance, which may already make a big difference.

[vsoch]

It's just on my local machine, which is ubuntu and fairly reasonable with respect to memory, etc.

[boegel]

It's not so much how beefy the machine is, but to which Stratum 1 (mirror) server it's talking to pull in the files required to run the workload. CernVM-FS decides by itself, based on geographical location.

That said, there's several things we can (and will do) to improve the latency.

Your suggestion to cache metadata in the client container image doesn't make much sense though, because the container image is static (while the contents of the EESSI CernVM-FS repository is not).
Also, most end users wouldn't be using our client container at all, but access EESSI through a native CernVM-FS installation.

Beyond that, do you have any other feedback on the project?

[vsoch]

I think that was the main bottleneck - how slow it was to get everything setup and working. I think that early user interaction where everything is installed / loaded and discovered by the user is really important, because in ~5 minutes they are going to quickly decide if they like it (is it easy? intuitive?) or not (and then move elsewhere). The question I want to pose to you next is - what use cases are the strongest to tackle first, or in other words, who is the audience? As an example, if the audience is developers and these are for development environments, it would make sense to have a bunch of tutorials that show how that works for different (common) scenarios. If it's for reproducing a workflow, do the same but for workflows. I can definitely help more with this, although I'd definitely want the entire thing to be a bit speedier first, and then maybe we can write down a plan for what use cases should be tackled first.