Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to have different .cache directories for different workspaces #407

Open
kelek opened this issue Jun 27, 2019 · 10 comments
Open

Comments

@kelek
Copy link

kelek commented Jun 27, 2019

Currently if I have 2 IntelliJ workspaces that both use the Pants plugin, they both share the same ~/.cache directory, which can cause issues when syncing different workspaces to the same directory. Aside from hacking together a solution that runs IntelliJ as a different user, and piping that through to the same window session of another user (or using VNC), there doesn't seem a clean way of working around this when using multiple workspaces.

This request would simply move the .cache directory inside the workspace (if possible), rather than in the user's home directory.

@wisechengyi
Copy link
Collaborator

@kelek I am not sure why the cache would cause a conflict. Nevertheless,

  1. ~/.cache/pants/ managed by Pants, that's managed by pants, so you can change it by tweaking the env var https://github.com/pantsbuild/pants/blob/ea74e998027b0b35b10a732819a07b847a1f85cc/src/python/pants/base/build_environment.py#L34-L40

  2. It's coursier cache having issue, you can also override it in pants.ini https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/jvm/tasks/coursier/coursier_subsystem.py#L31-L32

@kelek
Copy link
Author

kelek commented Jun 28, 2019

For 1, the issue is I don't think you can set a different environment variable that is tied to a workspace in IntelliJ. You can setup PATH variables, but it's shared across workspaces. You can set environment variables in run configs, but it doesn't apply to general pants builds/sync processes. I don't think you can set environment variables in pants.ini (only pants variables), so I'm not sure if anything can be done there to have different cache directories (i.e., different env vars) for different workspaces.

For 2, it's not using coursier.

@wisechengyi
Copy link
Collaborator

I see. Could you elaborate a bit what kind of conflicts are you running into?

@kelek
Copy link
Author

kelek commented Jul 2, 2019

It seems to produce a similar issue to
pantsbuild/pants#3339

which produces a similar stack, with the error

Exception message: Ivy returned 1.

which I suspect is because the plugins are sharing the same ivy cache. (The same branch works fine in another workspace.) That said, I'm still not seeing what exact content in ~/.cache would cause that conflict, so I'm still looking.

@wisechengyi
Copy link
Collaborator

wisechengyi commented Jul 2, 2019 via email

@kelek
Copy link
Author

kelek commented Jul 2, 2019

Thanks!
https://www.pantsbuild.org/setup_repo.html

[ivy]
ivy_settings: %(pants_supportdir)s/ivy/ivysettings.xml
cache_dir: ~/.ivy2/pants

Only caveat was that I couldn't set it in pants.ini to ~/.ivy3/pants since it would give the same error, but running the exact command produced something more informative:

java.text.ParseException: ... ivy.cache.dir must be absolute: ~/.ivy3/pants

Using the absolute path seemed to make much more progress, but it still has issues that I'm looking into. Oddly enough, the error messages never show when the pants IntelliJ plugin fails (it only shows Exception message: Ivy returned 1.) where the real issues are only visible if you copy/paste the command after Exception message: Ivy returned 1. cmd= in the same directory. (The latest issue is some dependencies resolve fine for one workspace, but not the other, so I suspect some repo setting, but I can't quite figure out how that would matter yet. Or that could be an artifact of running it on the command line and not the real issue returning the 'Ivy returned 1' error in IntelliJ.)

@wisechengyi
Copy link
Collaborator

@kelek you could use

[DEFAULT]
ivy_cache_dir: %(homedir)s/.ivy3

[ivy]
cache_dir: '%(ivy_cache_dir)s'

@kelek
Copy link
Author

kelek commented Jul 2, 2019

I tried that, but it had exactly the same result; the issue was resolving some internal repos. At the end of the error it mentions

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

but since it's running java, not pants, I'm not sure how explicitly to enable it yet.

/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java --resolve-ivy-args=-debug -Dsun.io.useCanonCaches=false -Divy.cache.dir='/Users/mtylutki/.ivy3' -cp ../../.ivy2/pants/org.apache.ivy/ivy/jars/ivy-2.4.0.jar org.apache.ivy.Main -settings /Users/mtylutki/<workspace>/<project>/build-support/ivy/ivysettings.xml -ivy /Users/mtylutki/<workspace>/<project>/.pants.d/ivy/5ef659cf7948fda952561a07540eacac7ddefcf9/resolve-ivy.xml -confs default sources javadoc -cachepath /Users/mtylutki/<workspace>/<project>/.pants.d/ivy/5ef659cf7948fda952561a07540eacac7ddefcf9/classpath.raw.tmp.5522a01fcf70407ea1ed98106b060d8e

I also noticed that despite pants.ini settings, it uses -cp ../../.ivy2/pants/org.apache.ivy/ivy/jars/ivy-2.4.0.jar, but I don't anticipate that making a large difference, but trying to change that might help.

@wisechengyi
Copy link
Collaborator

@kelek possibly you need to do a clean-all first, or a bump the cache key? seems like it's re-using a symlink under .pants.d/ that links back to ~/.ivy2

@kelek
Copy link
Author

kelek commented Jul 3, 2019

I purged the entire <workspace>/<project>/.pants.d directory between rebuilds, and it produced the same error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants