Allow both user-specific repo configs and repo configs managed by version control

[matts1]

Fixes #3684

This adds the following workflow to jj to support repo-level workflows:

1. Repo updates config.toml in the VCS
2. User retrieves changes
3. User runs jj command, gets a warning saying "please run jj config update-repo"
4. jj config update-repo adds a TUI requesting the user to approve the diff
5. The approved changes are copied to a permanent config file.

Checklist

If applicable:

• I have updated CHANGELOG.md
• I have updated the documentation (README.md, docs/, demos/)
• I have updated the config schema (cli/src/config-schema.json)
• I have added/updated tests to cover my changes

[PhilipMetzger]

Please format the commits to adhere to our commit guidelines here: https://github.com/jj-vcs/jj/blob/main/docs/contributing.md#commit-guidelines and add a motivation to each commit. This is also missing tests and a changelog entry since the second commit even introduces a new command.

[matts1]

I know, that's why it's a draft commit. Just want to make sure the concept is approved before I add finishing touches.

[yuja]

cc @arxanas, @ilyagr

The idea sounds generally good to me. We might also want a similar mechanism for the zip file problem. Since the .jj directory may be untrusted, it might be better to store trusted config contents or hashes in e.g. ~/.local directory.

#1595

I have no idea about TUI. I personally would want to just see the diff and accept/reject.

[matts1]

The idea sounds generally good to me. We might also want a similar mechanism for the zip file problem.

I'm not familiar with "the zip file problem". Could you link to the issue?

Since the .jj directory may be untrusted, it might be better to store trusted config contents or hashes in e.g. ~/.local directory.

I'm not opposed to that. I chose this method because it was the same directory that the repo config was stored. I would request, however, that we submit this first and worry about that later, since this is explicitly not making the security any worse than it already is.

I have no idea about TUI. I personally would want to just see the diff and accept/reject.

I do agree, as a user, that that's what I would want, but that's also relatively easy to achieve with the TUI. I think, however, that from a security perspective, a binary accept / reject choice is not great, because it might feel like you have to make a choice between new features and security. With a binary choice, if there's one particular line that you are concerned about and so you reject it, that means you have to reject every future change to the repo config.

[martinvonz]

I'm not familiar with "the zip file problem". Could you link to the issue?

I don't think we have a link, but it's a familiar problem from other VCSs (familiar to VCS developers). The problem is what to do about repo-level configs in .jj/.git/.hg. For example, an attacker can send you a zip file of a repo where the repo-level config has rm -rf / as diff tool. They might ask you to help them troubleshoot some problem with their repo, for example, and you might not expect that simply running jj/git/hg diff in the repo would be harmful.

[matts1]

Makes sense. So I assume the solution would be to store it out of the repo, and require users to run jj config edit --repo instead of something like what I currently do where I type vim .jj/repo/config.toml?

[martinvonz]

I think Yuya's proposal was to record trusted config contents somewhere in ~/.local. When you unpack a zip file and run a command in that repo, you'd get a warning saying that it has untrusted configs. Then you run some command to say which of the configs you approve.

I'm not sure if it's safe to trust configs at the key-value level or only at coarser granularity. Could there be some case where you say that you trust key1="value1" and key2="value2" but somehow the combination is still harmful? I hope not, and it feels like it should be safe.

[matts1]

I'm not sure if it's safe to trust configs at the key-value level or only at coarser granularity. Could there be some case where you say that you trust key1="value1" and key2="value2" but somehow the combination is still harmful? I hope not, and it feels like it should be safe.

AFAICT, the attack surfaces are, in order of danger:

• Mess with the repo
• Send private code
• Remote code execution

This is my analysis of the danger:

• Revsets: Minor nuisances
  • Can mess with the repo
  • Can send code, but generally only to existing remotes
  • Can't escape the repo
• New remotes:
  • Should be obvious to spot, so easy to mitigate
  • RCE possible via pulling code
  • Possible to send private code by accident
• Arbitrary command execution. Only two sources of this are (to the best of my knowledge):
  • jj fix tools
  • Command aliases to jj util exec

I think that it's possible to make one harmful thing and one alias to it, but I don't think it's possible to make two things that are independently safe but together harmful (it's technically possible with two arbitrary commands that execute files in the repo, but if they're doing that then they can already achieve this).

[PhilipMetzger]

Here's a bunch of actual review comments.

[ilyagr]

I think this is probably OK. My worries are:

• Is this too inflexible to be useful? For example, as the user jumps from commit to commit, the repo-managed config may change, and I don't see a way to decide automatically whether they want their config to change accordingly (even if we trusted the config; see my hash idea in the other comment for how we could).

• How easy or hard will it be for the users to miss something important in the diff interface?

Figuring this out might require real-world testing. I'd probably merge some version of this, but disable it by default and call this feature "experimental" for quite a bit after. This is to give us time to polish it, but also to see whether it's worth the complexity/security tradeoff in practice.

Most of my other comments are various ideas about how to polish this further, which only make sense once we have a consensus that we're going forward with this (It's not clear to me how close we are at the moment).

[matts1]

• Is this too inflexible to be useful? For example, as the user jumps from commit to commit, the repo-managed config may change, and I don't see a way to decide automatically whether they want their config to change accordingly (even if we trusted the config; see my hash idea in the other comment for how we could).

My thoughts were, when designing this:

• The simplest approach is to simply not have the config to change at all
• The config will need to be approved, so it should not change accordingly IMO
• The only other option I can see is to have it, instead of reading the config from disk, automatically track trunk().
  • This presents potential problems in the sense that a change could be breaking (eg. jj fix referencing a file format.py that appeared in the same commit as the config change).
  • We could do something akin to dynamic approval where we have a map from hash of config file on disk to approved config. This seems way too complex for the initial solution though.
• I figured that it was unlikely someone would jump from commit to commit across different configs, and even if they did they'd only get a warning
  • Maybe this is wrong. It might just be because my workflow involves every commit being rebased on top of trunk() whenever I sync. Maybe people who use long-term feature branches that eventually get merged into main might not agree that it's unlikely?

• How easy or hard will it be for the users to miss something important in the diff interface?

I think that if the user cares about security, it's pretty hard to miss. It is also relatively easy to just blindly approve, though.

Figuring this out might require real-world testing. I'd probably merge some version of this, but disable it by default and call this feature "experimental" for quite a bit after. This is to give us time to polish it, but also to see whether it's worth the complexity/security tradeoff in practice.

I wouldn't be opposed to putting this behind a flag. Since this is inherently pre config-load, do you know if this would be able to be in our config schema, or would it have to be something like an environment variable flag?

[matts1]

I've added a config setting to toggle this feature, as requested by @ilyagr.

Does anyone have opinions on what the config file should be? My candidates were:

• .jjconfig.toml
• jj/config.toml
• .config/jj.toml
• .config/jj/config.toml

I ended up settling on the last one for a few reasons:

• I saw someone opining somewhere (can't find it now though) that workspace root tends to get cluttered already, so it was probably better to put it under a subdirectory.
  • My two cents is that for ownership purposes, it's nice to be a directory rather than a file.
• .jj is taken by the repo itself, and the file should be hidden, so jj/config.toml is ruled out.
• .config seems like a good place to put it
• .config/jj.toml doesn't give much flexibility if we need to add more jj-related files later (eg. .jjignore, .jjattributes)
• Adding the directory .config/jj allows users to store both the jj config and any commands they reference in the same directory
  • Eg. alias.sync = ["jj", "util", "exec", ".config/jj/sync.py"]
  • Then you can have .config/jj/sync.py in the same directory, which makes things much easier to review from a security standpoint.
  • This also allows .config/jj/OWNERS to exist

That being said, I'd welcome other opinions.

[PhilipMetzger]

@matts1 How do you imagine this working with multiple "correct" upstreams, like Eidolon's which inspired #3670? It isn't clear to me what should happen then, since there are multiple options which available beside erroring out.

[matts1]

@matts1 How do you imagine this working with multiple "correct" upstreams, like Eidolon's which inspired #3670? It isn't clear to me what should happen then, since there are multiple options which available beside erroring out.

I think in general, my answer is:

• They probably have the same config, in which case there's no problem
• If they don't have the same config, you probably spend a lot more time on one branch than the other. Just use the config from that branch.
• Otherwise, they can disable this feature if they like (and potentially manually specify the config file on the command-line)

There is a potential solution to this, but it's definitely not within the scope of the MVP. We can create a mapping from hash of checked-in config -> approved config. Then when you switch between revisions, it would just use the previously approved config instead of warning you that it's out of date.

[joyously]

with people adding a jj util exec alias which blasts everything away.

What if repo configs don't allow aliases?

[matts1]

with people adding a jj util exec alias which blasts everything away.

What if repo configs don't allow aliases?

That wouldn't really solve the problem. You could achieve the same just as easily with a malicious jj fix command, which definitely should be in the repo config, since each repo has their own rules for formatting.

Also, I think that would somewhat defeat the point of the repo config. In chromium, for example:

• The process to sync is:
  1. Run a git fetch
  2. jj rebase onto trunk()
  3. Run gclient sync, which updates the git submodules
• The process to upload (with git) is to run git cl upload, which:
  1. Runs a formatter
  2. Runs pre-upload checks
  3. Uploads to gerrit

While the former might be (somewhat) unique to chromium, I think that an alias jj pre-upload would be pretty reasonable and standard, and I've been running python scripts in lieu of hooks to run the equivalent of jj fix && jj pre-upload && jj git push.

So TLDR, I think they have too much value to disable. I think one approach that could be taken though is some stricter security measures. For example:

• We could execute jj fix in a read-only sandbox so it can't write (or delete) files
• We could, by default, make jj util exec run in a read-only sandbox using overlayfs or similar, with flags such as:
  • --no-sandbox: completely disable the sandbox
  • --allow-repo-writes: allows writes to all files in the repo excluding .jj
  • --allow-global-writes: allows writes to any file in the filesystem
• We could sandbox file reads to, by default, only allow:
  • entries in PATH
  • system directories
  • files in repo
  • $HOME/.*/** (eg. ~/.config/..., ~/.bazelrc

In general, sandboxing reads is a lot more painful, and I'm not sure it's a good idea, but sandboxing writes does sound like a very good idea.

[matts1]

Since the .jj directory may be untrusted, it might be better to store trusted config contents or hashes in e.g. ~/.local directory.

This seems like the right direction to me. @matts1: Are you willing to work on the feature in that form?

That feels like it's better as a follow-up PR, especially since I've disabled it with a feature flag by default. I think that the local directory is probably better in the long term, but in the short term I think there's nontrivial decisions that need to be made.

• A trusted config hash for one repo should not be trusted for another, since paths are relative to the repo
  • Doing so would introduce an attack surface where you copy a config file with a relative path, and add your own malicious file at that path. For example, copy chromium's config.toml and have your own malicious tools/jj/sync.py
• Is it based on the local path on disk?
  • What happens if you delete a repo and put another repo at the same location on disk? That new repo inherits the trust of the old repo
  • Maybe this is fine?
• Is it based on the remotes? That could have the nice property that if I have multiple checkouts of the same repo I only need to approve them once, but how do you check that two repos are the same? Probably based on remotes or something?
  • What happens with multiple remotes?

Btw, the PR title is confusing to me. We already support repo-level configs. What's the "user+repo level" about?

Sorry, my original PR renamed repo configs to user-repo configs, and called this "repo configs", but I didn't like that so I changed it. Updated the title to make it more clear.

What should be the precedence of the various configs?

Currently repo-managed configs are between the global user config and the repo's config.toml

What happens when checking out a revision with different repo-level config?

Once config has been "approved", I think that should be remembered between checking out different revisions. For example, suppose these commits change the repo-managed config:

This is the current behaviour, with the hope that in future versions we'll store a map from checked in config -> approved config instead of just remembering the most recently approved version.

Where is approval information stored?

• Approvals can't be stored in in the .jj directory because of the zip file problem.

As stated above, I think that for now we should store it in the .jj directory, and move it to .local once we answer some of the more difficult questions.

What exactly is stored when config is approved?

• Rather than copying the whole repo-managed config file here, we could probably just store the SHA-256 of every approved version?

I think we would want to store:

• filename is the hash of the content stored in version control
• contents is the actual content that was approved (since you can approve or reject it line by line)

• What if the user does mv ~/A ~/C? Is the approval forgotten?

I don't think we should worry about this use case. It's not frequent enough to be a concern.

Miscellaneous

• Different platforms probably need different tool config (sigh, Windows). I would want this to support --scope blocks so I could, eventually, have --when.platform = ["windows"] to create per-platform config.

Yeah, I was looking at this for chromium's fix tools. We have clang-format and clang-format.exe respectively. I ended up calling clang-format.py which chose the correct clang-format based on the platform.

[martinvonz]

A trusted config hash for one repo should not be trusted for another, since paths are relative to the repo

• Doing so would introduce an attack surface where you copy a config file with a relative path, and add your own malicious file at that path. For example, copy chromium's config.toml and have your own malicious tools/jj/sync.py

That's a very good point! I don't think we had considered that the config would contain repo-relative paths. I guess we'll have to think more about this.

Sorry, that's all I have time to reply about right now.

[emilazy]

Just wanted to share some thoughts; I talked about some of this on the Discord before but wanted to get it posted here and expanded upon.

That feels like it's better as a follow-up PR, especially since I've disabled it with a feature flag by default. I think that the local directory is probably better in the long term, but in the short term I think there's nontrivial decisions that need to be made.

I agree that since .jj/repo/config.toml is already trusted, it’s fine to store trust status in .jj for now – the threat model does not change.

I do also think that we should make .jj not trusted in this way; that unpacking a repository in a .zip file and running a Jujutsu command inside it should not be able to achieve arbitrary code execution. So I want to explicitly disclaim that none of this should be a blocker for the current PR, but since I think the current behaviour (before your PR) is a security issue and it’s very much worth thinking about as part of the design for this stuff, I hope you don’t mind me talking about the points raised a little more :)

I think we would want to store:

• filename is the hash of the content stored in version control
• contents is the actual content that was approved (since you can approve or reject it line by line)

• A trusted config hash for one repo should not be trusted for another, since paths are relative to the repo

  • Doing so would introduce an attack surface where you copy a config file with a relative path, and add your own malicious file at that path. For example, copy chromium's config.toml and have your own malicious tools/jj/sync.py

(The following treats configuration files in the working copy as monolithic things to accept or reject in their entirety, since I’m not sure the diff‐selecting interface makes sense if we move from just copying stuff into .jj. But it can be adapted as appropriate for partial approval.)

I agree that it makes sense to key trust on the repository, because configuration that you are okay with for one repository may not be okay for another. (For instance, perhaps the origin remote is a trusted internal repository in one, but in another, origin is the FOSS upstream and fork is the internal repository. You are happy agreeing to push things to origin in the former, but the latter would result in leaks. Or consider private commit configuration, and so on.)

I am not sure to what extent it makes sense to include the other contents of the working copy in the threat model, however:

1. If it’s something like rustfmt.toml being passed as a configuration file, then the trust we’re expressing is “it’s okay to run rustfmt”, and it’s rustfmt’s responsibility to ensure that a malicious rustfmt.toml can’t do anything evil. If you’re worried about that and don’t trust rustfmt, then you should not approve the configuration that invokes it. In other words, we’re the first line of defence against executing things that shouldn’t be executed at all, but beyond that we have to pass the responsibility for safety on to the things we invoke.

2. The reason to have approval for in‐working‐copy configuration is that we may switch to malicious commits. Therefore, it makes no sense to want to approve each configuration change individually in case it comes from switching to a malicious commit for review, but not worry about the fact that the malicious commit could simply put the malicious code in tools/jj/sync.py. This kind of threat applies within a single repository, therefore, not just when transferring configuration across repositories.

   In other words: if you’re trying to mitigate “developers getting owned” risks, then you don’t want to directly invoke arbitrary code as part of the workflow, and if the user approves a configuration that does that, they’re accepting the risk that involves. In an ideal world, we could prompt the user for re‐approval when tools/jj/sync.py changes. But we do not have a reliable, practical method to pin hashes of every single file or bit of state that could affect an arbitrary command inside the configuration. (Unless we make people use Nix.)

   One way to mitigate this is to have an explicit pinned revision for tooling, i.e. doing something like python3 <(jj file show -r abcdef tools/jj/sync.py) (or -r main@upstream) in the alias configuration. Then the user will be prompted if the tooling will potentially‐untrusted source. Of course, the tooling must then be careful not to trust the working copy. Nixpkgs does something like this, where an explicit revision of the repository is pinned inside Nixpkgs itself to obtain CI tooling from, though mostly for unrelated reasons.

Therefore, I believe that our principle should be that changes to the working copy other than to the configuration file are out of scope for the threat model, and that users approving a configuration accept the risk of trusting the invoked tooling to handle it safely, or trusting arbitrary future states of the working copy itself if the approved configuration implicitly hands it privileges. (The latter does not necessarily make the feature useless, in the presence of a trusted remote enforcing ACLs or similar.)

• Is it based on the local path on disk?

  • What happens if you delete a repo and put another repo at the same location on disk? That new repo inherits the trust of the old repo
  • Maybe this is fine?

• Is it based on the remotes? That could have the nice property that if I have multiple checkouts of the same repo I only need to approve them once, but how do you check that two repos are the same? Probably based on remotes or something?

  • What happens with multiple remotes?

FWIW, the direnv tool has been in the business of executing arbitrary code to set up development environments when you cd into a directory for a long time now. The model it uses is that trust is conditioned on the combination of the path to the .envrc and the hash of its contents. The same .envrc at two different paths isn’t necessarily trusted, because it can be (and often is) as simple as “run this arbitrary code from another file the same directory to get environment variables”, and an .envrc at a trusted path that changes needs re‐approval. Explicit denials (to stop it notifying about an untrusted .envrc) block the path itself regardless of contents.

I think this is a broadly reasonable solution, especially if approvals for paths that no longer exist are automatically pruned. However, for Jujutsu, I wonder if we can’t do something that is both simpler and more effective, given our position as the VCS: why don’t we key approvals on, say, the hash of the @ operation? (Or maybe the commit hash, but I feel less confident about that, especially in the presence of remote configuration and Git subtrees and so on.)

That should close over the hashes of basically everything we could care about that contributes to the state of the configuration. We might want to throw in the workspace path too just in case you do something wonky with ../, but in general I feel good about reusing an existing mechanism here rather than trying to manually throw a bunch of stuff into the hash.

I think it’s best to err on the side of requiring re‐approval, because it’s only a mild annoyance unless you are cloning entirely new copies of a repository constantly, but being too lax is a CVE.

As stated above, I think that for now we should store it in the .jj directory, and move it to .local once we answer some of the more difficult questions.

FWIW, my proposal for .jj/repo/config.toml when we move approvals to $XDG_DATA_HOME (or $XDG_STATE_HOME?) would be to extend the same trust mechanism to the configuration in there, and have jj git init, jj git clone, and jj config set --repo automatically update the approved revision. That would cover the “unzipping a malicious repository” risk, at the cost of having to approve configuration changes that are made from outside of Jujutsu, while still allowing repositories to be reasonably portable.

(We could also of course have a list of configuration options that are guaranteed to be safe to read from untrusted sources to reduce the prompting here, but it’s risky business if you haven’t architected your whole program for paranoia from the ground up; seemingly benign values can end up load‐bearing, especially in the presence of user configuration that looks at things like remotes.)

[joyously]

Just an opinion: it's oh so much easier to avoid the risk when the tool never loads config from the repo it is managing. And that applies to all VCS, not just jj.
If you implement this, wouldn't it affect commands that extract historical commits with a different config? And some of those commands you want to run unattended?
If you don't implement this, can't a team document in their contributing.md how they want code formatted or whatever? Does the config of the VCS really belong in the repo? (I think it doesn't belong there.)

Edit: I forgot to mention, I was reading an article alleging the US 2024 election was manipulated, which tied back to a small approved change concerning the config file...that its hash was in the dynamic list instead of the static list to check for. This seemed related.

[matts1]

IMO, the config of a VCS doesn't need to load config from the repo it's managing. However, by choosing to implement several features, I believe that jj has chosen a route where it should.

I think that if a repo wants to tell a user how to format their code, that's fine. If a VCS wants to do formatting of code, that's also fine. But IMO you can't do both at the same time without allowing a repo to tell jj how to format their code. The same goes, in my opinion, for jj util exec aliases, which constitute workflows.

I strongly believe that adding this feature increases security, not decreases it. While you're right that this feature introduces potential security concerns, I think that by doing this you are ensuring that everyone does things in a relatively secure way. I think that if you do not add this feature, you will get things such as:

• Repos that instruct users to run ln -s <repo config> .jj/repo/config.toml (or cp), thus bypassing any review (this is what we currently recommend for chromium)
• Repos that instruct users to run a script to set up your jj config

To use a metaphor, a door with a lock on it might not be perfectly secure, but if you don't make a door, people are gonna smash a hole through the wall, and that's definitely not secure.

[joyously]

I think that by doing this you are ensuring that everyone does things in a relatively secure way.

You didn't address what happens with multiple versions of the config file that get surfaced when you switch to that version, or when there are both a user and a repo config in the repo, or the CI hitting this approval thingy (command that isn't supposed to be interactive suddenly is).
My own opinion is that a VCS shouldn't format code, only version it. It doesn't really need to have util exec either, but aliases are really handy (and are in several VCSs).

[matts1]

I've addressed these in earlier comments, but I'll reiterate:

What happens with multiple versions of the config file that get surfaced when you switch to that version

When you switch to a version, you simply get a warning that you're using a different version of the config file. You can keep using your current version (as mentioned in previous commits, this isn't ideal, but is a good MVP, and I mentioned in an earlier comment a strategy we could use to improve on this by using hashes).

or when there are both a user and a repo config in the repo

The order of application is (with the last overriding):

1. Builtins
2. User config (~/.config/jj/config.toml)
3. Managed repo config (this PR)
4. User's repo config (<repo>/.repo/config.toml)
5. Command-line

or the CI hitting this approval thingy (command that isn't supposed to be interactive suddenly is).

Nothing becomes interactive here. It simply outputs a warning that says "The config is out of date. We suggest updating the config by running jj config review-managed".

My own opinion is that a VCS shouldn't format code, only version it. It doesn't really need to have util exec either, but aliases are really handy (and are in several VCSs).

That's a fair opinion, and I think that if jj had chosen to go down that path, you'd probably be right that we don't need this, but I think that given the decisions that jj has made, this path is the right one.

[jennings]

What happens when checking out a revision with different repo-level config?

Once config has been "approved", I think that should be remembered between checking out different revisions. For example, suppose these commits change the repo-managed config:

This is the current behaviour, with the hope that in future versions we'll store a map from checked in config -> approved config instead of just remembering the most recently approved version.

That's actually the opposite of what I thought, but I now see how what I said was ambiguous :)

But as I think about it, I think only remembering "the most recently approved version" is probably a perfectly fine way to start. Repo-managed config shouldn't be changing often, so it isn't much of a problem in practice.

(I previously had here a suggestion about using the diff editor to review config changes. Then I realized there was prior art for that: The implementation of the PR I'm commenting on. Whoops.)

[matts1]

I was hoping we could get this reviewed and submitted soon.

I think there are some open questions, but they're a non-issue for this particular PR IMO:

• How do you manage switching between configs
  • While it's somewhat important, it doesn't affect the MVP
• Can we avoid the zip file problem
  • It's definitely possible, but there's not a single solution. There's many different options we could take. I don't want to block this CL on a discussion on that, since this CL does not degrade security (it uses an insecure method, but it's the same method we already use for existing config files).

[joyously]

Has anyone looked up the discussions on other VCS using config from the repo it is managing, or is the only one?

[yuja]

• Can we avoid the zip file problem
  • It's definitely possible, but there's not a single solution. There's many different options we could take. I don't want to block this CL on a discussion on that, since this CL does not degrade security (it uses an insecure method, but it's the same method we already use for existing config files).

I prefer to address this security concern because repo-managed config is new feature, but I'm not strongly against adding it as the current form if people agree.

[martinvonz]

I prefer to address this security concern because repo-managed config is new feature, but I'm not strongly against adding it as the current form if people agree.

I agree with this, but it sounded like we couldn't come up with a solution that works for both cases because of the problem with approving a config that points a jj fix tool path that's inside the repo (and similar problems with paths into the repo). I unfortunately don't think I'll find time to think much about this in the near future and I don't want to block this feature on that.