Add support for repositories with their own submodules

[richleigh]

When trying to use braid to manage a respository with its own submodules git cannot update the dependency's submodules.

E.g.

$ braid.sh add https://github.com/Valloric/YouCompleteMe
$ git submodule update --init --recursive
fatal: no submodule mapping found in .gitmodules for path 'YouCompleteMe/third_party/requests-futures'

Braid needs to find any .gitmodule files in the repository, recursively start tracking them, and check them out into the right location.

On update it would also need to refuse to change version for anything grandfathered in, and recursively get the appropriate versions based on the submodule requirements of the new parent version.

[realityforge]

That sounds like a neat idea but it is not something we use at this stage. Next time I am down in the code I will try to have a look at implementing it but it could be a while before that happens ;)

[mattmccutchen]

What is the exact proposed behavior? For a given revision of the upstream project, Braid should recursively retrieve submodules and flatten the content into a single tree, which serves as the upstream tree of the Braid mirror? This would add significant complexity to Braid that would have to be maintained over time, so if it were my decision, I wouldn't do it without stronger evidence of demand for the feature.

[richleigh]

I would imagine the only way to have it work would be to keep in sync with the submodules of a repository. I.e. peek inside the .gitmodule file, and get those versions. And so on recursively.

As it stands, the lack of a .git directory in the managed revision means it isn't possible to manually intervene and fetch the other components. In turn, this means that for any moderately complex project with it's own dependencies, braid can't be used.

[realityforge]

I guess that most people who vendor in external repositories choose either the submodule strategy or subtree strategy. Braid is wholly designed as an easier way to implement the subtree strategy.

I imagine it is possible to blend the two strategies but it would be a bit of effort to get right and maintain. I am not really motivated to do so given the last time I saw anyone use submodules was back in ~2011 sometime :)

Anyhoo - if you were interesting in implementing the feature and sufficient tests to keep it working over time I would not see any problem including this feature. I just can't see myself getting motivated to implement it anytime soon.

[mattmccutchen]

Re @richleigh:

As it stands, the lack of a .git directory in the managed revision means it isn't possible to manually intervene and fetch the other components. In turn, this means that for any moderately complex project with it's own dependencies, braid can't be used.

I realized this is not really true. When Braid copies the tree from the upstream repository, it doesn't do anything special about submodule links, so they come along by default; this is why git submodule update --init --recursive is failing rather than doing nothing. The problem is the missing configuration in the root .gitmodules file. In your example, you could add it based on the YouCompleteMe/.gitmodules file like this:

sed -e 's,^\[submodule ",&YouCompleteMe/,; s,^\tpath = ,&YouCompleteMe/,' YouCompleteMe/.gitmodules >>.gitmodules

Then git submodule update --init --recursive will succeed.

Of course, if you wanted to change all submodules recursively to be managed by Braid to avoid the hassles of submodules, the above technique doesn't do that. You could maintain a fork of YouCompleteMe in which you manually replace each submodule with a Braid mirror (every time a submodule pointer changes, you would get a merge conflict and have to manually update the Braid mirror to the matching revision) and then add this fork to your main project using Braid. Obviously, this is a hassle, but it might be less of a hassle than having the submodules in your main project depending on your usage patterns.

So now I see two ways Braid could conceivably handle submodules in a Braid mirror, besides the current naive behavior (which requires manual intervention to use the submodules in any way and blocks git submodule update --init unless you delete the submodules or exclude them from the git submodule update --init command):

• Propagate changes from the Braid mirror's .gitmodules file to the root .gitmodules file (like the sed command above, but handling updates too) so that the submodules can be initialized and used as such.
• Flatten submodules, as I think you originally wanted.

I'm still not convinced there's enough demand for either to justify the complexity in Braid.

[YOUR1]

Hopefully this feature is still to be added, since submodules are still a thing - and most of big projects use them a lot.

[mattmccutchen]

As I said above, I have serious concerns about the amount of complexity that would have to be added to Braid to support (in some form) subprojects that have their own submodules. To the extent that it's my decision, if people want such a feature, what would help convince me would be an explanation of a specific workflow you actually want to perform on a specific real-world project for which using Braid on a subproject with submodules is the best solution.

More detail about a proposed implementation (or even draft code) would also be helpful. Given the very limited maintenance energy available to Braid at this time (basically a few hours a month from me, with a number of higher-priority issues competing with this one), I'm wary of adding a bunch of code that does things in a specific way and then getting a bunch of follow-up issue reports that it doesn't quite work in another user's slightly different scenario. It might make sense for most of the logic for either generating the flattened mirror content or updating the main .gitmodules (depending on which way the feature works) to live in a separate script that would interface with Braid somehow but could be edited by the user to accommodate special requirements.