Merge rspec repos

[p8]

Proof of concept for merging the other repos into this repo while keeping all history (See: rspec/rspec-core#2509).
I had some success with the steps described here (https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804

mkdir rspec-mono
cd rspec-mono
git clone [email protected]:rspec/rspec.git
git clone [email protected]:rspec/rspec-core.git
git clone [email protected]:rspec/rspec-expectations.git
git clone [email protected]:rspec/rspec-mocks.git
cd rspec-core
mkdir rspec-core
git mv -k * rspec-core
git rm .gitignore
git rm .document
git commit -m 'Moving repo into its own subdirectory'
cd ..
cd rspec-expectations
mkdir rspec-expectations
git mv -k * rspec-expectations
git rm .gitignore
git rm .document
git rm .rspec
git rm .rubocop.yml
git rm .rubocop_rspec_base.yml
git rm .travis.yml
git rm .yardopts
git commit -m 'Moving repo into its own subdirectory'
cd ..
cd rspec-mocks
mkdir rspec-mocks
git mv -k * rspec-mocks
git rm .gitignore
git rm .document
git rm .rspec
git rm .rubocop.yml
git rm .rubocop_rspec_base.yml
git rm .travis.yml
git rm .yardopts
git commit -m 'Moving repo into its own subdirectory'
cd ..
cd rspec
git remote add rspec-core ../rspec-core
git remote add rspec-expectations ../rspec-expectations
git remote add rspec-mocks ../rspec-mocks
git fetch rspec-core
git fetch rspec-expectations
git fetch rspec-mocks
git co -b merge-rspec-repos
git merge --allow-unrelated-histories rspec-core/master
git merge --allow-unrelated-histories rspec-expectations/master
git merge --allow-unrelated-histories rspec-mocks/master
git commit -m 'Import sub repos'

[p8]

@pirj Sorry for the late reply.
I was indeed under the presumption that everything was to be merged in a single gem.
I've changed the script to keep repo's as separate gems in separate folders.
So the folder structure is now similar to the mono-repo-prototype:

mono_repo/
mono_repo/rspec/
mono_repo/rspec/README.md
mono_repo/rspec/lib/
mono_repo/rspec/...
mono_repo/rspec-core
mono_repo/rspec-core/README.md
mono_repo/rspec-core/lib/
mono_repo/rspec-core/...
...

It can now also create branches for different versions.

#!/usr/bin/env ruby

working_dir = 'working'

Repo = Struct.new(:name, :url, :branch)
repos = [
  Repo.new('rspec',              '[email protected]:rspec/rspec.git'),
  Repo.new('rspec-core',         '[email protected]:rspec/rspec-core.git'),
  Repo.new('rspec-expectations', '[email protected]:rspec/rspec-expectations.git'),
  Repo.new('rspec-mocks',        '[email protected]:rspec/rspec-mocks.git'),
  Repo.new('rspec-support',      '[email protected]:rspec/rspec-support.git')
]

# merge everything into the rspec-monorepo-prototype repo for now
mono_repo = Repo.new('rspec-mono', '[email protected]:rspec/rspec-monorepo-prototype.git')

# Clone the mono repo
%x(
  mkdir #{working_dir}
  cd #{working_dir}
  git clone #{mono_repo.url} #{mono_repo.name}
)

# Merge sub repos into the mono repo while keeping the commit history.
# 1bf79d2bf is the initial commit of the mono-repo without the sub repos
# to prevent merge conflicts
%w[v3.8.0 v3.9.0].each do |branch|
  %x(
    cd #{working_dir}
    cd #{mono_repo.name}
    git co 1bf79d2bf
    git co -b #{branch}
  )
  repos.each do |repo|
    repo.branch = branch
    # Check out the sub repo and move all files to a sub directory
    # with the same name as the repo.
    # Merge the sub repo into the mono repo while keeping history.
    %x(
      cd #{working_dir}
      rm -rf #{repo.name}
      git clone #{repo.url} #{repo.name}
      cd #{repo.name}
      mkdir #{repo.name}
      git fetch --tags
      git co -b #{repo.branch} #{repo.branch}
      git mv -k * #{repo.name}
      git mv -k {.[!.]*,..?*} #{repo.name}
      git commit -m 'Moving #{repo.name} into its own subdirectory'
      cd ../#{mono_repo.name}
      git remote add #{repo.name} ../#{repo.name}
      git fetch #{repo.name}
      git merge --allow-unrelated-histories --ff #{repo.name}/#{repo.branch}
    )
    # rspec-expectations and rspec-mocks both moved License.txt to LICENSE.md.
    # Resolve this merge conflict by removing License.txt and adding the
    # LICENSE.md for both sub repos.
    %x(
      cd #{working_dir}
      cd #{mono_repo.name}
      git rm License.txt
      git add rspec-expectations/LICENSE.md
      git add rspec-mocks/LICENSE.md
      git commit -m 'Fix merge conflict'
    )
  end
end

[p8]

I'll see if I can make a PR for the mono-repo-prototype and get the build running.

[pirj]

Thanks for pushing this forward!

[p8]

I can now run the specs for all sub repo's in the rspec-monorepo-protoype:
rspec/rspec-monorepo-prototype#1
Some specs are failing though...

[p8]

Hmm, weird I'm not seeing Travis in the checks anymore.
Here is a link to a job that ran the specs: https://travis-ci.org/rspec/rspec-monorepo-prototype/jobs/643931521

[pirj]

Wow, nice, good job!
I actually see a link to a build job in the checks. https://travis-ci.org/rspec/rspec-monorepo-prototype/builds/643953326

[p8]

Yes, the problem was that my branch could not merge to master.
I now have some green jobs on my fork which used the script mentioned above:
https://travis-ci.org/p8/rspec-monorepo-prototype/builds/643957490

[pirj]

script/update_rubygems_and_install_bundler: line 8: is_ruby_23_plus: command not found

I guess the problem to the red builds might be this little thing.

[p8]

Yes, setting the dist to trusty fixes it on travis:
https://github.com/rspec/rspec-monorepo-prototype/pull/1/files

[pirj]

1.1) Failure/Error: let(:file_1) { File.open(File.join("tmp", "file_1"), "w").tap { |f| f.sync = true } }
          Errno::ENOENT:
            No such file or directory @ rb_sysopen - tmp/file_1

Maybe tmp is not present?

[pirj]

@p8 Have you had a chance to take a look at the tmp issue?

[p8]

@pirj I'll try to have a look this week.

[p8]

@pirj The script now merges the repo's, adds the travis config, and runs script/run_build in every subrepo. So Rubocop, Cucumber and other checks are run as well.
https://gist.github.com/p8/33563f7378376218a9ce078578b6c095

And the build is green: https://travis-ci.org/github/p8/rspec-monorepo-prototype/builds/679697190

[pirj]

Awesome!

[JonRowe]

So I've created a branch called setup on https://github.com/rspec/rspec-monorepo-prototype which contains this script and have run it to create master-20200503

Some thoughts I have so far are that, we are going to lose a large amount of our history in the form of PR references and overlaps, but I'm not sure how we can do much about that.

We're also going to need to work on all of our tags to get them into line, otherwise they will overwrite one another...

We'd also need to transfer all open issues from the other repos to the monorepo.

I've reviewed @p8's build and it works but would need tweaking / optimising afterwards, specifically I think we might actually want to break out the individual specs in a matrix, maybe this would also be a good opportunity to use Github actions, but my branches seem not to want to run for some reason

[pirj]

Nice!

work on all of our tags

I guess we can add a repo prefix to tags for all repos before merging them.

lose a large amount of our history in the form of PR references

That might be solvable, but I don't have a good idea how exactly. Before hub existed I used a trick to also fetch PR branches by tweaking the local repo's .git/config. Fetching looks like:

$ git fetch origin
From github.com:joyent/node
 * [new ref]         refs/pull/1000/head -> origin/pr/1000
 * [new ref]         refs/pull/1002/head -> origin/pr/1002

And even though those pull request won't exist in the resulting repo, the pull request heads will.
Maybe also prefix them as refs/pull/core-1000 refs/pull/expectations-1002 to avoid clashing?

It makes sense to add those two tweaks to the merge script.

@p8 Good job!

[JonRowe]

Its more that we have commits which reference the pull request number, which would not exist in the new repo, but the old repos will continue to exist, we'll just archive them

[benoittgt]

maybe this would also be a good opportunity to use Github action

I can start again my investigation on this. :)

For the tags, if we choose to prefix them. I am wondering how will manage the github release page

[JonRowe]

This is how rails handles it https://github.com/rails/rails/releases They just release all versions in lock step, maybe thats easier, we could adopt that from rspec 4

[benoittgt]

This is how rails handles it https://github.com/rails/rails/releases They just release all versions in lock step, maybe thats easier, we could adopt that from rspec 4

I am in favor of this.

[pirj]

all versions in lock step, we could adopt that from rspec 4
All for doing it this way 👍

@p8 I can take it from here, add tag prefixes and keep PR references, if you don't want to continue.

[p8]

@pirj Sure, glad I could help :)

[pirj]

Just in case https://github.com/tj/git-extras/blob/master/Commands.md#git-merge-repo

[eregon]

It'd be great to get this over the line. TBH I feel it's a huge pain to work with so many git repos, issue trackers, separate projects for PRs, etc. This probably also drives potential contributors away, because to contribute to RSpec one needs a far more complicated setup than about any other gem.

[p8]

@eregon Making contributing easier was one of my motivations for working on this.
Currently the merging of the repo's is being worked on by @yujinakayama.
rspec/rspec-core#2509 (comment)

[JonRowe]

I'd love to merge the repos into one, but its something that will take time to do correctly and my intention was to coordinate this change with an RSpec 4 release, believe me its just as much of a maintainer headache as it is a contributor one, especially when it comes down to coordinating 4 releases of gems, but there is also a large time cost involved in merging the repos, verifying that everything is correct and then sorting out the history of github issues, that last part was my main reason for targeting a major version release, hopefully that would create a sort of minima of issues to transfer.

[eregon]

then sorting out the history of github issues, that last part was my main reason for targeting a major version release, hopefully that would create a sort of minima of issues to transfer.

Wouldn't the issues be copied to the monorepo (maybe with an appropriate tag) and then that would be mostly automatic, except for the script/GitHub Action doing that migration? If so I don't think it would change things much whether it's done at the same time than a RSpec 4 release, and it might even be helpful to not have to do both tasks around the same time.