Issue with git archive on Ubuntu 16.04

[petems]

Description of problem

• What did you do?
  Tried to run an octocatalog-diff between two branches with a custom plugin (https://github.com/dylanratcliffe/onceover-octocatalog-diff)
• What happened?

+ bundle exec onceover run diff --from production --to add_jenkinsfile
�[32mINFO�[0m	 -> Preparing environment for role::vanilla on CentOS-7.0-64
�[32mINFO�[0m	 -> Preparing environment for role::vanilla on Ubuntu-16.04-64
�[32mINFO�[0m	 -> Deploying Puppetfile for role::vanilla on CentOS-7.0-64
�[32mINFO�[0m	 -> Deploying Puppetfile for role::vanilla on Ubuntu-16.04-64
�[32mINFO�[0m	 -> Compiling catalogs for role::vanilla on Ubuntu-16.04-64
�[32mINFO�[0m	 -> Storing results for role::vanilla on Ubuntu-16.04-64
�[32mINFO�[0m	 -> Compiling catalogs for role::vanilla on CentOS-7.0-64
�[32mINFO�[0m	 -> Storing results for role::vanilla on CentOS-7.0-64
�[1mTest:�[0m role::vanilla on Ubuntu-16.04-64
�[1mExit:�[0m 1
�[1mStatus:�[0m �[31mfailed�[0m
�[1mErrors:�[0m
E, [2017-06-17T18:57:59.382768 #3955] ERROR -- : Git checkout error: Git archive production->/tmp/d20170617-3955-1cv1jfz failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
E, [2017-06-17T18:57:59.385227 #3959] ERROR -- : Git checkout error: Git archive add_jenkinsfile->/tmp/d20170617-3959-8519q7 failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
OctocatalogDiff::Errors::BootstrapError: Git archive production->/tmp/d20170617-3955-1cv1jfz failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:109:in `rescue in git_checkout'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:104:in `git_checkout'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:93:in `bootstrap_directory'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:130:in `bootstrap'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:141:in `build_catalog'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:64:in `build'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog.rb:59:in `build'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:237:in `build_catalog'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:in `call'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:in `execute'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:202:in `execute_task'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:117:in `block (2 levels) in run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:in `fork'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:in `block in run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `each'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `each_with_index'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:93:in `run_tasks'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:92:in `build_catalog_parallelizer'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:29:in `catalogs'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1/catalog-diff.rb:34:in `catalog_diff'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1.rb:19:in `catalog_diff'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/cli.rb:112:in `cli'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/bin/octocatalog-diff:34:in `<top (required)>'
  /usr/local/bundle/bin/octocatalog-diff:17:in `load'
  /usr/local/bundle/bin/octocatalog-diff:17:in `<top (required)>'


�[1mTest:�[0m role::vanilla on CentOS-7.0-64
�[1mExit:�[0m 1
�[1mStatus:�[0m �[31mfailed�[0m
�[1mErrors:�[0m
E, [2017-06-17T18:58:01.369349 #4056] ERROR -- : Git checkout error: Git archive add_jenkinsfile->/tmp/d20170617-4056-13m9i4l failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
E, [2017-06-17T18:58:01.369968 #4053] ERROR -- : Git checkout error: Git archive production->/tmp/d20170617-4053-qxyqqv failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
OctocatalogDiff::Errors::BootstrapError: Git archive add_jenkinsfile->/tmp/d20170617-4056-13m9i4l failed: STDOUT:
STDERR:
  fatal: Not a valid object name
  tar: This does not look like a tar archive
  tar: Exiting with failure status due to previous errors
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:109:in `rescue in git_checkout'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:104:in `git_checkout'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog-util/bootstrap.rb:93:in `bootstrap_directory'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:130:in `bootstrap'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:141:in `build_catalog'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog/computed.rb:64:in `build'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/catalog.rb:59:in `build'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:237:in `build_catalog'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:in `call'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:33:in `execute'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:202:in `execute_task'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:117:in `block (2 levels) in run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:in `fork'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:116:in `block in run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `each'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `each_with_index'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:113:in `run_tasks_parallel'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/parallel.rb:93:in `run_tasks'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:92:in `build_catalog_parallelizer'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/util/catalogs.rb:29:in `catalogs'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1/catalog-diff.rb:34:in `catalog_diff'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/api/v1.rb:19:in `catalog_diff'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/lib/octocatalog-diff/cli.rb:112:in `cli'
  /usr/local/bundle/gems/octocatalog-diff-1.3.0/bin/octocatalog-diff:34:in `<top (required)>'
  /usr/local/bundle/bin/octocatalog-diff:17:in `load'
  /usr/local/bundle/bin/octocatalog-diff:17:in `<top (required)>'

• What did you expect to happen?
  Comparison between to branches to work.

• How can someone reproduce the problem?
  For context, I'm using a custom plugin that runs an r10k run of a Puppetfile in a temporary folder, then runs an octocatalog-diff between the two runs.

• Command used and debugging output
  The actual octocatalog-diff command is this:

bundle exec octocatalog-diff --fact-file /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v/spec/factsets/Ubuntu-16.04-64.yaml --from production --to add_jenkinsfile --basedir /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v --puppet-binary /var/lib/jenkins/workspace/ontrol-repo_add_jenkinsfile-YF67IPOJSZEA4N7KRTNFGNR644LHR2V3YPYE6BSAEQYMBAKBXGLQ/vendor/bundle/ruby/2.3.0/bin/puppet --bootstrap-script '/tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v/bootstrap_script.rb' --hiera-config /var/lib/jenkins/workspace/ontrol-repo_add_jenkinsfile-YF67IPOJSZEA4N7KRTNFGNR644LHR2V3YPYE6BSAEQYMBAKBXGLQ/spec/hiera.yaml --pass-env-vars BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT,RUBYLIB

Platform and version information

• Your OS: Ubuntu 16.04
• Your Ruby version: ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
• Your version of Puppet:

bundle exec puppet --version
4.7.1

• Your version of octocatalog-diff:

bundle exec octocatalog-diff --version
octocatalog-diff 1.2.0

Do the tests pass from a clean checkout?
Yes

Anything else to add that you think will be helpful?
The same command works fine on OSX, but on Ubuntu I get the git archive error

Working on OSX shown below:

bundle exec onceover run diff --from production --to add_jenkinsfile --debug --trace
DEBUG	 -> Reading /Users/petersouter/projects/psouter-control-repo/environment.conf
DEBUG	 -> Searching puppet code for roles and profiles
DEBUG	 -> Searching puppet code for roles and profiles
DEBUG	 -> Reading factsets
DEBUG	 -> Reading factsets
DEBUG	 -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-1opdrc5
DEBUG	 -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-1gjpxqa
DEBUG	 -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-vk4vtf
DEBUG	 -> Creating r10k cache for thread at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/r10k_cache20170618-24795-tsy6ty
INFO	 -> Preparing environment for role::vanilla on CentOS-7.0-64
INFO	 -> Preparing environment for role::vanilla on Ubuntu-16.04-64
DEBUG	 -> Creating temp directory
DEBUG	 -> Temp directory created at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo
DEBUG	 -> Copying controlrepo to /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo
DEBUG	 -> Creating temp directory
DEBUG	 -> Temp directory created at /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1
DEBUG	 -> Copying controlrepo to /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1
DEBUG	 -> Deploying vendored factsets
DEBUG	 -> Deploying vendored factsets
INFO	 -> Deploying Puppetfile for role::vanilla on Ubuntu-16.04-64
INFO	 -> Deploying Puppetfile for role::vanilla on CentOS-7.0-64
DEBUG	 -> Creating before script that overwrites site.pp
DEBUG	 -> Getting Puppet binary
DEBUG	 -> Running Octocatalog diff
INFO	 -> Compiling catalogs for role::vanilla on Ubuntu-16.04-64
DEBUG	 -> Running: bundle exec octocatalog-diff --fact-file /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1/spec/factsets/Ubuntu-16.04-64.yaml --from production --to add_jenkinsfile --basedir /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1 --puppet-binary /Users/petersouter/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/bin/puppet --bootstrap-script '/var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1/bootstrap_script.rb' --hiera-config /Users/petersouter/projects/psouter-control-repo/spec/hiera.yaml --pass-env-vars RUBYMOTION_ANDROID_SDK,RUBYMOTION_ANDROID_NDK,RUBYLIB,BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT
DEBUG	 -> Creating before script that overwrites site.pp
DEBUG	 -> Getting Puppet binary
DEBUG	 -> Running Octocatalog diff
INFO	 -> Compiling catalogs for role::vanilla on CentOS-7.0-64
DEBUG	 -> Running: bundle exec octocatalog-diff --fact-file /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo/spec/factsets/CentOS-7.0-64.yaml --from production --to add_jenkinsfile --basedir /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo --puppet-binary /Users/petersouter/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/bin/puppet --bootstrap-script '/var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo/bootstrap_script.rb' --hiera-config /Users/petersouter/projects/psouter-control-repo/spec/hiera.yaml --pass-env-vars RUBYMOTION_ANDROID_SDK,RUBYMOTION_ANDROID_NDK,RUBYLIB,BUNDLER_ORIG_PATH,BUNDLE_BIN_PATH,BUNDLE_GEMFILE,BUNDLER_VERSION,RUBYOPT
INFO	 -> Storing results for role::vanilla on CentOS-7.0-64
DEBUG	 -> Backing up modules to thread cache /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_CentOS-7.0-6420170618-24795-up8vmo
DEBUG	 -> Removing temporary build cache
INFO	 -> Storing results for role::vanilla on Ubuntu-16.04-64
DEBUG	 -> Backing up modules to thread cache /var/folders/nn/408ddhln26s1b356ry19q6yr0000gp/T/role__vanilla_on_Ubuntu-16.04-6420170618-24795-huqie1
DEBUG	 -> Removing temporary build cache
Test: role::vanilla on CentOS-7.0-64
Exit: 0
Status: no differences

Test: role::vanilla on Ubuntu-16.04-64
Exit: 0
Status: no differences

I know that the plugin isn't developed by you folk, but this error is really weird looking, and it's strange that the exact same commands work on OSX but no Ubuntu, so I was hoping you'd be able to help. I can give any debug information needed 😄

[kpaulisse]

fatal: Not a valid object name is an error message coming from git. That's the root of the problem here (that output is piped to tar, so the complaints from tar are downstream from whatever's wrong).

The code actually being run here is: https://github.com/github/octocatalog-diff/blob/master/scripts/git-extract/git-extract.sh

My suggestion: cd to the directory where this code actually lives, and run the git archive command that octocatalog-diff is actually running. From your example:

cd /tmp/role__vanilla_on_Ubuntu-16.04-6420170617-9058-1vp701v
git archive --format=tar add_jenkinsfile

Perhaps that will lead you to discover an issue, either with the repository itself, or with different command line arguments that are needed for the particular version of the git command needed for this platform. If there's something that needs to be fixed in git-extract.sh to make that work then I certainly welcome pull requests or suggestions on what you had to do to make the above command work in your environment.

[asenci]

FWIW, I had the same error using octocatalog-diff with GitHub Actions, turns out the "checkout" action only fetches the committed ref by default. Had to update my workflow to fetch both the source and target branches:

    steps:
      - name: Checkout target branch
        uses: actions/checkout@v2
        with:
          ref: ${{ github.base_ref }}
      - name: Checkout source branch
        run: |
          git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin ${{ github.head_ref }}
          git checkout --progress --force -B ${{ github.head_ref }} refs/remotes/origin/${{ github.head_ref }}