--blob-exec. Improved code originally developed by Paul Draper

[dmgerman]

hi Roberto,

I have improved the code by Paul Draper. The main issue is that his original code did not properly wait for the children to complete. This lead to a crash under linux (see #169).

I also added the ability to specify a regular expression to indicate the files that should be processed (my own bias, since I don't like file masks, as they are not
as powerful as regexps). I have been using this patch to process several large repos (including linux and git) and it seems to be working well.

The script to execute should read the file from stdin, and output to stdout. Two environment variables are used to communicate between BFG and the script: BFG_BLOB (contains the blobid of the file to process) and BFG_FILENAME (contains the basename, including extension) of the file to process.

If the following conditions happen, the blob in the commit is not replaced:

1. If the contents returned by the script are identical to the ones received, or
2. The script returns an error code != 0

This pull request is a superset of #83 and addresses #169.

Regarding licensing: my commits are licensed under the same license as BFG. Paul Draper submitted his commits in another pull request, who I presume, licensed the patch according to the contributing guidelines (https://github.com/rtyley/bfg-repo-cleaner/blob/master/CONTRIBUTING.md)

--dmg

[yyfearth]

Could you give some examples how to use it?

  --blob-exec:<key>=<value>
                           execute the system command for each blob

What is the key and what should be the value?

[dmgerman]

Wilson Young <[email protected]> writes:

Could you give some examples how to use it? --blob-exec:<key>=<value> execute the system command for each blob What is the key and what should be the value?

i didn't know how to add two different parameters to the command line, so I hacked it.

…

--blob-exec:pathToScript=<regularExpressionOfFilesToProcess> The first is the script to run for each file blob. The second is the regular expression to determine if the script is run on the specific blob. The script is run if the regular expression matches its filename (no path included). Note that this script reads from stdin and outputs to stdout. It is expected to read other information from two environment variables: # BFG_BLOB: the id of the blob (a SHA) # BFG_FILENAME: basename of the file to process. it has an extension remember that the current implementation of BFG does not keep the full path of the file. hopefully this will make it easier to use it. Check github/cregit for use of this. There might be a bug in my patch that only allows the execution of bfg from the main directory of the repository, but I have not debugged it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- Daniel M. German "It is good to be alive" Malcolm http://turingmachine.org/ http://silvernegative.com/ dmg (at) uvic (dot) ca replace (at) with @ and (dot) with .

[yyfearth]

@dmgerman I tried your RP with conflicts resolved with latest master.
It getting compiled, and all other functions works, but the added blobexec cannot getting though:

bfg --blob-exec:clean=js$ connector.git
command clean mask js$
Error: Option --blob-exec failed when given 'clean=js$'. None.get

[dmgerman]

Wilson Young <[email protected]> writes:

@dmgerman I tried your RP with conflicts resolved with latest master. It getting compiled, and all other functions works, but the added blobexec cannot getting though: bfg --blob-exec:clean=js$ cloud-service-connector.git command clean mask js$ Error: Option --blob-exec failed when given 'clean=js$'. None.get

in the command line? you might need to escape the $ . Try bfg --blob-exec:clean='js$' cloud-service-connector.git and let me know if that works. $

[yyfearth]

@dmgerman No difference, I tried both without $ or quoted. Always None.get

[dmgerman]

I just tried. I think the bug has to do with running the command outside the root git directory: dmg@iodine:/tmp|⇒ java -jar /home/dmg/git.hacking.mine/bfg-repo-cleaner/jars/bfg.jar '--blob-exec:/usr/bin/head=\.c$' xournal command /usr/bin/head mask \.c$ Error: Option --blob-exec failed when given '/usr/bin/head=\.c$'. None.get but if you run it in the root directory it would work: dmg@iodine:/tmp/xournal|master ⇒ java -jar /home/dmg/git.hacking.mine/bfg-repo-cleaner/jars/bfg.jar '--blob-exec:/usr/bin/head=\.c$' . command /usr/bin/head mask \.c$ command /usr/bin/head mask \.c$ Using repo : /tmp/xournal/./.git Found 125 objects to protect Found 27 commit-pointing refs : HEAD, refs/heads/master, refs/remotes/origin/HEAD, ... Protected commits -----------------

[yyfearth]

@dmgerman The command works on git dir, but got another error:

Cleaning commits:        59% (161/272)Exception in thread "Thread-1686" java.io.IOException: Broken pipe
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:355)
	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
	at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
	at com.madgag.git.bfg.cleaner.BlobExecModifier.writeJob$1(BlobExecModifier.scala:44)
	at com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4(BlobExecModifier.scala:49)
	at com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4$adapted(BlobExecModifier.scala:49)
	at scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$2(ProcessBuilderImpl.scala:75)
	at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)

Any clue?

[dmgerman]

Wilson Young <[email protected]> writes:

@dmgerman The command works on git dir, but got another error: ``` Cleaning commits: 59% (161/272)Exception in thread "Thread-1686" java.io.IOException: Broken pipe at java.base/java.io.FileOutputStream.writeBytes(Native Method) at java.base/java.io.FileOutputStream.write(FileOutputStream.java:355) at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123) at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108) at com.madgag.git.bfg.cleaner.BlobExecModifier.writeJob$1(BlobExecModifier.scala:44) at com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4(BlobExecModifier.scala:49) at com.madgag.git.bfg.cleaner.BlobExecModifier.$anonfun$execute$4$adapted(BlobExecModifier.scala:49) at scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$2(ProcessBuilderImpl.scala:75) at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23) ``` Any clue?

I might have seen this before... it might be that for certain blobs your script is failing. I would first try it with a very narrow regular expression (perhaps matching one file) and see if it succeeds.

…

---dmg

[vlsi]

Note: blobexec command is nice, however it might be complicated to make the command reproducible across various operating systems.

For instance. Suppose one runs pngquant to optimize png files across all commits.
Then different versions of the tool might result in different outputs. That would be really bad since it would basically make all the forks unusable (everybody would have to clone from the new repo and manually reapply the changes).

I have implemented "blobexec" + store the result of processing, so the results could be reproduced even without running the command.
The release can be found is here: https://github.com/vlsi/bfg-repo-cleaner/releases/
A sample use can be found here (see blobExecCache/pngoptimizer subfolder): https://github.com/vlsi/jmeter-git-cleanup/blob/v2.2.0/clean_fork