A tool to clone efficiently all the repos in an organization
When installing from PyPi, use wr-cloner, cloner was already taken :sad:
pip install wr-cloner
Note: If using cloner after cloning the project and not a PyPi package, add "python" at the start. There's an example of that in the examples section.
Usage: cloner [OPTIONS] GITHUB_ORGANIZATION
A tool to clone efficiently all the repos in an organization.
Options:
--version Show the version and exit.
--token TEXT GitHub token to read private repos. This
parameter is needed when cloning from a
GitHub Enterprise server.
--ghe TEXT GitHub Enterprise URL. It needs the
GITHUB_ORGANIZATION parameter to clone repos
from there and the TOKEN option as well.
--threads INTEGER Number of threads and processes to use. For
maximum threads and processes on the system,
use '--max-threads' [default: 4]
--logging [ERROR|WARNING|INFO|DEBUG]
Logging level [default: INFO]
--path TEXT Sets a path where to clone the repositories
(eg: ./another/path/)
--git-options TEXT Add options to the clone command (eg: --git-
options "--depth 1"). By default, clones
quietly (--quiet).
--max-threads If declared, uses the maximum available
threads and processes in the system. As per
physical cores on the system cpu.
--ignore-archived If declared, will ignore archived repos when
cloning.
--help Show this message and exit.
# For github.com with 8 threads
cloner --threads 8 GITHUB_ORGANIZATION
# For github.com with the maximum threads on the system running
cloner --max-threads GITHUB_ORGANIZATION
# For GHE, default threads
cloner --ghe GHE_URL --token SUPER_SECURE_TOKEN GITHUB_ORGANIZATION
# Cloning with options
cloner --git-options "--depth 1" GITHUB_ORGANIZATION
# Install dependencies with
make install
# Adjust PYTHONPATH
export PYTHONPATH=$PYTHONPATH:.
# Use it
python cloner GITHUB_ORGANIZATION
Check the CONTRIBUTING.md file.
Follow the instructions in the SECURITY.md file.
Multithreading doesn't work to clone repos, since the os.system
call is 1 for each PID. The splitting is done with
multithreading, the cloning with multiprocessing. Same amount of threads and processes.