Skip to content

Update Python Requirements

Jump to bottom
Andy Neff edited this page Dec 5, 2016 · 2 revisions

Summary

This script helps track all python dependencies for a project, so no version is left uncontrolled. It automated the process of "checking every package" that needs to be updated, updating the version number, and adding new derived dependencies and removing old dependencies

TL;DR

Example requirements_my_file.txt

numpy

Example requirements_my_file_custom.txt

certbot==0.8.1
https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/winpdb/winpdb-1.4.8.tar.gz

To run, simply

cd {my_requirements_dir}
files2=(my_file)
{vsi_common}/linux/update_python_requirements.bsh
...
### my_file ###
numpy==1.11.2
(Hint: You can edit the files before saying yes)
Do these requirements look ok? (y) y
...

And your requirements files are re-written and updated

requirements_my_file.txt

numpy==1.11.2

requirements_my_file_dervived.txt

certbot==0.8.1
acme==0.8.1
ConfigArgParse==0.11.0
configobj==5.0.6
cryptography==1.6
parsedatetime==2.1
psutil==5.0.0
PyOpenSSL==16.2.0
pyrfc3339==1.0
python2-pythondialog==3.4.0
pytz==2016.7
setuptools==30.2.0
six==1.10.0
zope.component==4.3.0
zope.interface==4.3.2
mock==2.0.0
ndg-httpsclient==0.4.2
pyasn1==0.1.9
requests==2.12.3
idna==2.1
enum34==1.1.6
ipaddress==1.0.17
cffi==1.9.1
zope.event==4.2.0
funcsigs==1.0.2
pbr==1.10.0
pycparser==2.17
numpy==1.11.2

Note: The custom file is never updated, and should be used for all packages you don't want updated or are not on pypi

#Environment variables

  • PIP2 - The pip used for updating python2. Change this for different docker/virtualenvs
  • PIP3 - The pip used for updating python2. Change this for different docker/virtualenvs
  • PIP_BINARY_ONLY - Currently packages that are not available for source download break the script, unless they are added to this comma separated variable. For example entrypoints,bash_kernel
  • files2 - Bash array list of python2 files to be processed. Pattern is
    • requirements_{name}.txt - Input file listing library and versions to use. These are libraries you want. IF you purposefully use a dependency, it should probably be added to this list, as it might be removed in the future. The [] notation is supported, for example requests[security]
    • requirements_{name}_custom.txt - List of all non-pypi packages and any pypi packages you don't want updated. These packages will all be checked for pypi dependencies. For non-pypi dependencies, you should add them yourself to the custom file
    • requirements_{name}_derived.txt - The output file. This is the file you should use for pip install, e.g. pip install -r requirements_{name}_derived.txt
  • files3 - Same as files2, but for python3

Functions

  • update_python_requirements - Main do everything function based on PIP2, PIP3, files2, and files3 values
  • update_main_requirements {file_name} - Update the main files. E.g. update_main_requirements /opt/test/requirements_project.txt
  • update_derived_requirements {main_filename} {custom_filename} {derived_filename} - Updates the derived requirements file. E.g. update_derived_requirements /opt/test/requirements_project.txt /opt/test/requirements_project_custom.txt /opt/test/requirements_project_derived.txt
  • update_requirements {input_filename} {output_filename} [other optional arguments] - The base update file function. The other optional arguments are passed to the pip command
  • get_packages {pip arguments} - Gets all the latest version numbers for the pip arguments specified. This can be a single package or a requirements file. Uses the pip in the environment variable PIP, pip2 by default
  • write_requirements {output_filename} {package_name}@{package_version} - Write package info to file. E.g. write_requirements /opt/project/requirements_project.txt [email protected]

Advanced Use Script

#!/usr/bin/env bash

set -eu

if [ "$(id -u)" == "0" ]; then
  apt-get update
  apt-get install -y ca-certificates wget --no-install-recommends
  pip install pip==9.0.1
  #This script works specifically with this version, no idea if it's forwards compatible
  groupadd user -g ${GROUP_ID} -o && \
  useradd -u ${USER_ID} -o --create-home --home-dir /home/user -g user user && \
  exec gosu user "$0" "${@}"
fi

source ${VIP_VSI_DIR}/linux/update_python_requirements.bsh

#Thanks cffi for this :(
if ! command -v x86_64-linux-gnu-gcc > /dev/null; then
  cp /bin/true /tmp/x86_64-linux-gnu-gcc
  export PATH="${PATH}:/tmp"
fi

#comma separated list of binary only packages
PIP_BINARY_ONLY="entrypoints,bash_kernel"

files2=(common flower uwsgi nginx notebook_1_2)
files3=(notebook_1_3 notebook_2)

cd "${VIP_PROJECT_DIR}/docker"

update_python_requirements
Clone this wiki locally