PackagingQuESt

Packaging QuESt

QuESt can be packaged in such a way to create an executable version. This version can be executed without the need for a Python installation. However, a solver is still required for solving optimization problems.

The packaging is handled by PyInstaller.

The general packaging process is as follows:

1. Activate the Python environment to be bundled.
2. Use a fresh clone/copy of the codebase.
3. Run PyInstaller, targeting the fresh clone.
4. Compress the resulting distribution into an archive.

The resulting executable is only for the operating system on which the packaging was performed. For example, if the packaging is done on a Windows 10 machine, the resulting .exe is only usable on Windows 10.

Setting up the Python environment

The Python environment in which the packaging process is performed is the environment that will be bundled with the executable. Therefore, the environment must be capable of running QuESt.

Corollary: Any installed packages in the environment will be bundled into the executable package. Therefore, in order to reduce the distributed package size, it is recommended to use a virtual environment with the bare minimum requirements for running QuESt in addition to the PyInstaller package.

Instructions for installing PyInstaller can be found here.

The conda spec list for the environment used to package QuESt 1.2.x executables for Windows 10 is shown below. This can be used in Anaconda to reproduce the environment as shown here.

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/win-64/blas-1.0-mkl.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/ca-certificates-2019.11.28-hecc5488_0.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/icc_rt-2019.0.0-h0cc432a_1.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/intel-openmp-2019.4-245.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/vs2015_runtime-14.0.25420-0.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/mkl-2019.4-245.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/vc-14-0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/glew-2.0.0-he025d50_1002.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/glpk-4.65-h2fa13f4_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/jpeg-9c-hfa6e2cd_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libblas-3.8.0-14_mkl.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/openssl-1.1.1d-hfa6e2cd_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sdl2-2.0.9-h6538335_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sqlite-3.26.0-hfa6e2cd_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/tk-8.6.9-hfa6e2cd_1002.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/xz-5.2.4-h2fa13f4_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/zlib-1.2.11-h2fa13f4_1004.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/zstd-1.3.3-vc14_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libcblas-3.8.0-14_mkl.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/liblapack-3.8.0-14_mkl.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libpng-1.6.37-h7602738_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libtiff-4.0.10-h016b793_1002.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/python-3.7.3-hb12ca83_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/smpeg2-2.0.0-h6538335_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/alabaster-0.7.12-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/appdirs-1.4.3-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/asn1crypto-1.0.0-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/certifi-2019.11.28-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/chardet-3.0.4-py37_1003.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/docutils-0.14-py37_1001.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/freetype-2.10.0-h5db478b_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/idna-2.8-py37_1000.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/imagesize-1.1.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/kiwisolver-1.1.0-py37he980bc4_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/markupsafe-1.1.1-py37hfa6e2cd_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/numpy-1.17.5-py37hc71023c_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/olefile-0.46-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/ply-3.11-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pycparser-2.19-py37_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pyparsing-2.4.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pytz-2019.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sdl2_image-2.0.4-h63225fd_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sdl2_mixer-2.0.4-h6538335_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/six-1.12.0-py37_1000.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/snowballstemmer-2.0.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-applehelp-1.0.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-devhelp-1.0.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-htmlhelp-1.0.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-jsmath-1.0.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-qthelp-1.0.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinxcontrib-serializinghtml-1.1.3-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/tornado-6.0.3-py37hfa6e2cd_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/win_inet_pton-1.1.0-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/wincertstore-0.2-py37_1002.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/babel-2.7.0-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cffi-1.12.3-py37hb32ad35_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/cycler-0.10.0-py_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/mkl-service-2.3.0-py37hfa6e2cd_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/packaging-19.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pillow-6.0.0-py37h9a613e6_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pysocks-1.7.1-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.8.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/sdl2_ttf-2.0.15-h4636d2b_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/setuptools-41.0.1-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/cryptography-2.7-py37hb32ad35_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/jinja2-2.10.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/matplotlib-base-3.1.2-py37h2981e6d_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/nose-1.3.7-py37_1002.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pandas-1.0.1-py37he350917_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pygments-2.4.2-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/scipy-1.3.1-py37h29ff71c_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/wheel-0.33.1-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/kivy-1.11.0-py37hbc14df2_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/patsy-0.5.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pip-19.1-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pyopenssl-19.0.0-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pyutilib-5.6.5-py37_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/pyomo-5.6.1-py37_3.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/statsmodels-0.11.0-py37hfa6e2cd_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/urllib3-1.25.6-py37_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/requests-2.22.0-py37_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/seaborn-0.9.0-py_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/sphinx-2.2.0-py_0.tar.bz2

Packaging on Windows

Currently, the packaging process is only worked out for Windows. The process for packaging can be automated using the following batch script:

set root=C:\Users\rconcep\AppData\Local\Continuum\anaconda3
set package_name=snl-quest-1.2.f

call %root%\Scripts\activate.bat %root%
call conda activate quest-kivy-1.11.0
call pip freeze
call del /S/Q build\
call del /S/Q dist\
call python -m PyInstaller --name %package_name% --icon Quest_App_Icon_256.ico ..\snl-quest-master\main.py -y
call xcopy /Y snl-quest_template.spec %package_name%.spec
call python -m PyInstaller %package_name%.spec -y

call del /S/Q dist\%package_name%\patch_note_resources\

• root should point to the root directory of the Anaconda installation such that line 4 points to the activate.bat script for establishing an "Anaconda Command Prompt"
• line 2 should be adjusted based on the name of the resulting executable (i.e., version number)
• line 5 will activate the virtual environment that will be bundled in the package. The name of the environment should be adjusted as necessary.
• lines 7-8 clean up previous packaging processes
• line 9 starts the PyInstaller process
  • The path for the --icon argument points to an .ico file used for the application icon.
  • The path after the icon path points to the entry function for the executable (i.e., main.py). In this example, it points to the main.py of a fresh copy of the master branch of the QuESt repository (download .zip from GitHub and extract)
• line 10 creates a .spec file for PyInstaller based on a template. That template can be found in the next section.
• line 11 starts the packaging process.
• line 13 cleans up some unnecessary files.

After the process completes successfully, the resulting package can be found in

%batch_file_location%\dist\%package_name%

Run the %package_name%.exe file to test the result. You can add extra content before compressing the directory into an archive for distribution. For example, you can adjust settings files for QuESt, include pre-downloaded data, include solver executables, etc.

.spec file

This is the snl-quest_template.spec referenced above. Paths should be adjusted as necessary. Of note are the fixes for Pyomo.

# -*- mode: python -*-
from kivy.deps import sdl2, glew
block_cipher = None


a = Analysis(['..\\snl-quest-master\\main.py'],
             pathex=['D:\\workspace\\quest-package'],
             binaries=[],
             datas=[],
             hiddenimports=['pyomo.opt.plugins', 'pyomo.core.plugins',
             'pyomo.dataportal.plugins', 'pyomo.duality.plugins',
             'pyomo.checker.plugins', 'pyomo.pysp.plugins', 'pyomo.neos.plugins',
             'pyomo.gdp.plugins', 'pyomo.mpec.plugins', 'pyomo.dae.plugins',
             'pyomo.bilevel.plugins', 'pyomo.scripting.plugins', 'pyomo.network.plugins', 'win32timezone'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name=specnm,
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True , icon='Quest_App_Icon_256.ico')
coll = COLLECT(exe, Tree('..\\snl-quest-master\\'),
               a.binaries,
               a.zipfiles,
               a.datas,
               *[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)],
               strip=False,
               upx=True,
               name=specnm)