Skip to content

Commit

Permalink
Implemented and tested initial script and REST API to support stopgap…
Browse files Browse the repository at this point in the history
… line creation from a spreadsheet of ICE parts.

1) Created a "jbei" package for use in distributing library code to support standalone Python scripts. This code should eventually be versioned separately from EDD, since it's a dependency for EDD, but also dependent on EDD's and ICE's REST APIs.
   A) Extracted ICE REST code from the EDD Django codebase, removed direct dependencies on Django, and made it available in the standalone package
      i) Added client-side support for session-based authentication in IceApi.
      ii) Added code in IceApi to support returning objects from REST API calls (though this needs more work). This should improve abstraction and make it easier to maintain multiple ICE REST API clients (e.g. EDD, EDD analytics, the script added in this issue, and future researcher code)
      iii) Updated EDD code that referenced IceApi to reference it in its new location and to use its minorly-altered API
   B) Added edd.py to implement similar functionality to ice.py and provide EDD REST API clients the beginnings of a client-side library.  Includes client-side authentication for sessions and Django REST Framework (DRF) authentication foibles.
   C) Added create_lines.py as a stopgap for bulk line creation in EDD (the main thrust of this task)
   D) Made some updates to the new REST API
      i) Some reorganization of module organization
      ii) Updates to the API/documentation based on testing (needs follow-on polishing/unit testing under separate linked issues)
  • Loading branch information
chimaerase committed Feb 24, 2016
0 parents commit 4f58756
Show file tree
Hide file tree
Showing 19 changed files with 2,716 additions and 0 deletions.
155 changes: 155 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# JBEI Python Code

This package contains Python code for general JBEI use, particularly for client interface to JBEI's
web application at the API level. Code here is a work in progress, and should eventually be
versioned and distributed independently of (though in coordinated with) specific application code
such as EDD or ICE.

If you aren't familiar with what an [API](https://en.wikipedia.org/wiki/Application_programming_interface)
is, you probably shouldn't be using these scripts, or you should do so with help and with great
care to avoid destroying important data.

This initial version of these scripts and API's are only supported for the purpose of automating
line creation in EDD. Feel free to write your own code against the API's defined here, but expect
some changes as we polish them in upcoming versions.

## Conventions in this document

Don't panic! :-)

These directions assume you're basically comfortable using the OSX Terminal. If not, or if you use
other Python tools such as iPython, Jupyter, Pandas, etc and aren't comfortable working with
virtual environments, it's probably best to ask for help.

File names and terminal commands below are specially-formatted to clarify that they're
associated with the `Terminal`. In a multi-line terminal block below, each line should be
executed on its own, followed by Enter. We've made an attempt to point out how to verify that
commands work correctly on your computer, but you should generally pay attention to the command
output to notice any obvious signs that something went wrong (though unfortunately, the signs may
not always be obvious).

This stuff can be intimidating! Ask for help!

## Set up a Python 2 environment to run this code
### Mac OSX


#### Install basic development tools needed to support the scripts.
Depending on what's already installed / in use on your computer, you'll want to consider
following directions in four of the first three sections below, plus (Xcode, Homebrew, Python, pip,
virtualenvwrapper).

1. Install XCode: <a name="XCode"/>
Install XCode (and associated Developer Tools) via the App Store
* As of OS X 10.9 "Mavericks": you can just run `xcode-select --install` at the terminal to just
only get the command-line tools
* Run the command below to stablish `/usr/include`:

``sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include``
2. Install (Homebrew)[http://brew.sh/] <a name="HomeBrew"/>

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew doctor
3. Install Python 2 <a name="Python"/>
* Replace default OS X version of Python with the more up-to-date Homebrew version

`brew install python`
* You may need to relaunch the terminal to see the proper Python version. Test by running
`python --version`

4. Create a [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) to
isolate
dependencies for these scripts from other Python code on your computer. Even if you don't do any
other Python work at present, it's best to start off on the right foot in case you need to do it
later on.

* Install virtualenvwrapper

`sudo pip install virtualenvwrapper`
* Add the following lines to your shell startup file (e.g. `~/.bash_profile`), or create one
if it doesn't exist

# configure virtualenvwrapper to isolate Python environments
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
* Include the changes you just made in the current Terminal:

source ~/.bash_profile

* Create a virtual environment for running these scripts

mkvirtualenv jbei-scripts
workon jbei-scripts

5. Check that your Terminal is working in the context of the the virtual environment you just
created.

After running commands above to create a virtual environment, you'll want to get in the habit of
checking that your terminal is using the correct virtual environment before running scripts
included in this package, and especially before using `pip` to change the installed Python
packages.

To check which virtual environment your Terminal is in, run the Terminal and look at the
Terminal's command prompt. The virtual environment name will be in parenthesis at the
beginning of the prompt. For example:

(jbei-python)mark.forrer@mforrer-mr:/Users/mark.forrer$
Alternately, you can edit change your `.bash_profile` to use this virtual environment by default
by appending the line `workon jbei-scripts` after the commands you added above.

#### Check out code to run the scripts

* Download scripts from https://repo.jbei.org/projects/EDD/repos/edd-django/browse. These files may eventually be hosted elsewhere, but for now the initial versions are being developed/maintained concurrently with EDD.
* Do a [sparse checkout](http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/) to get just the subsection of EDD code that you need to run these scripts. You won't want the whole application codebase. For example, run the following commands:
* Create and initialize your local repo (replacing sample below with your own LDAP username):

mkdir jbei\ python\ scripts && cd jbei\ python\ scripts
git init
git remote add –f edd-django https://[email protected]/scm/edd/edd-django.git
* Enable sparse checkout so you can get just the scripts you need.

git config core.sparsecheckout true
* Configure git's `sparse-checkout`` file to get just the script code and its dependencies in the
EDD code

echo jbei/ >> .git/info/sparse-checkout
* Checkout the scripts

git pull edd-django master
* Install required Python packages
workon jbei-scripts
pip install -r requirements.txt

* Add the `jbei` directory, and any desired subdirectories to the $PYTHONPATH

cd jbei/
PYTHONPATH=$PYTHONPATH:`pwd`/jbei:`pwd`/jbei/edd/rest/scripts/

alternately, add the `PYTHNONPATH line above to your .bash_profile

* Edit the `jbei/edd/rest/scripts/settings.py` file if needed. Its purpose is to set defaults used
by the scripts to contact EDD and ICE. Unless you're using it for code development or testing,
the defaults should be fine.

## Run a script!

For example: `python create_lines.py my_csv_file.csv -study 1234`

## Get the latest code

From the repository directory you configured, just run

git pull

Keep in mind that new code may have been added somewhere other than where your sparse checkout is
looking for it! You can always browse the rest of the code in [BitBucket](https://repo.jbei.org/projects/EDD/repos/edd-django/browse/jbei/)
if that's needed.




5 changes: 5 additions & 0 deletions __init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""
Contains Python code for general use with JBEI applications. Code within this package and
sub-packages should have minimal dependencies on specific application, with the goal of eventually
distributing it seperately from application as a set of general utilities for researchers' use.
"""
30 changes: 30 additions & 0 deletions django-drf-requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
arrow==0.6.0
Django==1.9
django-auth-ldap==1.2.6
django-debug-toolbar==1.4
django-extensions==1.6.1
django-form-utils==1.0.3
django-hstore==1.4.1
django-registration-redux==1.2
django-threadlocals==0.8
djangorestframework==3.3.2
drf-nested-routers==0.11.1
future==0.15.2
idna==2.0
ipaddress==1.0.14
jdcal==1.2
jsonpickle==0.9.2
mccabe==0.3.1
openpyxl==2.2.6
psycopg2==2.6.1
pyasn1==0.1.9
pysmb==1.1.16
python-dateutil==2.4.2
python-ldap==2.4.25
pytz==2015.4
PyYAML==3.11
requestes==0.0.1
requests==2.9.0
six==1.10.0
sqlparse==0.1.18
wheel==0.24.0
Empty file added edd/__init__.py
Empty file.
Empty file added edd/rest/__init__.py
Empty file.
Loading

0 comments on commit 4f58756

Please sign in to comment.