-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implemented and tested initial script and REST API to support stopgap…
… line creation from a spreadsheet of ICE parts. 1) Created a "jbei" package for use in distributing library code to support standalone Python scripts. This code should eventually be versioned separately from EDD, since it's a dependency for EDD, but also dependent on EDD's and ICE's REST APIs. A) Extracted ICE REST code from the EDD Django codebase, removed direct dependencies on Django, and made it available in the standalone package i) Added client-side support for session-based authentication in IceApi. ii) Added code in IceApi to support returning objects from REST API calls (though this needs more work). This should improve abstraction and make it easier to maintain multiple ICE REST API clients (e.g. EDD, EDD analytics, the script added in this issue, and future researcher code) iii) Updated EDD code that referenced IceApi to reference it in its new location and to use its minorly-altered API B) Added edd.py to implement similar functionality to ice.py and provide EDD REST API clients the beginnings of a client-side library. Includes client-side authentication for sessions and Django REST Framework (DRF) authentication foibles. C) Added create_lines.py as a stopgap for bulk line creation in EDD (the main thrust of this task) D) Made some updates to the new REST API i) Some reorganization of module organization ii) Updates to the API/documentation based on testing (needs follow-on polishing/unit testing under separate linked issues)
- Loading branch information
0 parents
commit 4f58756
Showing
19 changed files
with
2,716 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
# JBEI Python Code | ||
|
||
This package contains Python code for general JBEI use, particularly for client interface to JBEI's | ||
web application at the API level. Code here is a work in progress, and should eventually be | ||
versioned and distributed independently of (though in coordinated with) specific application code | ||
such as EDD or ICE. | ||
|
||
If you aren't familiar with what an [API](https://en.wikipedia.org/wiki/Application_programming_interface) | ||
is, you probably shouldn't be using these scripts, or you should do so with help and with great | ||
care to avoid destroying important data. | ||
|
||
This initial version of these scripts and API's are only supported for the purpose of automating | ||
line creation in EDD. Feel free to write your own code against the API's defined here, but expect | ||
some changes as we polish them in upcoming versions. | ||
|
||
## Conventions in this document | ||
|
||
Don't panic! :-) | ||
|
||
These directions assume you're basically comfortable using the OSX Terminal. If not, or if you use | ||
other Python tools such as iPython, Jupyter, Pandas, etc and aren't comfortable working with | ||
virtual environments, it's probably best to ask for help. | ||
|
||
File names and terminal commands below are specially-formatted to clarify that they're | ||
associated with the `Terminal`. In a multi-line terminal block below, each line should be | ||
executed on its own, followed by Enter. We've made an attempt to point out how to verify that | ||
commands work correctly on your computer, but you should generally pay attention to the command | ||
output to notice any obvious signs that something went wrong (though unfortunately, the signs may | ||
not always be obvious). | ||
|
||
This stuff can be intimidating! Ask for help! | ||
|
||
## Set up a Python 2 environment to run this code | ||
### Mac OSX | ||
|
||
|
||
#### Install basic development tools needed to support the scripts. | ||
Depending on what's already installed / in use on your computer, you'll want to consider | ||
following directions in four of the first three sections below, plus (Xcode, Homebrew, Python, pip, | ||
virtualenvwrapper). | ||
|
||
1. Install XCode: <a name="XCode"/> | ||
Install XCode (and associated Developer Tools) via the App Store | ||
* As of OS X 10.9 "Mavericks": you can just run `xcode-select --install` at the terminal to just | ||
only get the command-line tools | ||
* Run the command below to stablish `/usr/include`: | ||
|
||
``sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include`` | ||
2. Install (Homebrew)[http://brew.sh/] <a name="HomeBrew"/> | ||
|
||
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" | ||
brew doctor | ||
3. Install Python 2 <a name="Python"/> | ||
* Replace default OS X version of Python with the more up-to-date Homebrew version | ||
|
||
`brew install python` | ||
* You may need to relaunch the terminal to see the proper Python version. Test by running | ||
`python --version` | ||
|
||
4. Create a [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) to | ||
isolate | ||
dependencies for these scripts from other Python code on your computer. Even if you don't do any | ||
other Python work at present, it's best to start off on the right foot in case you need to do it | ||
later on. | ||
|
||
* Install virtualenvwrapper | ||
|
||
`sudo pip install virtualenvwrapper` | ||
* Add the following lines to your shell startup file (e.g. `~/.bash_profile`), or create one | ||
if it doesn't exist | ||
|
||
# configure virtualenvwrapper to isolate Python environments | ||
export WORKON_HOME=$HOME/.virtualenvs | ||
source /usr/local/bin/virtualenvwrapper.sh | ||
* Include the changes you just made in the current Terminal: | ||
|
||
source ~/.bash_profile | ||
|
||
* Create a virtual environment for running these scripts | ||
|
||
mkvirtualenv jbei-scripts | ||
workon jbei-scripts | ||
|
||
5. Check that your Terminal is working in the context of the the virtual environment you just | ||
created. | ||
|
||
After running commands above to create a virtual environment, you'll want to get in the habit of | ||
checking that your terminal is using the correct virtual environment before running scripts | ||
included in this package, and especially before using `pip` to change the installed Python | ||
packages. | ||
|
||
To check which virtual environment your Terminal is in, run the Terminal and look at the | ||
Terminal's command prompt. The virtual environment name will be in parenthesis at the | ||
beginning of the prompt. For example: | ||
|
||
(jbei-python)mark.forrer@mforrer-mr:/Users/mark.forrer$ | ||
Alternately, you can edit change your `.bash_profile` to use this virtual environment by default | ||
by appending the line `workon jbei-scripts` after the commands you added above. | ||
|
||
#### Check out code to run the scripts | ||
|
||
* Download scripts from https://repo.jbei.org/projects/EDD/repos/edd-django/browse. These files may eventually be hosted elsewhere, but for now the initial versions are being developed/maintained concurrently with EDD. | ||
* Do a [sparse checkout](http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/) to get just the subsection of EDD code that you need to run these scripts. You won't want the whole application codebase. For example, run the following commands: | ||
* Create and initialize your local repo (replacing sample below with your own LDAP username): | ||
|
||
mkdir jbei\ python\ scripts && cd jbei\ python\ scripts | ||
git init | ||
git remote add –f edd-django https://[email protected]/scm/edd/edd-django.git | ||
* Enable sparse checkout so you can get just the scripts you need. | ||
|
||
git config core.sparsecheckout true | ||
* Configure git's `sparse-checkout`` file to get just the script code and its dependencies in the | ||
EDD code | ||
|
||
echo jbei/ >> .git/info/sparse-checkout | ||
* Checkout the scripts | ||
|
||
git pull edd-django master | ||
* Install required Python packages | ||
workon jbei-scripts | ||
pip install -r requirements.txt | ||
|
||
* Add the `jbei` directory, and any desired subdirectories to the $PYTHONPATH | ||
|
||
cd jbei/ | ||
PYTHONPATH=$PYTHONPATH:`pwd`/jbei:`pwd`/jbei/edd/rest/scripts/ | ||
|
||
alternately, add the `PYTHNONPATH line above to your .bash_profile | ||
|
||
* Edit the `jbei/edd/rest/scripts/settings.py` file if needed. Its purpose is to set defaults used | ||
by the scripts to contact EDD and ICE. Unless you're using it for code development or testing, | ||
the defaults should be fine. | ||
|
||
## Run a script! | ||
|
||
For example: `python create_lines.py my_csv_file.csv -study 1234` | ||
|
||
## Get the latest code | ||
|
||
From the repository directory you configured, just run | ||
|
||
git pull | ||
|
||
Keep in mind that new code may have been added somewhere other than where your sparse checkout is | ||
looking for it! You can always browse the rest of the code in [BitBucket](https://repo.jbei.org/projects/EDD/repos/edd-django/browse/jbei/) | ||
if that's needed. | ||
|
||
|
||
|
||
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
""" | ||
Contains Python code for general use with JBEI applications. Code within this package and | ||
sub-packages should have minimal dependencies on specific application, with the goal of eventually | ||
distributing it seperately from application as a set of general utilities for researchers' use. | ||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
arrow==0.6.0 | ||
Django==1.9 | ||
django-auth-ldap==1.2.6 | ||
django-debug-toolbar==1.4 | ||
django-extensions==1.6.1 | ||
django-form-utils==1.0.3 | ||
django-hstore==1.4.1 | ||
django-registration-redux==1.2 | ||
django-threadlocals==0.8 | ||
djangorestframework==3.3.2 | ||
drf-nested-routers==0.11.1 | ||
future==0.15.2 | ||
idna==2.0 | ||
ipaddress==1.0.14 | ||
jdcal==1.2 | ||
jsonpickle==0.9.2 | ||
mccabe==0.3.1 | ||
openpyxl==2.2.6 | ||
psycopg2==2.6.1 | ||
pyasn1==0.1.9 | ||
pysmb==1.1.16 | ||
python-dateutil==2.4.2 | ||
python-ldap==2.4.25 | ||
pytz==2015.4 | ||
PyYAML==3.11 | ||
requestes==0.0.1 | ||
requests==2.9.0 | ||
six==1.10.0 | ||
sqlparse==0.1.18 | ||
wheel==0.24.0 |
Empty file.
Empty file.
Oops, something went wrong.