Merge pull request #378 in EDD/edd-django from ~WCMORRELL/edd:websock…

…et-notify to master Squashed commit of the following: commit 9e8d955c6586b8c317b7ad764a1a0b40eb2036b3 Author: William Morrell <[email protected]> Date: Mon Mar 12 14:42:09 2018 -0700 Improve test_ice_communication command Squashed commit of the following: commit 25fa46ba8b33cc716cea0ad6ea7f484e37325ca2 Author: Mark Forrer <[email protected]> Date: Mon Mar 5 15:37:42 2018 -0600 Addressed PR comments commit 2a0f94ca40561d7339b992f177163bb2b6c7ac33 Author: Mark Forrer <[email protected]> Date: Mon Mar 5 11:37:13 2018 -0600 Reverted simple rebase error commit 86b03ac7a6d02630ce63b4f4063fa2cc75d3dd11 Author: Mark Forrer <[email protected]> Date: Mon Mar 5 11:28:44 2018 -0600 Added configurable support for running simple ICE search as in UI commit a0da65a60bcd780d9c4a390185d15148c6ed40c9 Author: Mark Forrer <[email protected]> Date: Wed Feb 28 13:14:40 2018 -0800 Correct handling of 403 from ICE in ExpDesc commit 087816e70bd2837a8519215b7a00f127a6ee4bab Author: William Morrell <[email protected]> Date: Tue Mar 6 07:15:22 2018 -0800 Import dropzone allows direct text input again commit 0f0fdd1fe916f13ba079ac948fda13e3a60b733d Author: William Morrell <[email protected]> Date: Mon Mar 5 07:52:41 2018 -0800 Remove dropzones from read-only studies (EDD-1034) commit 30131dc4b9949a4dbf3d4685e3316d1d48b35e81 Author: William Morrell <[email protected]> Date: Thu Mar 1 11:14:01 2018 -0800 Auth code handles Requests request and Django request commit 3b22bf9e5a027f3547a2505d6b97e637b4bde2d7 Author: William Morrell <[email protected]> Date: Wed Feb 28 15:25:02 2018 -0800 Add HTTP 403 handling to ExpDesc communication Squashed commit of the following: commit ef97183bc21c3cc7a62a51d2be925d120e13776c Author: Mark Forrer <[email protected]> Date: Wed Feb 28 14:09:29 2018 -0800 Recast loop as a set comprehension commit 8b491b2ecfa2cf4798360c3fd1d0fd9e6fc663fd Author: Mark Forrer <[email protected]> Date: Wed Feb 28 13:14:40 2018 -0800 Back-end fix commit 6363c8d0867d771d3d5d3fd03965d485bcd43b9b Author: William Morrell <[email protected]> Date: Wed Feb 28 14:54:17 2018 -0800 Fixing admin type merge autocompletes (EDD-962) commit 541d329bb30a609568037dfaa457910871d62c2a Author: William Morrell <[email protected]> Date: Tue Feb 27 09:57:24 2018 -0800 Adding short commit hash to version display commit 54f1d78b6ec21f5f14341da55abbc0ebf8dbfb48 Author: William Morrell <[email protected]> Date: Mon Feb 26 21:47:15 2018 -0800 Resolving npm warnings for typescript build image commit 158859245163340828aeceb68f92f4bab2d036ce Author: William Morrell <[email protected]> Date: Mon Feb 26 15:25:57 2018 -0800 Set study contact as line experimenter if blank (EDD-880) commit 747e7b8ed0730dd1cdb6e5650192cd606dcf05aa Author: William Morrell <[email protected]> Date: Mon Feb 26 12:42:49 2018 -0800 Set current user as study contact if blank (EDD-879) commit 58d63f6b08fc4ddff62655c102151492d677ee54 Author: William Morrell <[email protected]> Date: Mon Feb 26 12:42:20 2018 -0800 Adding a better permission filter API commit e3d22d633a13507e4fd900f7dc0f2bcc4dd89000 Author: William Morrell <[email protected]> Date: Thu Feb 22 13:04:49 2018 -0800 Stop showing metabolite selection for skyline import commit 0e6f3b448a90a4fab5d5f86b989f9345d1bc1da9 Author: William Morrell <[email protected]> Date: Wed Feb 21 12:37:25 2018 -0800 Clean up HTML templates and relative URLs commit 21ea82e49490487432b39958f844be90465f083e Author: William Morrell <[email protected]> Date: Tue Feb 20 14:09:15 2018 -0800 Make the 'Add Measurement' dialog work again Also fixing some issues with the 'Add Assay' dialog, as it has similar problems to the ones on the 'Add Measurement' dialog. Fixes EDD-1005 and EDD-1010. commit d3088311a63c00414c35db7a0154b50cb12ae45f Author: William Morrell <[email protected]> Date: Mon Feb 19 14:45:32 2018 -0800 Remove popup menus to resolve flickering problems Cleaning up HTML templates as well: - moving to two-space indent, to accommodate the deeper nesting in HTML - using trans and blocktrans for most user-visible text - prefer putting child elements on new lines over stuffing into one line - prefer putting attributes on new lines over multiples on same line commit 8fbfa2a26bb7456b62bf6e71c07bb19b83dc0f58 Author: William Morrell <[email protected]> Date: Thu Feb 15 13:33:58 2018 -0800 Fix EDD-1003, help tooltip rendering off page commit c22aaf0654f0cd6111e58cc45786c70269330a84 Author: Mark Forrer <[email protected]> Date: Thu Feb 15 10:16:40 2018 -0800 Updated ED parsing to skip entirely blank rows Improved related error/log messages. Untested, but changes here should be simple & testing easy with files attached to the ticket. commit e0b1aa472edf1675bda7e60a0090e349da9855b1 Author: Mark Forrer <[email protected]> Date: Tue Feb 6 14:46:57 2018 -0800 Collected changes for rest-client Docker image 1) Added a Dockerfile for building a separate container to run rest client and example code 2) Pared down and updated initial requirements.txt file that was a superset of libraries actually needed to run the code. 3) Bug fixes and minor improvements from re-testing sample code in Python 3 4) Removed ICE package-level import that caused an error when dealing with EDD/ICE strains at the same time. Removed other client package-level imports too. 5) Re-applied client code changes to correctly convert server side REST replies into Unicode (server is currently Python 2, sometimes generates bytestrings) 6) Tweaked Docker commands in README commit 0d796f1ed3dd12714b1d99093d13b0d7488d1e6a Author: William Morrell <[email protected]> Date: Wed Feb 14 18:54:40 2018 -0800 Removing (broken) line replicate data model commit bc2d5a5024ae5fa3bbb908181d16338f2703c257 Author: William Morrell <[email protected]> Date: Wed Feb 14 18:24:14 2018 -0800 Removing the (broken) group line feature commit 1d8788b3715de469434d1db833139ab129f28828 Author: William Morrell <[email protected]> Date: Wed Feb 14 17:39:01 2018 -0800 Views added to delete attachments In process of adding in attachment deletion also did: * Removed old unused /data/____ URLs and views * Refactored popup menu code to a function in Utl.ts module * Removed popup menu from attachments table, use a delete button instead commit 5f4256983270c29121ee1f6e5fdd29b7cc483bb9 Author: William Morrell <[email protected]> Date: Mon Feb 12 17:57:26 2018 -0800 Adding confirmation to delete from data page commit 383d23a9fb5379b4f684f441e13e296720b5f656 Author: William Morrell <[email protected]> Date: Mon Feb 12 16:41:43 2018 -0800 Removing unused templates commit b74e145ca47f8cde0b5abae00e6b7da4c1817225 Author: William Morrell <[email protected]> Date: Thu Feb 8 12:54:26 2018 -0800 Fix for EDD-996, literal HTML tags in description The TinyMCE editor returns formatted HTML, but the EDDEditableElement code was inserting the value as a text node, causing entity escaping for the HTML instead of inserting the HTML as normal nodes. commit 68e3d563968811ec48541278b0ea4044f383c94c Author: William Morrell <[email protected]> Date: Thu Feb 8 12:33:47 2018 -0800 Fix for EDD-995, overview page dropzone Lots of tweaks to the code; adding dropzone styling, replacing several instances of URL building via string concatenation with use of native URL class API; better handling of dropzone events. commit 7d5a99e3b1ae679561a885015e55c917aa27116e Author: William Morrell <[email protected]> Date: Thu Feb 8 10:06:32 2018 -0800 Add a decorator to simplify parser registration commit 400cefd21135e9f3ec65517cf0b86e995f588b2b Author: William Morrell <[email protected]> Date: Tue Feb 6 17:46:22 2018 -0800 Fix 'Latest Viewed Studies' encoding issue commit 6d86090824ecbb98a6a46b5d6a66d22919e48755 Author: William Morrell <[email protected]> Date: Tue Feb 6 17:45:29 2018 -0800 Better checks around indexing measurement types commit a8282f6b890090fac5a07e99780c59957924e76e Author: William Morrell <[email protected]> Date: Tue Feb 6 12:57:50 2018 -0800 Adding ability for optional ICE to email commit 252de5b68257610d1a3eac7157f84628d7a144b5 Author: William Morrell <[email protected]> Date: Tue Feb 6 12:57:07 2018 -0800 Daphne/Twisted require OpenSSL to be available commit ba59a41615119560264a0e781301e5d6e460db70 Author: William Morrell <[email protected]> Date: Tue Feb 6 11:58:40 2018 -0800 Correcting additional python3 conversion errors The python3 conversion branch was rebased onto further changes to master, after the 2to3 conversion was run. Since the conversion is tedious, the full process was not completed again with every rebase. This commit fixes some of the python3 incompatible code that worked in following rebases. commit 53342e69f6f69b09dfa24b492fa01a41c01f158a Author: William Morrell <[email protected]> Date: Thu Jan 25 13:44:30 2018 -0800 Fix settings so REST API uses HTTPS links commit 2a0140412eb45100a615038af823c3dfff47a309 Author: William Morrell <[email protected]> Date: Wed Jan 24 14:36:02 2018 -0800 Update maintainer, use quoting around variables commit acf9647fa83d03161a4dc9da9c2f57d11bdd2ca9 Author: William Morrell <[email protected]> Date: Thu Nov 30 12:32:49 2017 -0800 Initial work on a notifications Django app commit a2048665ad4c97071626d392abdc7742631a1572 Author: William Morrell <[email protected]> Date: Mon Nov 27 15:37:17 2017 -0800 Fix for nginx hitting timeout on websockets commit eb6d4a4ec099c3c7a6f6ae2606c528f84b0389e7 Author: William Morrell <[email protected]> Date: Mon Feb 5 15:12:07 2018 -0800 Simple demo of websocket cursor tracking commit 60b719d378999888a8287066ff99d7acd65a3bbe Author: William Morrell <[email protected]> Date: Tue Nov 7 17:24:52 2017 -0800 EDD working with daphne terminating HTTP requests commit 708e1b0b35a33d8ef66256aa278e6fc4e2765b24 Author: William Morrell <[email protected]> Date: Tue Nov 7 12:56:38 2017 -0800 Move from Debian stretch to buster Stable has python 3.5, while testing has 3.6. Had to move numpy, scipy, scikit-learn from dpkg installs to pip installs. The python 3.5 runtime was throwing errors around use of weakrefs, and the problem is fixed in 3.6. commit bcf3fce7c36f819a2a65b4d0521e49b79f3b0cd6 Author: William Morrell <[email protected]> Date: Mon Nov 6 14:43:29 2017 -0800 Removing unused files found by coverage.py commit 6d847cf699e4b05d2f7b0eb0f702166927c2fdc8 Author: William Morrell <[email protected]> Date: Mon Nov 6 14:27:40 2017 -0800 Correcting profile views and adding test coverage commit f88f38e96f55b3838366d0c4ddb1d77c3f0a55ac Author: William Morrell <[email protected]> Date: Thu Nov 2 15:47:01 2017 -0700 Excel parser header detection broken The excel parser would assume any first row containing a number is *not* a header row. This breaks tests for the PCAP tutorial OD import file. Changed the 'header detection' to always use the first row as a header, whether it has a number or not. commit 444f1b10b44c819361253fb0f93113a255ff1f57 Author: William Morrell <[email protected]> Date: Wed Nov 1 14:55:45 2017 -0700 Correcting string encoding issues for Python3 commit 5669182c04f4d6289db0586758437c4d5979f898 Author: William Morrell <[email protected]> Date: Wed Nov 1 14:46:05 2017 -0700 Updating the format of redis cache URL The old format of rediscache://{host}:{port}:{database} was causing an error under Python 3. Changing the URL format to instead be rediscache://{host}:{port}/{database} resolves the problem. commit 7f6a10be75bc59bd291de70bf02eee6025c525b9 Author: William Morrell <[email protected]> Date: Wed Nov 1 14:22:21 2017 -0700 Update code based on patches from 2to3 Ran 2to3 -f all -x future -f idioms -f set_literal -f ws_comma . Then, manually resolved all identified diffs to use code compatible with both Python 2 and Python 3. commit 331cfbc6ff602bf29da9f611a757b5dac18dc101 Author: William Morrell <[email protected]> Date: Tue Oct 31 14:56:20 2017 -0700 Create buildable python3 image Change the python apt packages to python3 versions, use pip3 instead of pip, and replace python2-only python-ldap with pyldap.
JBEI · Mar 14, 2018 · 9842e1a · 9842e1a
1 parent f569eac
commit 9842e1a
Show file tree

Hide file tree

Showing 18 changed files with 325 additions and 209 deletions.
diff --git a/README.md b/README.md
@@ -5,10 +5,9 @@ web applications at the API level. Code here is a work in progress, and should e
 versioned and distributed independently of (though coordinated with) specific application code
  such as EDD or ICE.
 
-<em>If you aren't familiar with what an [API][1]
-is, you probably shouldn't write your own code using these scripts, or <font color="red"><u>you
-should do so with help and with great care to avoid destroying important scientific data hosted in
-JBEI's web applications.</u></font>.</em>
+<em>If you aren't familiar with what an [API][1] is, you probably shouldn't write your own code 
+using these scripts, or <font color="red"><u>you should do so with help and with great care to 
+avoid destroying important scientific data hosted in JBEI's web applications.</u></font>.</em>
 
 The initial version of these scripts and API's are being published for comments, and will 
 likely need to evolve over time.  While breaking API changes will be avoided like the plague, 
@@ -31,39 +30,54 @@ not always be obvious).
 
 This stuff can be intimidating! Ask for help!
 
-## Set up a Python 2 environment
-
-See [the directions][11] for setting up a Python environment to run 
-this code. Please note that EDD is in the process of transitioning to Python 3, and these 
-published client-side libraries and example scripts will also be migrated to Python 3. The
- Python 2 versions should continue to work.
+## Setting up a Python 3 environment
+
+You have two options for setting up a Python environment to run this code. 
+
+1. __Use our pre-built Docker image__
+
+   This approach takes some investment in learning [Docker][3], but also helps to avoid the 
+   typical hassles involved with running multiple Python projects on a single computer. To check 
+   out the Docker image, just run the following example steps:
+
+       docker pull jbei/rest-client:latest
+       docker run -it --rm --entrypoint /bin/bash jbei/rest-client:latest
+       
+   At this point, you'll get a terminal prompt for the running Docker container, and you can 
+   execute the sample scripts (see sample commands below). 
 
-#### Configure the target URL's for the script
-
-If you're running a command line tool that targets a specific EDD and/or ICE deployment, you should 
-edit configuration files to adjust which URL's are used to access those deployments.
-
-* `jbei/edd/rest/scripts/settings/` contains the default settings used by all the scripts in this
- directory. If you need to change the defaults in this file, create a `local.py` in the same 
- directory, and any values defined in your `local.py` will override the defaults, but won't show 
- up as edits when you use `git` to check out the latest code. 
+   For repeated EDD sample script runs, you'll likely want to create a `local.py` file under 
+   `/code/jbei/edd/rest/scripts/settings` to store commonly-used parameters in lieu of 
+   providing them as command line arguments.  You can use Docker's `--mount` option to make a 
+   file from your computer visible inside the Docker container, e.g.
+
+       docker run -it --rm --entrypoint /bin/bash  \
+           --mount type=bind,src=/your/local/path/local.py,
+           dst=/code/jbei/edd/rest/scripts/settings/local.py \
+           jbei/rest-client:latest
+
+2. __Configure your own custom Python environment__
+
+   See [the directions][11] for setting up a Python environment to run this code. Note that you'll
+   have to manage interactions with any other Python tools already installed on your system. Third
+   party tools you installed on your system may create some headaches with this approach.
 
 ## Provided Code
 
-Three types of code are provided in this package:
+Several types of code are provided in this package:
 
 1. Client-side Python API's for accessing JBEI's web applications
 2. Example scripts, e.g. scripts that demonstrate basic and advanced use of EDD's REST API.
 3. Special-purpose scripts that use the Python API's to accomplish an administrative task 
 (likely to be relocated later).
 4. General utility code, mostly in support of #1
 
-### Python API's <a name="python_apis">
+### Python API's <a name="python_apis"></a>
 
 Client-side Python libraries for accessing ICE's and EDD's REST API's have just been released, but 
 are already in limited production use by EDD and by its command line tools.
 These libraries aren't mature yet, but may already be helpful for other uses (e.g. in researchers' 
-iPython notebooks). This code is still in active development, and is likely to change over time. 
+Jupyter notebooks). This code is still in active development, and is likely to change over time. 
 Feel free to use it, but use at your own risk!
 
 See `api.py` modules for EDD and ICE under [`jbei/rest/clients/`][8], as well as other supporting 
@@ -96,11 +110,12 @@ For examples of more advanced use, see usage of EddApi in the main() methods of
 
 ### Command Line Tools
 
-The following command-line tools re provided. Run each with the `--help` parameter for more 
-detailed information on the available options.
+The following command-line tools re provided as working examples. Run each with the `--help` 
+parameter for more detailed information on the available options.
 
 * `basic_rest_queries.py` This script demonstrates the most basic approach extracting study data
-  from EDD using its REST API.
+  from EDD using its REST API. It makes a number of simplifying assumptions that may not hold up
+  during actual use.
 * `sample_rest_queries.py` This script is a more powerful and realistic client-side example of 
   using EDD's REST API.  It allows clients to perform sample searches of EDD to extract data of
   interest, and supports optionally writing the results to a simple CSV file similar to EDD's 
@@ -115,9 +130,21 @@ detailed information on the available options.
 Running an example script from the base EDD directory: 
 `python -m jbei.edd.rest.scripts.sample_rest_queries`
 
+#### Configuring target URL's for the sample scripts
+
+If you're running one of the sample scripts that targets a specific EDD and/or ICE deployment, you 
+may want to create a configuration file to adjust which URL's are used to access EDD and ICE.
+
+* `jbei/edd/rest/scripts/settings/` contains the default settings used by all the scripts in this
+ directory. If you need to change the defaults in this file, create a `local.py` in the same 
+ directory, and any values defined in your `local.py` will override the defaults, but won't show 
+ up as edits when you use `git` to check out the latest code. 
+
 
 [1]:    https://en.wikipedia.org/wiki/Application_programming_interface
 [2]:    edd/rest/scripts/Maintain_Links.md
+[3]:    https://docker.io
+[4]:    https://docs.docker.com/compose/overview/
 [8]:    rest/clients/
 [9]:    edd/rest/scripts/basic_rest_queries.py
 [10]:   edd/rest/scripts/sample_rest_queries.py

diff --git a/edd/rest/scripts/__init__.py b/edd/rest/scripts/__init__.py
@@ -1,3 +1,8 @@
 """
 Contains shared scripts for interacting with EDD's REST interface
-"""
+"""
+import os
+
+# set a default value for ICE_SETTINGS_MODULE (provides settings and prevents a django import in
+# jbei.rest.clients.ice.api)
+os.environ.setdefault('ICE_SETTINGS_MODULE', 'jbei.edd.rest.scripts.settings')
diff --git a/edd/rest/scripts/basic_rest_queries.py b/edd/rest/scripts/basic_rest_queries.py
@@ -1,6 +1,6 @@
 """
 This sample file shows a simplistic example use of EDD's client-side REST API access. For a more
-robust and realistic example, see sample_rest_queries.py. To run this script
+robust and realistic example, see sample_rest_queries.py.
 
 To provide the most basic example, this script makes several simplifying assumptions that
 aren't likely to hold true in real-world examples:
@@ -24,52 +24,67 @@
 more realistic scenario in which all of these simplifying assumptions are accounted for.
 """
 import argparse
+import logging
+import requests
 
-from jbei.rest.clients import EddApi
+from jbei.rest.clients.edd.api import EddApi
 from jbei.rest.auth import EddSessionAuth
 from logging.config import dictConfig
+from requests.packages.urllib3.exceptions import InsecureRequestWarning
 
 from . import settings
 
+logger = logging.getLogger(__name__)
+
 dictConfig(settings.LOGGING)
 
+# silence library warnings if we're skipping SSL certificate verification for local
+# testing. otherwise the warnings will swamp useful output from this script
+if not settings.VERIFY_EDD_CERT:
+    logger.warning('Skipping SSL certificate checking due to VERIFY_EDD_CERT setting. This '
+                   'option should only be used for local testing!')
+    requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
+
 
 def parse_arguments():
     ############################################################################################
     # Configure command line parameters. In this sample, username/password can be provided in a
-    # local_settings.py file, overridden at the command line, or the command line user is prompted
-    # if they aren't found in any other source.
+    # local.py file, overridden at the command line, or the command line user is prompted
+    # if they aren't provided by any other source.
     ############################################################################################
     parser = argparse.ArgumentParser(description='A sample script that demonstrates anticipated '
                                                  'use of EDD REST API to simplify integration '
                                                  'work for client applications.')
-    parser.add_argument('--username', '-u',
+    parser.add_argument('study', help='the identifier for the study ')
+    parser.add_argument('-username', '-u',
                         help='The username used to authenticate the EDD REST API'
                              'APIs. If provided, overrides username in the '
                              'settings file. If not provided, a user prompt will appear.')
-    parser.add_argument('--%password', '-p',
+    parser.add_argument('-password', '-p',
                         help='The password used to authenticate with both EDD & ICE APIs. If '
                              'provided, overrides the password provided in the settings file.  '
                              'If not provided, a user prompt will appear.')
-    parser.add_argument('--edd_url', '-e', help='the URL to use in contacting EDD.')
-    parser.add_argument('--study', '-s', help='the identifier for the study ')
-
+    parser.add_argument('-edd_url', '-e', help='the URL to use in contacting EDD.')
     return parser.parse_args()
 
 
-def __main__():
+def main():
 
     args = parse_arguments()
 
     study_id = args.study
     base_url = args.edd_url if args.edd_url else getattr(settings, 'EDD_URL')
-    username = args.username if args.username else getattr(settings, 'EDD_USERNAME')
-    password = args.password if args.password else getattr(settings, 'EDD_PASSWORD')
+    username = args.username if args.username else getattr(settings, 'EDD_USERNAME', None)
+    password = args.password if args.password else getattr(settings, 'EDD_PASSWORD', None)
     verify_ssl = getattr(settings, 'VERIFY_EDD_CERT', True)
 
+    logger.info('Logging into EDD at %s...' % base_url)
     session_auth = EddSessionAuth.login(base_url=base_url, username=username, password=password,
                                         verify_ssl_cert=verify_ssl)
 
+    if not session_auth:
+        return
+
     edd = EddApi(session_auth, base_url)
     study = edd.get_study(study_id)
     lines = edd.search_lines(study_id=study_id)
@@ -82,4 +97,13 @@ def __main__():
         measurement_type = edd.get_measurement_type(meas.measurement_type)
 
         # look up units used in the data
-        units = edd.get_measurement_unit(meas.y_units)
+        units = edd.get_measurement_unit(meas.y_units)
+
+    logger.info('Done reading data from study %s' % study_id)
+
+    return 0
+
+
+if __name__ == '__main__' or __name__ == 'jbei.edd.rest.scripts.basic_rest_queries':
+    result = main()
+    exit(result)
diff --git a/edd/rest/scripts/get_usage_statistics.py b/edd/rest/scripts/get_usage_statistics.py
@@ -17,10 +17,10 @@
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
 
 from jbei.rest.auth import EddSessionAuth
-from jbei.rest.clients import EddApi
+from jbei.rest.clients.edd.api import EddApi
 from jbei.rest.utils import is_url_secure
 from jbei.utils import session_login, TerminalFormats, UserInputTimer
-from .settings import (EDD_REQUEST_TIMEOUT, EDD_URL, VERIFY_EDD_CERT)
+from . import settings
 
 now = arrow.utcnow()
 zero_time_delta = now - now
@@ -32,8 +32,6 @@
 SEPARATOR_CHARS = 75
 OUTPUT_SEPARATOR = ('*' * SEPARATOR_CHARS)
 
-os.environ.setdefault('ICE_SETTINGS_MODULE', 'jbei.edd.rest.scripts.settings')
-
 
 def main():
 
@@ -64,16 +62,19 @@ def main():
                         default='US/Pacific')
     args = parser.parse_args()
 
+    username = args.username if args.username else getattr(settings, 'EDD_USERNAME', None)
+    password = args.password if args.password else getattr(settings, 'EDD_PASSWORD', None)
+
     ############################################################################################
     # Repeat back important parameters
     ############################################################################################
     print(OUTPUT_SEPARATOR)
     print(os.path.basename(__file__))
     print(OUTPUT_SEPARATOR)
     print('\tSettings module:\t%s' % os.environ['ICE_SETTINGS_MODULE'])
-    print('\tEDD URL:\t%s' % EDD_URL)
-    if args.username:
-        print('\tEDD Username:\t%s' % args.username)
+    print('\tEDD URL:\t%s' % settings.EDD_URL)
+    if username:
+        print('\tEDD Username:\t%s' % username)
     print(OUTPUT_SEPARATOR)
 
     ############################################################################################
@@ -82,12 +83,12 @@ def main():
     # easy to make!
     ############################################################################################
 
-    if not is_url_secure(EDD_URL, print_err_msg=True, app_name='EDD'):
+    if not is_url_secure(settings.EDD_URL, print_err_msg=True, app_name='EDD'):
         return 0
 
     # silence library warnings if we're skipping SSL certificate verification for local
     # testing. otherwise the warnings will swamp useful output from this script
-    if not VERIFY_EDD_CERT:
+    if not settings.VERIFY_EDD_CERT:
         requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
 
     ############################################################################################
@@ -118,15 +119,20 @@ def main():
     ##############################
 
     years = range(args.start, args.end)
-
-    edd_login_details = session_login(EddSessionAuth, EDD_URL, 'EDD', username_arg=args.username,
-                                      password_arg=args.password, user_input=input_timer,
-                                      print_result=True, verify_ssl_cert=VERIFY_EDD_CERT,
-                                      timeout=EDD_REQUEST_TIMEOUT)
+    timeout = settings.EDD_REQUEST_TIMEOUT
+    edd_login_details = session_login(EddSessionAuth, settings.EDD_URL, 'EDD',
+                                      username_arg=username,
+                                      password_arg=password, user_input=input_timer,
+                                      print_result=True, verify_ssl_cert=settings.VERIFY_EDD_CERT,
+                                      timeout=settings.EDD_REQUEST_TIMEOUT)
     edd_session_auth = edd_login_details.session_auth
 
-    edd = EddApi(base_url=EDD_URL, auth=edd_session_auth, verify=VERIFY_EDD_CERT)
-    edd.timeout = EDD_REQUEST_TIMEOUT
+    # remove password from memory ASAP
+    password = None
+    edd_login_details.password = None
+
+    edd = EddApi(base_url=settings.EDD_URL, auth=edd_session_auth, verify=settings.VERIFY_EDD_CERT)
+    edd.timeout = settings.EDD_REQUEST_TIMEOUT
 
     # query EDD for studies created each quarter within requested years
     time_zone = tz.gettz(args.timezone)

diff --git a/edd/rest/scripts/maintain_ice_links.py b/edd/rest/scripts/maintain_ice_links.py
@@ -35,11 +35,11 @@
 from logging.config import dictConfig
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
 from requests.exceptions import HTTPError
-from urlparse import urlparse
+from urllib.parse import urlparse
 
 from jbei.rest.auth import EddSessionAuth, IceSessionAuth
-from jbei.rest.clients import EddApi, IceApi
-from jbei.rest.clients.ice import Strain as IceStrain
+from jbei.rest.clients.edd.api import EddApi
+from jbei.rest.clients.ice.api import IceApi, Strain as IceStrain
 from jbei.rest.clients.ice.api import ICE_ENTRY_TYPES
 from jbei.rest.utils import is_url_secure
 from jbei.utils import to_human_relevant_delta, UserInputTimer, session_login, TYPICAL_UUID_PATTERN