Cross-exchange OHLC management

[zbarge]

We need to read, write, analyze, and assign tasks based on OHLC data.

Fields

• Open
• High
• Low
• Close
• Volume
• Time
• ExchangeID

I/O Formats

• API (json) (in only)
• .csv
• sqlalchemy

Timeframes
Ideal scenario is making the root data set 5 minute candles that can flexibly transform to 15, 1hr, 4hr, daily candles. Most exchanges have downtime so we'd need to be able to smooth over gaps in time between candles using this scenario.

2nd scenario is downloading separate sets of data from API for each time frame, then updating the data source(s) with new data as it became available. On an interval we'd check what data we have, request new data to get us current, then update data.

Abstraction

• Make hook system for exchange objects to register formats and methods.
  • Time Conversion to UTC integer
  • Method to request new data (as pandas.dataframe)
  • Field name mapping to 'open', 'high', 'low', 'close', 'volume'
  • Max span of 5 minute candles that can be requested at once
  • Number of requests allowed per second.

Task Management
A user should be able to register OTask objects to the OHLC management system.

OTask object requirements:

1. Be able to run concurrently with other tasks.
2. Accept OHLC data in a pandas.DataFrame for one or more product/exchanges
3. Carry properties to store last run and trigger timestamps
4. Generate signals when triggered

Example Task:
Calculate 5min, 1min, 4hr RSI and moving averages
If any indicator falls within a certain threshold send an alert to an email address.

Example Process Flow

1. Check data storage for newest data
   • If no data: begin a data accumulation process going back X (3-6?) months.
   • If data is current: do nothing
   • If some data missing - request new data for missing timespan
2. Format new data and append to data storage(s)
3. Process registered tasks on data