Statefulness is critical

[mklasby]

Some progress to report. A few months ago, we had developed models that used dataset wide normalization (ie., min/max scale the entire training set at once). These models performed relatively OK on near term predictions (about 1 year from training period), but were totally incapable of predicting accurately many years into the future or during unusually high volatility events.

The first major insight was to use dynamic windowed normalization. That is, we normalize each window with respect to its lag period. We have been using 100-day lag windows in this case. As you can see below, we can follow the evolving mean much more accurately. However, this process excessively smoothes the predictions, so we end up missing the stock volitility.

The final piece was that we realized keras' LSTM network has some default parametres that are not well suited to very long data series. Given that the stock data we are using consists of very long seqeuences, it is critical that you DO NOT ALLOW keras to reset the state of the network between batches.

Using online learning, batch size = 1, with statefulness=True greatly improves model performance. See below:

Please review the following for more info:

• https://keras.io/guides/working_with_rnns/#crossbatch-statefulness
• https://machinelearningmastery.com/use-different-batch-sizes-training-predicting-python-keras/
• https://stackoverflow.com/questions/43702481/why-does-keras-lstm-batch-size-used-for-prediction-have-to-be-the-same-as-fittin

[stancsz]

@rmsouza01 ^fyi we got some progress with the initial time-series prediction

[rmsouza01]

@mklasby : curves look nice. I will look into the code carefully during the coming days. Can you remind me how many days ahead you are trying to predict the stock price?

[mklasby]

For this particular example, we are only predicting tomorrow's closing price. We are also developing models to predict the i^th day high and low prices., ie., what is the highest closing price in the next 7 days.

As you can imagine, performance degrades rapidly as the prediction window in increased. But the further out we can predict, the more actionable our insights in terms of actually executing a trade.

Currently, this model only exists in collab. So don't worry about code review for now. We are going to be pulling it all down into example notebooks for the github and of course, our final report for you.

[rmsouza01]

@mklasby : Thanks for clarifying. It seems you fixed the smoothing problem. One thing that people often do is to predict the 7-day average as opposed to daily predictions. You can always recover the daily predictions after prediction by solving a simple linear system of equations.

[mklasby]

Discussion updated to log the history of our results