TODO List for 0.1 release

[pavanky]

Base Classes

• nn::Module
• autograd::Variable
• Solver
• nn::Loss

Autograd

• Broadcasting (sum, tile)
• Math (exp, pow, abs, sqrt, log)
• Binary (add, subtract, multiply, divide, min, max, logical operators)
• matrix multiplication
• Convolutions and strided Convolutions
• Indexing and assignment

Neural Network

• Linear
• Convolve
• Pooling (Min, Max, Average)
• Activations: Sigmoid, Tanh, ReLU and related
• Recurrent neural networks (RNN, LSTM, GRU)
• Various Losses
• Containers (Sequential)
• Initializers

Solvers / Optimizers

• SGD
• ADAM

Examples

[jramapuram]

@pavanky : It doesn't necessarily have to be a classifier though.
Neural networks can easily do regression type problems as well.
Only difference would be the loss function used. I.e. L2 loss(regression) vs. Cross Entropy+Softmax(classification).

[jramapuram]

I think the following would be helpful from an API standpoint:

struct Model {
  int add(Layer layer_type);
  int compile(Optimizer opt, Loss loss, int max_iter = 200, bool early_stop = 1);
  float fit(DataSet train_data, DataSet target_data, std::tuple<DataSet, DataSet> validation = std::nullptr, float validation_split = 0.0f);  // for batch training up till either max_iter or early_stop if it is set. Has the option of either accepting cross validation data or splitting given data with ratio
  float train(DataSet train_data, DataSet target_data); // single step for online methods (can be called from fit)
  DataSet predict(DataSet test_data); // for evaluating new data
}

All the int's are return codes in the above.

This will give maximum flexibility in:

1. Layer creation
2. Model training
3. Online + batch setting

The Layer class as you mentioned should do the following:

struct Layer{
  int connect(Layer prev); // to connect to previous layer in a deep net
  DataSet derivative(DataSet x);
  DataSet forwardPass(DataSet x);
  DataSet input(); //merely returns the input that was last used (or the output of the previous layer)
  Weights weights(); //returns just weights
  Bias bias(); // returns the bias (or stack of bias') if any (otherwise std::nullptr maybe? )
  std::map<std::string, std::string> conf(); //getter to return config of the layer itself.
}

[futurely]

According to many recent researches, the most powerful neural networks are no longer stacked layers but rather arbitrarily complex graphs, e.g. the bunches of advanced recursive neural networks, Facebook AI research's Memory Networks, Google DeepMind's Neural Turing Machine etc. So node is a more general name than layer.

connect can't reflect the direction of connections between the nodes. Except for the nodes that connect to the input data, the other nodes' input are the output of their predecessor nodes. Caffe uses Google Protocol Buffer to define and serialize the network. Apache Thrift which was open sourced by Facebook supports many more languages.

The following API is inspired by Caffe's Blob, Layer and Net.

typedef  shared_ptr<array> ArrayPtr;

class Data {
  public:
    explicit Data(vector<int>& size);
    int nDimension() const;
    vector<int> size();
    // Caffe exposes the raw CPU/GPU pointers to use in BLAS functions.
    // array has high level API. So there's no need to to so.
    ArrayPtr weights() const;
    ArrayPtr gradients() const;
}

typedef shared_ptr<Data> DataPtr;
typedef vector<DataPtr> DataPtrVec;

class Node {
 public:
  explicit Node(NodeParam& nodeParam);
  virtual ~Node();
  // Calls initNode which subclass can override
  void init();
  // Input and output are more general than the top and bottom of Caffe
  virtual void forward(const DataPtrVec& input, 
      const DataPtrVec& output);
  // propagate_back is more general than propagate_down of Caffe
  virtual void backward(const DataPtrVec& input,
      const vector<bool>& propagate_back,
      const DataPtrVec& output);
  // The model is DAG(Directed Acyclic Graph)
  // it's more intuitive for the predecessor to add the successor
  void addSuccessor(Node& node);
  void addSuccessors(vector<Node>& nodes);  
 protected:
  virtual initNode();
};

// Dtype is float or double
template <typename Dtype>
class Graph {
 public:
  explicit Graph(GraphParam& graphParam);
  virtual ~Graph();
  virtual forward(const DataPtrVec& inputs, DataPtrVec* outputs,
      Dtype* loss = NULL);
 /**
   * (Caffe) The network backward should take no input and output, since it solely
   * computes the gradient w.r.t the parameters, and the data has already been
   * provided during the forward pass.
   */
  virtual backward();
  Dtype forwardBackward(const DataPtrVec& inputs) {
    Dtype loss;
    DataPtrVec outputs;
    forward(inputs, &outputs, &loss);
    backward();
    return loss;
  }
};

[futurely]

Microsoft Research's "Computational Networks A Generalization of Deep Learning Models" presented its open source deep learning framework CNTK as

[jramapuram]

I like the generalizations. Few notes:

  void addSuccessor(Node& node);

seems redundant with:

  void addSuccessors(vector<Node>& nodes);

since a node can have subnodes as well, right?
Why not just have a ptr to the next and previous nodes: a basic doubly-linked list.

One item that is very important that your schema is missing is some form of accuracy.
99% of the time you will need some form of data splitting and verification vs. a cross-validation data set.
It would be a good idea to incorporate this right off the back.

[pavanky]

@jramapuram

If I am reading correctly, the following is for cases when a single node is connected to multiple successors (like the CN with shared params diagram).

void addSuccessors(vector<Node>& nodes);

I am not sure a linked list is the solution here.

[pavanky]

@futurely Thanks for the great feedback! The proposed API looks solid. However the one issue I see is that going the Node / Graph route means that the amount of parallelism will be decreased in networks that use more traditional layers. I wonder if we can specialize a Layer class as well that sits on top of Node to achieve this.