Neural Network

This topic contains the following sections:

This section describes neural network models. Library supports next neural network types:

Multi-layer perceptron: classical multi-layer feed-forward network;
Deep Neural Network: the multi-layer feed-forward network with special train algorithm;
Autoencoder: one hidden layer network learned in an unsupervised manner.

Activation function

The neural network model in the library utilizes the fast sigmoid activation function in all except last layers: NN Activation Fast .

The last layer uses linear activation function: NN Activation Linear .

Loss Function

By default the neural network models uses Cross Entropy for loss function estimation:

NN Loss Cross Entropy

Also library allows to user set custom loss function. (See the CustomLoss property.)

Regularization

By default model training procedure uses L1 regularization with default weight equals to 0.0001.

Termination Function

The default model stops training if the best objective value do not improves on more than 0.00001 in last 100 train steps.

But the user can adjust them with any OR combination of the next conditions:

Either TerminationIterations are have passed;
Either TerminationTimeout time have passed;
The TerminationObjectiveChange conditions are met: objective function best value do not improves on more than Epsilon value in last Interval steps.

In any case user can specify his own train stop delegate function via the TerminationDelegate property.

Implementation

Next most important properties are featured in the FFNet class:

Property	Description
Classes	The list of classes in the model
Topology	The neural network model topology
Regularisation	Neural network L1 regularization parameter.

Termination conditions properties:

Property	Description
TerminationIterations	The maximum optimization iterations number. Zero means infinite iterations.
TerminationTimeout	The optimization algorithm timeout: the execution will be interrupted on the next step after the timeout expiration.
TerminationObjectiveChange	Objective function change stop criterion. Null means no objective change termination conditions.
TerminationDelegate	Custom stop criterion function.

Train statistics properties:

Property	Description
TrainBestValue	Last Train or Update operation best objective value.
TrainBestIteration	Last Train or Update operation best step index.
TrainTotalIterations	Last Train or Update operation total steps number.