hIPPYLearn : an inexact Stochastic Newton-CG method for training neural networks

In recent years, neural networks, as part of deep learning, became pop- ular because the ability to extract information from data and generalize it for new input. More and more classic problems get better solutions with help of neural network. One example is Google’s AlphaGo, built with neural network, beat Lee Sedol, a Go champion. In this paper, we study the use of Newton methods to train neural network. Our algorithms are implemented in hIP- PYLearn, a new package based on TensorFlow, the Google machine learning software. Newton CG demonstrates the improvement of speed and accuracy over steepest descent on solving neural network. In this report, we also compare stochastic Newton CG method with batch Newton CG method. Stochastic Newton CG shows great improvement in speed at the cost of a small loss in accuracy. The choice of the optional amount of regularization is also discussed. We discovered that a good selection of β value will speed up training, avoid overfitting and therefore lead to good accuracy.