Date of Publication :9th August 2017

Abstract: — Batch normalization is a boon to the training of a deep neural network. It acts as a panacea to the problem of internal covariate shift and facilitates the usage of higher learning rates. It also accounts for the inclusion of saturating non-linear functions, while excluding the need of drop outs for regularisation. However, mini-batch normalization is not self-sufficient and comes with a few limitations such as inability to deal with non-i.i.d inputs and decreased efficiency with a batch size of one. In this paper, we explore normalization, the need for its optimization, and evaluate the optimization technique provided by researchers.

Reference :