A Mean Field View of the Landscape of Two-Layer Neural Networks - Andrea Montanari
From Scott Jacobson on October 19th, 2018
We consider a simple case, namely two-layers neural networks, and prove that –in a suitable scaling limit– SGD dynamics is captured by a certain non-linear partial differential equation (PDE) that we call distributional dynamics (DD). We then consider several specific examples, and show how DD can be used to prove convergence of SGD to networks with nearly ideal generalization error. This description allows to ‘average-out’ some of the complexities of the landscape of neural networks, and can be used to prove a general convergence result for noisy SGD.