Normalized gradient flow optimization in the training of ReLU artificial neural networks

no code implementations • 13 Jul 2022 • Simon Eberle, Arnulf Jentzen, Adrian Riekert, Georg Weiss

The training of artificial neural networks (ANNs) is nowadays a highly relevant algorithmic procedure with many applications in science and industry.

Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation

no code implementations • 18 Aug 2021 • Simon Eberle, Arnulf Jentzen, Adrian Riekert, Georg S. Weiss

In the second main result of this article we prove in the training of such ANNs under the assumption that the target function and the density function of the probability distribution of the input data are piecewise polynomial that every non-divergent GF trajectory converges with an appropriate rate of convergence to a critical point and that the risk of the non-divergent GF trajectory converges with rate 1 to the risk of the critical point.