Search Results for author:
Found 9 papers, 3 papers with code
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers
Our theory shows that the predictor statistics are expressed as a sum of independent kernels, each one pairing different 'attention paths', defined as information pathways through different attention heads across layers.
Nonlinear classification of neural manifolds with contextual information
Understanding how neural systems efficiently process information through distributed representations is a fundamental challenge at the interface of neuroscience and machine learning.
Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization
"Forward-only" algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation.
Rigorous dynamical mean field theory for stochastic gradient descent methods
We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e. g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization.
Learning curves for the multi-class teacher-student perceptron
For Gaussian teacher weights, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality.
The effective noise of Stochastic Gradient Descent
In the under-parametrized regime, where the final training error is positive, the SGD dynamics reaches a stationary state and we define an effective temperature from the fluctuation-dissipation theorem, computed from dynamical mean-field theory.
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity.
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow.
The role of regularization in classification of high-dimensional noisy Gaussian mixture
We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.
