Search Results for author:
Found 4 papers, 0 papers with code
Upper and lower memory capacity bounds of transformers for next-token prediction
Given a sequence of tokens, such as words, the task of next-token prediction is to predict the next-token conditional probability distribution.
Memory capacity of three-layer neural networks with non-polynomial activations
The minimal number of neurons required for a feedforward neural network to interpolate $n$ generic input-output pairs from $\mathbb{R}^d\times \mathbb{R}$ is $\Theta(\sqrt{n})$.
Memory capacity of two layer neural networks with smooth activations
In order to analyze general activations, we derive the precise generic rank of the network's Jacobian, which can be written in terms of Hadamard powers and the Khatri-Rao product.
Optimization and Learning with Information Streams: Time-varying Algorithms and Applications
Approaches for the design of time-varying or online first-order optimization methods are discussed, with emphasis on algorithms that can handle errors in the gradient, as may arise when the gradient is estimated.
