On the Optimization and Generalization of Multi-head Attention

no code implementations • 19 Oct 2023 • Puneesh Deora, Rouzbeh Ghaderi, Hossein Taheri, Christos Thrampoulidis

Finally, we demonstrate that these conditions are satisfied for a simple tokenized-mixture model.