Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing

no code implementations • 8 May 2024 • Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, Zhi-Qin John Xu

Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate.