1 code implementation • 22 Apr 2024 • Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, Mingjie Tang
We also propose a new high-throughput framework to alleviate the computation and memory bottlenecks during the training and inference of MOE models.