MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

1 code implementation • 22 Apr 2024 • Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, Mingjie Tang

We also propose a new high-throughput framework to alleviate the computation and memory bottlenecks during the training and inference of MOE models.

Multi-Task Learning Quantization