Search Results for author:
Found 2 papers, 1 papers with code
Zamba: A Compact 7B SSM Hybrid Model
Zamba is pretrained in two phases: the first phase is based on existing web datasets, while the second one consists of annealing the model over high-quality instruct and synthetic datasets, and is characterized by a rapid learning rate decay.
BlackMamba: Mixture of Experts for State-Space Models
In this paper, we present BlackMamba, a novel architecture that combines the Mamba SSM with MoE to obtain the benefits of both.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
