no code implementations • 15 Apr 2024 • Daniil Merkulov, Daria Cherniuk, Alexander Rudikov, Ivan Oseledets, Ekaterina Muravleva, Aleksandr Mikhalev, Boris Kashin
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation.
no code implementations • 2 Feb 2024 • Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Aleksandr Mikhalev, Ivan Oseledets
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture.
no code implementations • 6 Dec 2023 • Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets
LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers.
2 code implementations • 31 Jan 2022 • Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa, Julia Gusak, Daniil Merkulov, Ivan Oseledets
Also, we investigate the variance of the gradient estimate induced by the randomized matrix multiplication.