Search Results for author:
Found 8 papers, 3 papers with code
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension.
Base of RoPE Bounds Context Length
We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability.
Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge
Consequently, the GPU spends most of its time on memory transfer instead of computation.
Checkpoint Merging via Bayesian Optimization in LLM Pretraining
The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs.
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters.
Baichuan 2: Open Large-scale Language Models
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering.
ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks
In compositional question answering, the systems should assemble several supporting evidence from the document to generate the final answer, which is more difficult than sentence-level or phrase-level QA.
