Search Results for author:
Found 2 papers, 1 papers with code
Vidur: A Large-Scale Simulation Framework For LLM Inference
Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates the end-to-end inference performance for different workloads by estimating several metrics of interest such as latency and throughput.
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
