u/Hairy_Goose9089

r/Vllm r/learnmachinelearning

Scaling my LLM inference for reply suggestions using disaggregated prefill

▲ 15 r/Vllm+1 crossposts

Scaling my LLM inference for reply suggestions using disaggregated prefill

Sharing my learning from separated prefill and decode into separate stages to increase the processing throughput and reducing TTFT significantly

saraswatmks.github.io

u/Hairy_Goose9089 — 6 days ago

▲ 1 r/learnmachinelearning

Deep dive into fine tuning a reranker using lora on Phi2 LLM

I tried doing ablation study to understand what layers, gradient norms contribute more to the reranking ability of the model

saraswatmks.github.io

u/Hairy_Goose9089 — 3 months ago