Record 043 · Runtimes · 2023
67%23.43.U
vLLM
vLLM team
“The serving engine that made a single box look like a proper LLM API.”
PagedAttention, continuous batching, OpenAI-compatible endpoints. Homelabs and offices run vLLM when they outgrow a single chat window.
Opened high-throughput local serving.
- PagedAttention
- OpenAI-compatible server
- Multi-GPU local serve
A local server that feels like an API farm.
