GitHub Trending

vllm-project/vllm — A high-throughput and memory-efficient inference and serving engine for LLMs

34 sec read9 views0 listens

94,317 stars | 23,654 forks | Python

A high-throughput and memory-efficient inference and serving engine for LLMs

What it does

vLLM is a high-throughput and memory-efficient inference and serving engine for Large Language Models (LLMs). It's designed to make LLM serving faster, easier, and more cost-effective, especially for those with limited resources. Its innovative PagedAttention and other optimizations significantly improve performance and reduce memory usage.

Why it matters: 🚀 Supercharge your LLM deployments! vLLM delivers blazing-fast inference & efficient serving. Check it out! ➡️ vllm.ai

View on GitHub

Trending today with 98 new stars

Want to create content about this repo? Use Nemati AI tools to generate articles, tutorials, and social posts.

You Might Like

GitHub TrendingMar 332 sec read

LMCache/LMCache — Supercharge Your LLM with the Fastest KV Cache Layer

7,275 stars | 954 forks | Python Supercharge Your LLM with the Fastest KV Cache Layer What it does LMCache is a high-performance KV cache layer for LLMs that significantly reduces TTFT and increases throughput by efficiently caching reusable text acr...

Ali Nemati

GitHub Trending3 days ago46 sec read

headroomlabs-ai/headroom — Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20

76,077 stars | 5,913 forks | Python Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server. What it does Headroom is a p...

Ali Nemati

GitHub TrendingOct 1538 sec read

lightseekorg/tokenspeed — TokenSpeed is a speed-of-light LLM inference engine.

2,364 stars | 331 forks | Python TokenSpeed is a speed-of-light LLM inference engine. What it does TokenSpeed is a high-performance LLM inference engine designed specifically for agentic workloads, aiming for speed comparable to TensorRT-LLM and usab...

Ali Nemati