GitHub Trending

LMCache/LMCache — Supercharge Your LLM with the Fastest KV Cache Layer

32 sec read156 views0 listens

7,275 stars | 954 forks | Python

Supercharge Your LLM with the Fastest KV Cache Layer

What it does

LMCache is a high-performance KV cache layer for LLMs that significantly reduces TTFT and increases throughput by efficiently caching reusable text across various storage mediums, thus saving GPU cycles and reducing user response delays.

Why it matters: 🚀 Supercharge your LLM with LMCache - the fastest KV cache layer that slashes TTFT by up to 10x! 🚀 #LLM

View on GitHub

Trending today with 140 new stars

Want to create content about this repo? Use Nemati AI tools to generate articles, tutorials, and social posts.

You Might Like

GitHub Trending2 days ago34 sec read

vllm-project/vllm — A high-throughput and memory-efficient inference and serving engine for LLMs

94,317 stars | 23,654 forks | Python A high-throughput and memory-efficient inference and serving engine for LLMs What it does vLLM is a high-throughput and memory-efficient inference and serving engine for Large Language Models (LLMs). It's designed...

Ali Nemati

GitHub Trending3 days ago46 sec read

headroomlabs-ai/headroom — Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20

76,077 stars | 5,913 forks | Python Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server. What it does Headroom is a p...

Ali Nemati

GitHub TrendingOct 1538 sec read

lightseekorg/tokenspeed — TokenSpeed is a speed-of-light LLM inference engine.

2,364 stars | 331 forks | Python TokenSpeed is a speed-of-light LLM inference engine. What it does TokenSpeed is a high-performance LLM inference engine designed specifically for agentic workloads, aiming for speed comparable to TensorRT-LLM and usab...

Ali Nemati