GitHub Trending

jundot/omlx — LLM inference server with continuous batching & SSD caching for Apple Silicon —

32 sec read286 views0 listens

7 stars | 0 forks | Python

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

What it does

oMLX is a powerful LLM inference server designed for Apple Silicon, enabling continuous batching and SSD caching directly from the macOS menu bar. It simplifies the management of local LLMs, making them practical for real coding tasks.

Why it matters: Transform your Mac into an efficient LLM inference server with oMLX — seamless, powerful, and user-friendly!

View on GitHub

Want to create content about this repo? Use Nemati AI tools to generate articles, tutorials, and social posts.

You Might Like

GitHub Trending3 days ago34 sec read

vllm-project/vllm — A high-throughput and memory-efficient inference and serving engine for LLMs

94,317 stars | 23,654 forks | Python A high-throughput and memory-efficient inference and serving engine for LLMs What it does vLLM is a high-throughput and memory-efficient inference and serving engine for Large Language Models (LLMs). It's designed...

Ali Nemati

GitHub Trending4 days ago46 sec read

headroomlabs-ai/headroom — Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20

76,077 stars | 5,913 forks | Python Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server. What it does Headroom is a p...

Ali Nemati

GitHub TrendingAug 1534 sec read

MakazhanAlpamys/Soup — Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB lapto

1,535 stars | 238 forks | Python Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU. What it does Soup simplifies fine-tuning large language models with a single YAML configuration, enabling efficient training on li...

Ali Nemati