Local LLMs

Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

755 stars+11 this week

Community maintained hardware plugin for vLLM on Apple Silicon

1,974 stars+32 this week

Run GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one dashboard, OpenAI- and Anthropic-compatible API.

2,712 stars+269 this week

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

7,406 stars+144 this week

Community maintained hardware plugin for vLLM on Huawei Ascend

3,063 stars+27 this week

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

2,010 stars+44 this week

An open, programmable decision layer for models and compute.

6,428 stars+52 this week

LLM Inference Engine for Metal, CUDA and Vulkan.

1,712 stars+106 this week

Laya is an open-source, local-first AI notification command center that aggregates Slack, Gmail, GitHub, Jira, Notion, Outlook, Calendar (and more) notifications using local LLMs via Ollama and LM Studio. Supports cloud models via BYOK.

809 stars+22 this week

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

23,475 stars+198 this week

Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.

938 stars+159 this week

TokenSpeed is a speed-of-light LLM inference engine.

2,895 stars+48 this week

AirLLM 2.8T LLM inference with single 4GB GPU

42,030 stars+420 this week

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

2,685 stars+18 this week

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

7,079 stars+29 this week

Hundreds of models & providers. One command to find what runs on your hardware.

38,802 stars+235 this week

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

2,034 stars+17 this week

A high-performance inference engine for AI models

3,244 stars+49 this week

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

44,240 stars+55 this week

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

11,136 stars+36 this week

ASR/STT subtitle generator. Uses Qwen3-ASR, local LLM, Whisper, TEN-VAD. Noise-robust for JAV

2,444 stars+26 this week

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

5,825 stars+107 this week

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

79,226 stars+278 this week

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

12,587 stars+8 this week

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

25,266 stars+27 this week

ReAlign models instantly, on demand. Per-request alignment for verified users and channels, or Always On. Use directly or with Hermes, OpenClaw and custom harnesses.

125 stars+6 this week

Kindi 0.4: local LLM inference on low-cost NVIDIA GPUs, Radeon 780M and CPUs. Bonsai 2B-27B, AH-Nanbeige4.3-33B and experimental Nemotron, with an OpenAI-compatible server.

148 stars+33 this week

Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Strata.

56 stars+31 this week

Btop-style ESP32-S3 touchscreen dashboard for vLLM and TensorFold inference servers

11 stars+5 this week

Source code of Local Code, a private AI agent for Mac, iPad and iPhone. Runs DeepSeek 4.1 on device, with Python, Node.js and a shell built into the app.

126 stars+24 this week

An easier way to use AI on your own computer. A free, open-source alternative to Claude and ChatGPT that runs open-weight models like Gemma and gpt-oss locally on Mac, Windows and Linux. Private, and works offline.

521 stars+58 this week

The open-source Cluely alternative for Mac: live answers from your own docs during video calls, hidden from screen sharing. Local Whisper, your own AI key.

11 stars+11 this week

llama.cpp fork that abliterates at inference

170 stars+8 this week

Native Apple Silicon inference engine for many AI agents at once: C++/Metal, agent-contexts API, 8-bit paged KV, SSD tier (Qwen3.8)

27 stars+27 this week

Offline emergency assistant: a local LLM and a 200 GB offline library on one Mac, with cited answers and no internet.

42 stars+38 this week

ChatGPT-style generative UI for Open WebUI: photo galleries, live maps with real routes, multi-day trips, budgets, calculators, live weather & stocks, link previews, and one-click Word/Excel/PowerPoint exports via ArtCraft. One file, no API keys.

65 stars+10 this week

PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.

156 stars+72 this week

Свой ИИ-чат на вашем ПК в один клик: подбирает модель под железо, ставит Ollama и модель. Windows. / One-click local AI chat for Windows: picks a model for your hardware, installs Ollama.

11 stars+7 this week

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

290 stars+8 this week

Fast Qwen3.8-27B for one RTX 3090. 200k context, 100+ tok/s. Custom CUDA decode engine, OpenAI-compatible API.

24 stars+24 this week

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

364 stars+101 this week

A MoE inference engine you can read in an afternoon.

29 stars+19 this week

A high-throughput and memory-efficient inference and serving engine for LLMs

96,588 stars+439 this week

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

1,672 stars+9 this week

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

659 stars+14 this week

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

7,006 stars+55 this week

Bonsai Demo

3,498 stars+23 this week

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

12,310 stars+54 this week

Nativ is an open-source local AI app for Mac. It runs open AI models on Apple Silicon, and nothing leaves your Mac.

1,581 stars+10 this week

A self-hosted job search and resume builder app that writes tailored resumes and cover letters from your profile.

4,872 stars+646 this week

A local inference engine for Apple silicon, built around the model.

2,159 stars+121 this week

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

2,626 stars+136 this week

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

734 stars+31 this week

Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.

195 stars+8 this week

Desktop app for game dev with AI: make games in Unreal Engine, Unity, Godot or the browser. The agent sets up the engine, builds your game and play-tests it by itself.

811 stars+27 this week

Atomic Agent is a local-first AI agent. Runs open-weight models on your own machine via llama.cpp.

3,340 stars+1 this week

Run any agent on any model, models stored as OCI images

585 stars+5 this week

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability

667 stars+15 this week

Rewrites AI drafts so they read like a person wrote them — runs locally, keeps the facts

823 stars+121 this week

Local-first healthcare AI: clinical NER and HIPAA PII de-identification on hardware you control. 2,200+ medical models, 35 model-backed PII languages, and Python, MLX, Android and browser runtimes. Apache-2.0 SDK.

5,794 stars+28 this week

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

818 stars+24 this week

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

299 stars+85 this week

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

6,202 stars+21 this week

The Swiss Army Knife of Offline AI. Chat, see, speak, and generate images on your phone or Mac — GGUF LLMs, vision, Whisper speech-to-text, Stable Diffusion, tool calling, and local-network servers. Runs on your CPU, GPU, or NPU. No account, no API key, zero data leaves your device.

3,355 stars+41 this week

A mobile application designed to provide a beautiful, fast, and privacy-respecting interface for on device LLM and local LLM servers and cloud providers

311 stars+10 this week

jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.

952 stars+15 this week

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

637 stars+13 this week

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

1,173 stars+22 this week

Sudoless Apple Silicon system monitor (native SwiftUI GUI) with ANE / Media Engine / memory-bandwidth tracking

1,018 stars+5 this week

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

901 stars+26 this week

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

6,004 stars+53 this week

First-class Linux packaging for DeepSeek Harness desktop — AppImage, .deb, .tar.gz for x64/arm64, built by native CI.

188 stars+19 this week

Kyojin: open-source engine that runs 100 GB-class open models (GLM-5.3, Qwen3.8, DeepSeek V4.1) on one desktop machine. AMD Strix Halo, NVIDIA RTX and Apple Mac. OpenAI API, built on ExLlamaV3.

233 stars+5 this week

A vLLM-inspired inference engine for looped transformers, with loop-level scheduling and execution across composable stages

221 stars+2 this week

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.

178 stars+8 this week

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

1,698 stars+61 this week

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

16,454 stars+120 this week

Local inference for diffusion language models (LLaDA, Dream), written in Mojo

495 stars+285 this week

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

782 stars+24 this week

Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source

624 stars+18 this week

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

678 stars+97 this week

Fast, flexible LLM inference

7,821 stars+28 this week

Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

490 stars+41 this week

Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.

289 stars+7 this week

Open-Source Personal Cloud OS for Always-On Agents

5,355 stars+19 this week

Run large language models locally on Intel Macs with AMD GPUs - native macOS app with Metal acceleration

246 stars+12 this week

WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

1,371 stars+15 this week

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

7,321 stars+213 this week

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

9,647 stars+9 this week

Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

736 stars+2 this week

Run frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of code

8,509 stars+26 this week

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

1,651 stars+7 this week

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

479 stars+3 this week

dgx-spark-qwen38 is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.

252 stars+7 this week

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

405 stars+10 this week

An offline coding agent for local models. One command sets up the best model your GPU can run, and on 24 GB+ RTX 30/40/50-series cards Qwen3.8-27B runs up to 1.9x faster than llama.cpp.

139 stars+8 this week

Open-source AI meeting notes for Mac. Records mic + system audio with no bot, transcribes on-device or via OpenAI / Deepgram / Groq, identifies speakers offline, and writes summaries that fuse your notes with the transcript. Ask your notes and get cited answers. Tauri 2 + Rust + Swift.

341 stars+4 this week

Cross-platform on-device AI toolkit

519 stars+9 this week

One CLI for all your robots. Connect them, command them, and let them work together, each with an LLM for a brain (Claude, OpenAI, Gemini, Grok, or local via Ollama or vLLM), VLAs for arms and decision LLMs (Jev, Laya, Kev). Drives Microduck, Open Duck Mini, LeRobot, XLeRobot, AlohaMini, ToddlerBot, or any ROS 2 base from a laptop, offboard.

292 stars+4 this week

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

3,130 stars+8 this week

🧱 Experimental library for building inference engines

422 stars+45 this week

Run your AI models at home in one binary: chat, persistent memory, web access, browser control, MCP tools, encryption at rest and end-to-end encrypted remote access. Linux, macOS, Windows. FR/EN docs.

110 stars+2 this week

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

3,112 stars+11 this week

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

604 stars+15 this week

An easy-to-use, fast toolkit to scale up RL post-training on a single node.

386 stars+3 this week

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

11,134 stars+14 this week

React Native binding of llama.cpp

1,072 stars+7 this week

DeepSeek-V4-Flash-0731 284B inference in ~25 GB of RAM / Qwen3.8-Next-Flash inference in ~15 GB of RAM on any M-series MacBook

152 stars+1 this week

RAM efficient terminal agent harness for coding, cybersecurity, and automation.

146 stars+26 this week

Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.

209 stars+45 this week

A simpler self-hosted alternative to Open WebUI. Bring your own API keys or local models. Native Android and iOS clients.

151 stars+4 this week

Live truecolor terminal dashboard for a locally running LLM: tok/s, TTFT, GPUs, context, MTP acceptance, layer and real expert routing (llama.cpp)

157 stars+2 this week

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

252 stars+1 this week

High-performance In-browser LLM Inference Engine

19,365 stars+19 this week

Optimized GPU compiler for LLM inference. Choose from a list of optimized recipes or optimize your own model via kernel fusion, autotuning, and advanced scheduling. Run benchmarks across different GPU types and configurations, track results and share experiments with the community.

123 stars+1 this week

Benchmark local LLM serving by replaying real agent trajectories

129 stars+3 this week

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

738 stars+8 this week

The agent that monitors anything for you. Open-source, running completely locally.

1,771 stars+108 this week

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

1,394 stars+7 this week

vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm

913 stars+7 this week

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

1,258 stars+9 this week

Pure Rust Inference Engine

714 stars+1 this week

A top for local LLMs: memory, paging, GPU and tokens/s for oMLX, Ollama, llama.cpp, vLLM and more

148 stars+5 this week

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

8,935 stars+4 this week

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

3,825 stars+3 this week

Python package for LLM compression

445 stars0 this week

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

797 stars+4 this week

The world's smartest system-wide grammar assistant; a better version of the Apple Intelligence Writing Tools. Works on Windows, Linux, & macOS, with the free Gemini API, local LLMs, & more.

2,520 stars+46 this week

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

524 stars+1 this week

A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

1,618 stars+3 this week

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

3,000 stars+4 this week

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level

2,201 stars+4 this week

DevoxxGenie is an agentic plugin for IntelliJ IDEA that uses local LLM's (Ollama, LMStudio, GPT4All, Jan and Llama.cpp) and Cloud based LLMs to help review, test, explain your project code. Latest version now also supports Spec Driven Development with CLI Runners.

689 stars+2 this week

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

366 stars+2 this week

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.

347 stars+2 this week

Home Assistant LLM integration for local OpenAI-compatible services (llamacpp, vllm, etc)

307 stars+2 this week

The easiest way to use Ollama in .NET

1,413 stars+5 this week

Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.

1,562 stars+14 this week

The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

476 stars+1 this week

Next-gen AI-native server ops panel with an in-workspace SRE agent. Crash self-healing, safe rollbacks, Docker/game servers, and local Ollama support.

184 stars0 this week

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

136 stars+4 this week

An open-source inference engine for AI functions in SQL

181 stars+5 this week