Local LLMs, page 2

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

138 stars+6 this week

An open-source inference engine for AI functions in SQL

183 stars+5 this week

Open-source iOS app connecting Meta Ray-Ban smart glasses to AI — 5 backends (on-device MLX models, Apple Intelligence, OpenAI, Gemini Live, OpenClaw), on-device neural voice, face recognition & live web search. Private and offline-capable.

196 stars+8 this week

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

221 stars0 this week

AI-powered data analysis workbench built for transparency, reproducibility and user control. Inspect the code and evidence behind results, verify calculations through replay, and choose from a wide range of models, including local open-weight models.

844 stars+6 this week

AI, fully local on your computer.

149 stars+3 this week

Open‑WebUI Tools is a modular toolkit designed to extend and enrich your Open WebUI instance, turning it into a powerful AI workstation. With a suite of over 15 specialized tools, function pipelines, and filters, this project supports academic research, agentic autonomy, multimodal creativity, workflows, and more

838 stars+6 this week

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

268 stars+7 this week

Your AI intranet: network the computers you already own for inference and training.

276 stars+2 this week

Run large AI models across your computers and servers. Powered by Colibri. Pure C.

218 stars+7 this week

Open-source local AI workspace — advancing on-device inference.

403 stars0 this week

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

6,015 stars+9 this week

Open-source desktop AI agent: built-in free model, local GGUF models via llama.cpp, dispatch tasks from WeChat/Feishu/DingTalk. Files, browser & MCP tools. | 开源桌面 AI 智能体：内置免费模型，llama.cpp 本地 GGUF 模型，微信/飞书/钉钉直接派任务。

162 stars0 this week

Granite Switch — Build AI models like you build software

122 stars+2 this week

A Web Interface for chatting with your local LLMs via the ollama API

1,270 stars+2 this week

Next-gen AI-native server ops panel with an in-workspace SRE agent. Crash self-healing, safe rollbacks, Docker/game servers, and local Ollama support.

184 stars0 this week

Local-first desktop AI: chat with your files, turn one line into a living 3D scene, investigate logs and harden configs. Runs on your laptop, works offline.

380 stars+7 this week

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

100 stars+1 this week

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

3,725 stars+1 this week

Local-first desktop AI workbench for roleplay, multi-character chat, long-form writing, RAG, MCP tools, plugins, and local models.

145 stars0 this week

Go manage your Ollama models

1,846 stars+5 this week

RESTai is an AIaaS (AI as a Service) open-source platform. Supports many public and local LLM suported by Ollama/vLLM/etc. Precise embeddings usage, tuning, analytics etc. Built-in image/audio generation with dynamic loading generators. Live chat deployment. Built-in block based graphical language. Prompt versioning and much more...

517 stars0 this week

LLM frontend for roleplay and creative writing

158 stars+6 this week

Not stated

167 stars0 this week

Large-scale LLM inference engine

1,907 stars+6 this week

Open-source control panel for the real llama.cpp: see where a model lands in your VRAM before loading, every llama-server flag per model, one-click llama.cpp updates, several models at once, and Embers (local agents whose quotes are checked by code). Windows · Linux · macOS.

121 stars+1 this week

Intent as Code: say what you want done, keep it as a file. Checked before it runs, run with the model you choose, proven afterwards. Rust engine, CLI/TUI and TypeScript SDK. AGPL-3.0 🦋

100 stars+3 this week

AMD MIVisionX is a computer vision toolkit built around a highly optimized, conformant open-source implementation of the Khronos OpenVX™ 1.3.2 specification. As of the 4.0.0 release, MIVisionX ships three components: the AMD OpenVX™ engine, the AMD RPP OpenVX extension, and the RunVX graph executor — across CPU, HIP, and OpenCL backends.

218 stars0 this week

From messy files to a checked report. A free, open-source desktop app for macOS, Windows and Linux: write and analyse with AI from your own files, check each sentence and figure against them, and export to Word. Meeting notes too. Works with ChatGPT, Claude, Gemini or a local model through Ollama.

856 stars+7 this week

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free.

38,093 stars+130 this week

User friendly GUI for configuring and launching llama.cpp

139 stars+3 this week

PyTorch library for cost-effective, fast and easy serving of MoE models.

382 stars+3 this week

A local-first workspace for chat, private knowledge, artifacts, and isolated model-driven work. Self-hosted. Provider-flexible. Apache 2.0.

103 stars0 this week

Java Inference Tornado toolkit: High performance LLM inference & serving for the JVM on GPUs

295 stars0 this week

Your private, on-device personal AI assistant for macOS and Windows. Chat, create, search files, and connect tools with local models. Includes an OpenAI-compatible API. Opt-in Pro sees, remembers, reflects, and acts with your approval. No cloud or account.

145 stars+3 this week

Camelid: a Rust-native local inference backend with evidence-gated model compatibility.

221 stars+1 this week

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

306 stars+1 this week

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

337 stars+6 this week

✈️ Kubernetes-native platform for deploying and managing AI inference across multiple providers

103 stars+2 this week

Accurate, large-scale, and extensible simulator for LLM inference Systems

721 stars+3 this week

A programmable system for high-performance model serving

240 stars+1 this week

A small utility to use Ollama models in LM Studio (and vice versa via symlinking) without duplicating disk space

187 stars+6 this week

Dyno Lab — an AI safety, alignment and interpretability research workbench for Apple Silicon. Explore activations, probes, SAEs and interventions with local inference, a Python SDK, APIs and MCP.

153 stars0 this week

Feel how fast a local LLM runs before you buy the hardware

171 stars+2 this week

A small proof-of-concept language model (not an LLM) that aims to incorporate latent-space prediction, internal state using recurrent trace units, and byte-by-byte output, built with MLX.

327 stars0 this week

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

904 stars+2 this week

A self-hosted, integration-focused IPAM, independently developed with an operation flow familiar to phpIPAM users, deeply integrated with multiple DNS servers, LibreNMS, OPNsense, Proxmox VE, Wazuh, and a local LLM.

133 stars+6 this week

Your car as a chat-room agent: Raspberry Pi 5 + dashcam + local AI. CodeWatch's sibling for the garage.

305 stars+4 this week

Hive compute for coding agent swarms - expand your personal compute by adding on-demand agentic swarms

204 stars-1 this week

Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.

161 stars0 this week

Gumbel-Softmax post-training quantization for LLMs (1–3 bit scalar, INT/GGUF-compatible).

135 stars+6 this week

The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

477 stars+2 this week

A desktop app for running Large Language Models locally.

480 stars0 this week

Close-to-metal Python API for AMD Ryzen™ AI NPUs, built on MLIR-AIE: declare operators, compile them into graphs, autotune them, and run models entirely on the NPU.

155 stars+3 this week

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

3,132 stars+7 this week

Synthetic Autonomic Mind - An AI assistant for everyone.

132 stars0 this week

A native LLM client for iOS & Apple Watch. Run local GGUF models offline via llama.cpp, or connect to OpenAI/Claude/Gemini. Features local RAG, Model Context Protocol (MCP) tools, Siri Shortcuts, and cross-device sync. Built with Swift.

220 stars+3 this week

Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.

3,926 stars+3 this week

Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and 2-bit experts on vLLM v0.24.0

571 stars+5 this week

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

284 stars+1 this week

llama.cpp (GGUF LLMs) and llava.cpp (GGUF VLMs) for ROS 2

268 stars0 this week

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

146 stars+1 this week

Open-Source AI Camera Skills Platform, AI NVR & CCTV Surveillance. Local VLM video analysis with Qwen, DeepSeek, SmolVLM, LLaVA, YOLO26. LLM-powered agentic security camera agent — watches, understands, remembers & guards your home via Telegram, Discord or Slack. Pluggable AI skills. OpenAI, Google, Anthropic or local AI. Runs on Mac Mini & AI PC.

3,138 stars+7 this week

One app for both cloud LLMs and on-device LLMs.

110 stars+3 this week

Open source, local first AI medical agent for desktop and web.

110 stars0 this week

It won't fix your bug. It'll fix you. The debugger that hands the question back instead of the answer.

555 stars+6 this week

A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)

120 stars+2 this week

Small local-first agent kernel: bounded tools, reversible edits, auditable runs. CLI + Web, Ollama + compatible APIs.

120 stars0 this week

OrbitKV: A KV cache for vLLM and SGLang.

279 stars+6 this week

Real-time hardware and LLM inference monitoring — GPU, CPU, memory, and vLLM metrics streamed to a dashboard.

142 stars+2 this week

On-device LLM Inference Powered by X-Bit Quantization

319 stars+1 this week

llama.cpp in pure Go

176 stars+3 this week

ONNX neural network inference engine

360 stars+2 this week

Local LLM Testing & Benchmarking for Apple Silicon

217 stars+1 this week

A local-first workspace for documents, tasks and meetings on macOS — nested spaces, typed properties, relations and saved views, with transcription, Smart Minutes and an agent running entirely on-device via MLX on Apple Silicon

173 stars0 this week

A native macOS app that allows users to chat with a local LLM that can respond with information from files, folders and websites on your Mac without installing any other software. Powered by llama.cpp.

3,324 stars+2 this week

Local-first AI chat client combining SillyTavern-style character cards, LM Studio-style GGUF model management, Ollama/OpenAI-compatible APIs, and an agentic workspace.

115 stars+16 this week

Evidence-backed AMD Strix Halo local-AI setup and benchmarks: Qwen3.8, Ollama, llama.cpp, Vulkan/ROCm, large GGUFs, and cross-OEM results.

371 stars+2 this week

Translate entire books with AI you already have.

247 stars+1 this week

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记

13,251 stars+1 this week

LLM Benchmark for Throughput via Ollama (Local LLMs)

397 stars0 this week

Efficient LLM inference on Slurm clusters.

111 stars0 this week

Self-hosted vLLM made simple. Git-friendly YAML for models and profiles, full lifecycle from an intuitive CLI and TUI.

101 stars0 this week

Local AI chat for iPhone, iPad and Mac. Connect Ollama, LM Studio or any OpenAI-compatible server, or run MLX models on-device. Includes tools and web search.

557 stars+1 this week

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

450 stars0 this week

Personal AI Notebooks. Organize files & webpages and generate notes from them. Open source, local & open data, open model choice (incl. local).

3,639 stars+23 this week

On-device, real-time multimodal AI with features similar to GPT-Live

2,113 stars+2 this week

A framework for generating realistic LLM serving workloads

192 stars+1 this week

Setup and run a local LLM and Chatbot using consumer grade hardware.

354 stars0 this week

InferrLM - On-device AI for iOS & Android

114 stars+1 this week

LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.

173 stars0 this week

A ternary, zero-heap tiny language model that runs inside a $2 microcontroller — bit-exact Python <-> C99 <-> Cortex-M3 (QEMU) parity. Apache-2.0.

153 stars0 this week

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

756 stars+4 this week

QONNX: Arbitrary-Precision Quantized Neural Networks in ONNX

195 stars0 this week

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

124 stars0 this week

Lightweight, standalone, multi-platform, and privacy focused local LLM chat interface with optional encryption

165 stars+1 this week

Swift package to run local LLMs on iOS, macOS, Linux

219 stars0 this week

Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

101 stars+1 this week

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.

492 stars+1 this week

Local LLM Powered Recursive Search & Smart Knowledge Explorer

265 stars0 this week