azrtydxb / Fastllm-proxy
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Why this job: says: openai compatible, llm router; topics: llm-gateway, ai-gateway, openai-api
What it does
- Cache-affinity routing with a load escape hatch. A shared prefix goes back to the node that already has its KV cache, unless that node is meaningfully hotter than the least-loaded one.
- Opaque response bodies. Upstream frames reach the client exactly as they arrived — never deserialised, never re-encoded, never buffered.
- Cache-affinity routing, frontend models, rule-based and semantic routing — see below.
- 81 providers, and any OpenAI-compatible endpoint. see below.
- RBAC with real API keys. Per-principal, per-model grants; keys hashed with SHA-256, passwords with Argon2id.
Get started
git clone https://github.com/azrtydxb/Fastllm-proxy && cd Fastllm-proxy
docker compose up -d
docker compose exec fastllm fastllm-proxy set-password --name you --password 'change-me' - Maintenance
- Active
- Last commit yesterday
- Stars
- 109
- On GitHub
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v0.3.6
- Released 11 days ago
- Built with
- Rust
- 79% of the code
Is it looked after?
- Contributors
- 2
- Last 90 days
- Issues
- 89% closed
- 3 still open
- Pull requests
- 56 merged
- 1 waiting
- Releases
- 3
- 3 in the last year
- Contributing guide (opens in new tab)
- Security policy (opens in new tab)
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| maximhq / bifrost | 9,461 | yesterday | Apache-2.0 | No downloads |
| Kong / kong | 44,369 | 22 days ago | Apache-2.0 | No downloads |
| ENTERPILOT / GoModel | 1,313 | 2 days ago | MIT | Mac, Windows, Linux |
More: languages, topics and project record
Languages
- Rust 79%
- JavaScript 17%
- Python 2%
- PLpgSQL 1%
Topics
- ai-gateway
- inference
- litellm
- llm
- llm-gateway
- load-balancing
- observability
- openai-api
- rbac
- reverse-proxy
- rust
- vllm
Project record
- JobModel gateways
- StatusListed
- Commits600
- Forks92
- Watching0
- Stars in 28 days+3
- CreatedAug 6, 2026
- First seen by usNov 7, 2026
- CheckedNov 10, 2026