shimmy

Shimmy — Local Inference, OpenAI-Compatible

🔒 Local GPU inference — one binary, 100% Rust 🚀

[License: MIT] [Security] [CI] [Crates.io] [Downloads] [Rust] [GitHub Stars] [💝 Sponsor this project] [Trans rights] [LGBTQ+ friendly]

Languages: 简体中文 · 繁體中文

Shimmy is independently maintained and free forever. Sponsorship funds certification, compatibility work, and releases.

Shimmy will be free forever. No asterisks. No "free for now." No pivot to paid.

What Is Shimmy?

Shimmy is a single-binary OpenAI-compatible inference server for GGUF models. Point your existing AI tools at Shimmy and they just work — locally, privately, and free.

Shimmy is the server. Airframe is the engine. Under the hood, Shimmy runs on Airframe (v0.4.4), a pure-Rust WebGPU (WGSL) transformer engine. No C++ toolchain, no Python runtime, no backend flags. 26 models certified across 12 families. Version history: CHANGELOG · Airframe CHANGELOG.

Why this matters:

• No Python runtime or C++ toolchain — Rust only, top to bottom
• F32 accumulation precision with deterministic output (same model + seed + params → same output)
• WGSL compute shaders via WebGPU — NVIDIA, AMD, Intel, integrated GPUs, Apple Silicon
• Model spec auto-derived from GGUF metadata — no hardcoded per-model constants
• YaRN RoPE scaling for extended context via SHIMMY_MAX_CTX (see Extended Context)

🎯 Supported Models

12 model families · 26 certified model/quant combinations — every model below passes Shimmy's 3-box certification regimen (MATH + INFERENCE + DETERMINISM) against the certification ledger. Certification applies to the named model/quant combination; architecture recognition does not automatically mean certification. GGUF files load as-is; no recompilation, no hardcoded per-model constants.

SafeTensors format (.safetensors) is supported for model loading via safetensors_native. Full Airframe-native inference for SafeTensors remains roadmap work; see docs/v2-roadmap.md.

Features

• ⚡ TurboShimmy INT4 KV Cache — About 7× lower KV-cache memory in tested configurations. Run Llama-3.2-3B on 4 GB GPUs.
• 🚀 OpenAI SDK Compatibility — Chat completions, text completions, streaming, and model endpoints. Works with OpenAI SDKs and tools using that surface.
• 🔧 Extended Context — YaRN RoPE scaling via SHIMMY_MAX_CTX.
• 📦 Migrating from v1.x — llama.cpp, MLX, HuggingFace, and RustChain backends removed in v2.0+. Shimmy is now a pure Airframe product.
• 🏆 Certification — Every model passes a 3-box certification regimen (MATH + INFERENCE + DETERMINISM). See docs/CERTIFICATION.md.
• 🧠 MOE support — Mixture-of-Experts CPU offloading is on the Airframe roadmap.

Quick Start

Local loopback access needs no key. Network binds require SHIMMY_API_KEY; browser clients must also be listed in SHIMMY_ALLOWED_ORIGINS. See the network-access instructions.

cargo install shimmy --locked
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435

Then in another terminal:

shimmy list --short
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'

Full install, model acquisition, GPU, VRAM sizing, platform-specific builds: docs/quickstart.md

Documentation

Development Testing

Shimmy maintains high code quality through comprehensive testing:

# Full test suite (default features = GPU engine)
cargo test --release --locked

# Quick CPU-only tests (no GPU required)
cargo test --release --locked --no-default-features --features fast

See docs/ppt-invariant-testing.md for technical details.

Community & Support

• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📖 Security: Security Policy

🚀 Momentum Snapshot

🌟 [GitHub stars] stars and climbing fast ⏱ <1s startup 🦀 100% Rust, no Python

📰 As Featured On

🔥 Hacker News · Front Page Again · IPE Newsletter

Companies: Need invoicing? Email michaelallenkuykendall@gmail.com

Performance

Measured on RTX 3060, Shimmy v2.6.0, TinyLlama-1.1B. Your results vary by hardware.

Sponsor Shimmy

Shimmy is independently maintained. Sponsorship funds certification, compatibility work, and releases.

• $5/month: Coffee tier ☕ — Sponsor badge + name in SPONSORS.md
• $25/month: Supporter 🐛 — Priority support + name in SPONSORS.md
• $100/month: Corporate backer 🏢 — Logo placement + release recognition
• $500/month: Infrastructure partner 🚀 — Office hours + roadmap consultation

Current sponsors: ZephyrCloudIO · alistairheath

🎯 Become a Sponsor · Invoicing

License & Philosophy

MIT License — see LICENSE. Shimmy will be free forever.

Promise: This will never become a paid product.

Shimmy is infrastructure: it should be invisible. Reliability through comprehensive validation and property-based testing.

Maintainer: Michael A. Kuykendall · Mission: Making local model inference simple and reliable

Support

This project is a safe space. Trans rights are human rights.

If you or someone you love needs support:

• The Trevor Project — 24/7 for LGBTQ+ young people. Call 1-866-488-7386 or text START to 678-678
• Trans Lifeline — peer support run by and for trans people. US: 877-565-8860
• 988 Suicide & Crisis Lifeline — call or text 988