PROJECT / Local AI / Model Routing
Local Model Gateway
An OpenAI-compatible gateway for routing requests across local and hosted language models.
Local AIModel RoutingBackend
- STATUS
- maintained
- YEAR
- 2026
- ROLE
- Maintainer
- STACK
- Go · Ollama · Redis · OpenTelemetry
SYSTEM / ARCHITECTURE
How it fits together
- 01Compatible API
- 02Policy router
- 03Provider adapters
- 04Response normalizer
- 05Telemetry exporter
- Providers
- 5
- Gateway overhead
- 14ms
- Fallback success
- 97%
Goal
Applications should be able to use a local model for private or inexpensive workloads without coupling business logic to a specific runtime.
- Compatible request format
- Policy-based routing
- Transparent fallback
Architecture
A small Go service normalizes requests, applies routing policy, invokes an adapter, and emits consistent usage and latency telemetry.
- Circuit breakers per provider
- Streaming passthrough
- Structured error taxonomy
Use
The gateway now supports local development, evaluation, and controlled hosted-model fallback through the same client configuration.