Switchyard
Open-source platform, built with Claude Code · Gateway in front of LLM inference servers
A gateway in Go, standard library only, that sits in front of LLM inference servers: it never sends a replica more than it can run, refuses work it cannot serve in time with a retry time instead of a timeout, and retries a failed request on another replica only while the client has seen nothing - measured on a real GPU against a plain load balancer.
Problem
A GPU runs only a few requests at once; past that, a plain load balancer keeps forwarding, the excess waits invisibly inside the servers, and almost every user ends up waiting behind requests that are already late.
Solution
Hold the queue in the gateway, never exceed a replica's slots, refuse at the door when a request would miss its deadline, and retry only while the client has seen nothing.
Built in
78 of 212 requests on time at 1.0/s against 6 for a plain proxy, no failures against 150; no client errors with a replica killed under load; a live dashboard, 47 tests and CI.
Overview
Switchyard is an open-source gateway I built with Claude Code for a few GPUs serving many users. An inference server can run only a few requests at once and queues the rest out of sight, where nothing can reorder or refuse them; a plain load balancer spreads that overload evenly and hides it everywhere. Switchyard keeps the queue in front of the servers, sends each replica only as many requests as it has slots, and refuses what cannot be served in time with a retry time. It is written in Go with no dependencies outside the standard library.
Measured on one RTX A4000 serving qwen3.5:9b through two replicas, Ollama and llama.cpp on the same weights, against a plain round-robin proxy fed identical open-loop chat traffic: at 1.0 requests a second the proxy got 6 of 212 requests a first token within 10 seconds and 150 failed; Switchyard got 78, failed none, and refused the rest in at most 3 ms. With a replica killed under load the proxy returned 15 errors to clients and Switchyard none, finishing 7 of the affected requests on the live replica. Routing conversations back to their cache made no measurable difference at this scale, and the write-up says so.
The first benchmark run found three bugs in the gateway, all of them making it look worse, and getting the second GPU replica running found a Docker and NVIDIA interaction that silently moved another service's model onto the CPU; both are fixed and written up. A live dashboard and the real-GPU results page are public, and every table is recomputed from the raw logs in the repository.
What it does
- Keeps the queue in front of the servers instead of inside them, ordered by priority, and refuses a request at the door when its expected wait is longer than its deadline: a 429 with Retry-After in milliseconds instead of a timeout a minute later
- Retries a failed or stuck request on another replica only before the first byte reaches the client; a stream that breaks mid-answer ends with an error event instead of another model's text spliced on
- Routes a conversation back to the replica holding its prompt cache, meters each API key in tokens as well as requests, and takes a failing replica out of rotation with a circuit breaker that probes it back in
- A live dashboard served by the gateway itself, Prometheus metrics, a simulated inference server for GPU-free tests, and a benchmark harness whose every table is recomputed from raw per-request logs
What's next
Hiring for a full-stack or Python role — or need something like this built? Let's talk.
I'm open to offers — on-site or hybrid in Moscow, or remote; full-time or contract. I reply within 48 hours — faster on Telegram.