iri Notes

Note

routellm-deep-dive

◈ Obsidian Startup March 17, 2026

RouteLLM — Deep Dive

Repo: https://github.com/lm-sys/RouteLLM Paper: ICLR 2025 — "RouteLLM: Learning to Route LLMs with Preference Data" Authors: UC Berkeley, Anyscale, Canva (LMSYS org — same people who built Chatbot Arena) Stars: 4,700+ | Forks: 360+ | Dependents: 137+ projects

What It Does

RouteLLM is a binary router: every incoming LLM call gets scored, then sent to either a "strong model" (GPT-4, expensive) or a "weak model" (Mixtral, cheap). The user sets a threshold α that controls the cost/quality tradeoff.

One sentence: Learns from human preference data which queries actually need the expensive model, and skips it when they don't.

Request flow

Incoming prompt
      ↓
Router scores complexity → "win probability for weak model"
      ↓
Is score > threshold α?
   YES → Weak model (cheap, fast)
   NO  → Strong model (expensive, quality)
      ↓
Response back to caller (transparent)

Works as a drop-in OpenAI client replacement — no code changes needed:

from routellm.controller import Controller

client = Controller(
    routers=["mf"],
    strong_model="gpt-4o",
    weak_model="gpt-3.5-turbo",
    threshold=0.5
)

Or run as an OpenAI-compatible server:

python -m routellm.openai_server --routers mf --config config.yaml
# point OPENAI_BASE_URL=http://localhost:8000

The 4 Router Types

Router How it works Speed Notes
Matrix Factorization (MF) ⭐ Learns hidden scoring fn s(M,q) — like a recommendation system Fast Best overall, recommended
Similarity-Weighted (SW) Embeds prompt, finds similar historical queries, votes Medium Good with historical data
BERT Classifier CLS token → logistic regression Fast Interpretable, easy to retrain
Causal LLM (Llama 3 8B) Full LLM predicts 1-5 quality score Slow Highest accuracy, high overhead

Winner: Matrix Factorization. Lightweight, best benchmarks, ~<10ms routing decision.

Training Data & Methodology

  • Source: 80,000 samples from Chatbot Arena (human preference labels)
  • Problem solved: Raw Arena has <0.1% cross-model comparisons → sparse signal. Fixed by clustering models into 10 Elo tiers.
  • Key finding on augmentation: Adding just 1,500 MMLU samples (2% of total data) + GPT-4-as-judge labels improved BERT classifier by 50%+ APGR. You don't need massive labeled data — small, well-chosen augmentation does most of the work.

Benchmark Results

Testing with GPT-4 (strong) + Mixtral 8x7B (weak):

Benchmark Cost savings Performance
MT Bench 85% 95% of GPT-4
MMLU 45% 92% of GPT-4
GSM8K 35% High quality

Best result: MF router with augmented data → 95% GPT-4 performance using only 14% GPT-4 calls (86% of calls go cheap)

vs. Martian (commercial):

  • Same MT Bench performance
  • RouteLLM: 29.66% GPT-4 calls
  • Martian: 50% GPT-4 calls
  • RouteLLM is >40% cheaper than Martian for identical quality

Overhead: <40ms per request. FAISS-based: microseconds.

Cross-Model Generalization

Routers trained on GPT-4 + Mixtral generalize to Claude 3 Opus + Llama 3 without retraining. The router learns query complexity, not model-specific features. This means a trained router works regardless of which model pair you swap in.

Key Gaps (Where to Beat Them)

  1. Binary routing only — exactly 2 models, no dynamic pool
  2. No local inference support — doesn't know about Ollama, llama.cpp, or on-device inference
  3. No system-state awareness — doesn't know battery level, GPU temperature, memory pressure, monthly budget
  4. No agent pipeline awareness — treats each call independently; can't optimize a LangGraph DAG holistically
  5. Static threshold — set once, doesn't adapt to runtime conditions or cost drift
  6. No retraining loop — concept drift degrades silently
  7. Cloud-only assumption — scoring step itself sends prompts over the wire; no privacy-preserving local scoring

Code Structure

routellm/
├── controller.py          # Main interface
├── openai_server.py       # Drop-in server
├── calibrate_threshold.py # Tune α threshold
├── routers/
│   └── routers.py         # All 4 routers + abstract base
├── evals/                 # Benchmarking
└── examples/

Pretrained weights: https://huggingface.co/routellm Install: pip install "routellm[serve,eval]"

Strategic Options — How We Position Against This

Option A: Build on top of RouteLLM

Use their MF router for complexity scoring. Add a local inference layer underneath:

  • When score says "weak model is fine" → route to Ollama instead of a cheap cloud model
  • Add system-state layer on top: throttle to local when battery/thermal constrained
  • RouteLLM handles the ML signal; we own the systems + local layer
  • Fast to build, academic credibility borrowed from ICLR paper

Option B: Replace RouteLLM's cloud assumption (our biggest moat)

  • Build the same classifier but treat local inference as first-class citizen
  • Routing isn't "strong cloud vs weak cloud" — it's "local vs cloud"
  • Privacy-preserving: complexity scoring model runs locally, prompt never leaves machine
  • This is where Hosung's C++ + power systems background creates a wall competitors can't easily climb

Option C: Go above RouteLLM — agent pipeline optimizer

  • RouteLLM optimizes individual calls
  • We optimize the full agent execution graph
  • Know what step we're on, cache intermediate states, batch similar sub-tasks
  • RouteLLM becomes a dependency we call internally, not a competitor
  • Bigger build (3 months to MVP) but much harder to replicate

Related Work

  • Router-R1 (2025): Multi-round routing via RL (ArXiv: 2506.09033) — shows next frontier is moving beyond binary routing
  • Martian: $9M raised, commercial router, less efficient than RouteLLM on benchmarks
  • OpenRouter: API marketplace — model selection only, no intelligent routing

Deep dive completed 2026-03-17