Agent Architecture Parameters Real-Time Simulation
Input: $0.042/1M | Output: $0.00 | P50: ~35ms
Input: $3.00/1M | Output: $15.00/1M | P50: ~1,200ms
85% of decisions (tool routing, binary classification, safe guardrails) resolve directly at the decision layer. Only the remaining 15% complex edge cases escalate to the Frontier LLM.
⚡ 1-Click Production Router Code
Why Decision Models Outperform Mini LLMs
- ✓ Zero Output Hallucination: Non-autoregressive forward pass returns calibrated probability distribution over structured classes, not random tokens.
- ✓ Zero Output Token Cost: No autoregressive decoding loop means outputs are completely free ($0.00/M tokens).
- ✓ Single Forward Pass Latency: P50 latency is ~8–35ms instead of waiting 300–800ms for mini models to generate JSON tokens.
- ✓ Confidence Threshold Routing: Route to frontier models only when decision model confidence is < 0.85, eliminating edge-case errors.
Architectural Guide & Technical FAQ
How to implement production-grade decision model routers without sacrificing agent intelligence.
Q: Why not just use GPT-4o-mini or Claude 3.5 Haiku as a router?
Mini LLMs are still autoregressive: they cost $0.15–$0.80/M input and $0.60–$4.00/M output, and require 300–600ms to generate schema tokens like {"route": "support"}. Dedicated decision models (Microsoft-Decision-1, Liquid d1) execute in 8–35ms and charge zero for outputs, saving 70–85% compared to mini LLMs.
Q: What is the optimal confidence threshold for fallback?
In benchmark evaluations, a threshold of 0.85 (85% probability) is the sweet spot. When the top decision class probability is ≥ 0.85, route directly to the designated tool or handler. If confidence is < 0.85 (ambiguous or multi-intent requests), pass the prompt to Claude 3.5 Sonnet or GPT-4o for deep reasoning.
Q: Can I self-host Microsoft-Decision-1 on a single GPU card?
Yes. Microsoft-Decision-1 is built on Qwen3.5-9B architecture. In FP8 or 4-bit AWQ, it fits comfortably inside an RTX 4090 (24GB VRAM) or A10G (24GB), achieving sub-20ms inference with vLLM / SGLang. Cloudflare Clef and Liquid d1 also offer edge and on-device runtime options.
Q: How do I integrate this into LiteLLM Proxy?
LiteLLM supports native Decision endpoints. Use our 'LiteLLM YAML' export button above to generate a router configuration with fallback routing policies based on model error rates and confidence tags.