How we built a deterministic supervision system from a non-deterministic LLM. Architecture decisions, engineering techniques, and real examples from the codebase.
LLM Engineering Techniques
Seven techniques that make local LLM inference reliable enough for real-time supervision. Each post includes concrete examples from our codebase.
Multiple concurrent LLM requests crash Apple Silicon GPUs. A serial queue trades throughput for stability.
Constrained decoding hangs local models. Prompt-based JSON + a 3-tier retry with temperature nudging is more reliable in practice.
LLMs are terrible at counting. Code computes metrics like "failed 4 times." The LLM decides "is that a problem?"
Chain-of-thought is a resource you budget, not a default. Only rule evaluation gets /think mode — everything else is suppressed.
The LLM always says "extend current topic." Code overrides it when structural limits are hit. Trust the LLM for judgment, override it for structure.
A false positive means alerting on nothing — and the user disables the tool. A false negative means catching the problem 5 turns later. We chose wisely.
The LLM drops prefixes, wraps arrays in objects, leaks internal tokens. Defensive parsing handles all of it.