When the LLM Is Wrong, Code Wins

The Yes-Man Problem

The RAG index builder asks the LLM: "should these 4 turns extend the current topic, start a new topic, or start a new phase?" The LLM almost always says extend_topic. Every time.

Why? Because consecutive turns in a coding session ARE related. The LLM sees coherence and thinks "same topic." You end up with one monolithic phase spanning 200 turns and one giant topic — useless for retrieval.

The Override

// LLM says "extend_topic" but the topic has 24 turns
if (llmDecision === 'extend_topic' && currentTopic.turns > 20) {
  decision = 'new_topic';  // Override: force a split
}

// LLM says "extend_topic" or "new_topic" but the phase has 85 turns
if (decision !== 'new_phase' && currentPhase.turns > 80) {
  decision = 'new_phase';  // Override: force a phase break
}

How the Thresholds Were Chosen

Empirical tuning, not theory. We ran the system against real 300+ turn sessions and observed the tree output:

Graceful Degradation

The same thresholds serve as the fallback when the LLM fails entirely:

// LLM failed twice — use heuristic fallback
if (phases.length === 0) return 'new_phase';       // No structure yet
if (currentPhase.turns > 80) return 'new_phase';   // Phase too long
if (currentTopic.turns > 20) return 'new_topic';   // Topic too long
return 'extend_topic';                              // Default: continue

LLM failure just means you lose custom titles and summaries — you keep correct tree structure. The structural invariants hold whether the LLM cooperates or not.

The Trust Boundary