Featured demo · Engineering teardown

How the retirement pipeline survives LLM non-determinism.

The hard part of an 80-page LLM document isn’t the prose — it’s the failure modes. One weak section corrupts the deliverable, naive retries burn tokens and trip rate limits, partial failures force full restarts, and prompt-controlled formatting drifts between runs. Here is the architecture that makes long-form generation ship-reliable, including the parts that fought back.

← Executive summary Discuss your use case

Why build orchestration at all

The pipeline didn’t start as a pipeline. It started as one real, high-stakes retirement scenario — the kind where RMD timing, Roth conversion ladders, Social Security taxation, and IRMAA surcharges interact to produce six-figure swings, and where simpler tools collapse. The deliverable was a defensible, year-by-year plan; the brief was “not ‘you’ll be fine.’”

Two decisions from that work became the architecture:

Verification over generation. Before writing anything, I treated the underlying financial model as untrusted and validated it against an LLM — which surfaced three material modeling defects in the planning workbook as configured for this scenario, each of which would have materially mis-stated taxes across the planning horizon if left unchecked. The principle that fell out — the LLM is most valuable for the unglamorous verification work, not the headline generation — is why every quantitative claim downstream must trace to a cell, a shown calculation, or a stated assumption.
Repeatability is the asset, not the output. A finished plan is valuable once; a system that reliably produces defensible plans is valuable on every scenario after it. Generalizing that one deep-dive into a reusable, scenario-agnostic pipeline — parameterized by a CONTEXT.md anchor and a finalized model workbook — is the whole reason the orchestration below exists.

A note on what the LLM is not allowed to do: it never invents numbers. The finalized financial model (a Bogleheads Retiree Portfolio Model workbook) is the quantitative source of truth, treated as immutable after a human-in-the-loop Phase 0. The pipeline below extracts from it and wraps it in narrative — it never recalculates or re-interprets it.

The architecture, end to end

A single gated orchestration script drives all seven stages. Each emits a manifest artifact; no stage proceeds until its gate passes. Content is sacred — post-generation passes never rewrite the model’s words, except documented in-table dollar abbreviation.

The orchestration

A Scout Agent meta-orchestrator sits above the stage workers. It never does stage work itself — at each stage boundary it evaluates that stage’s output artifact against a structured rubric and returns one of three decisions: PASS (proceed), RETRY (re-run, max 3 attempts, with the ability to target a specific deficient section rather than the whole batch), or REWRITE (compress or restructure the handoff artifact without re-running the stage). Every decision is logged. The Scout also compresses each handoff so downstream stages receive only what they need — which is how context stays clean across a very long run.

Model routing

Sections are routed by analytical complexity. Sonnet writes the five scaffolding/structured sections — Roth conversion, income sequencing, Medicare/IRMAA, liquidity, and the roadmap. Opus writes the four analytically heavy ones — multi-year tax optimization, annuity cost-benefit, concentrated-stock cost-basis recovery, and longevity/estate synthesis — plus the cross-section synthesis and executive summary. Running Opus on every section was cost-inefficient; split routing keeps quality where it matters and saves tokens where it doesn’t.

Resumability

State lives in phase_status.json — completed stages, current stage, failures, retry counts, and the append-only Scout decision log. To resume an interrupted run you don’t delete it: the orchestration loop reads it on startup, skips completed stages, and re-dispatches from current. A partial-run failure therefore costs one stage, not the whole pipeline, and completed sections are never re-billed.

The deterministic formatting layer

Formatting was pulled out of the prompts entirely. After generation, five sequential deterministic passes operate on the .docx OOXML directly via lxml XML surgery — dollar abbreviation (tables only), header/column compression, font scaling, landscape promotion of wide tables, and a final bold audit — followed by programmatic insertion of 175 native Word cross-reference hyperlinks against placed bookmarks. The whole thing runs as one unpack/repack cycle and is fully repeatable, where prompt-controlled formatting never was.

The synthetic build — and why it exists

A finished retirement plan is a permanently sensitive document, which created a real problem: the pipeline worked, and its output could never be shown. The fix was to make the demo a first-class artifact. The pipeline is exercised against “Robert” — a fully synthetic 66-year-old single Engineering Manager with roughly $5.29M across fictional Vanguard, Schwab, and Jackson National accounts, including a 401(k) consolidation, a fixed deferred annuity, and a legacy 3M stock lot with missing cost basis. He was engineered to carry the same analytical surface as a real scenario — equivalent hard problems, none of the real data — so the full pipeline is demonstrable and the output is freely shareable. Synthetic-by-design is the credential, and the persona spec doubles as the regression corpus for validating pipeline changes against a known-good output.

The parts that fought back

Reconciling 12 stages down to 7

The pipeline started as a 12-stage linear script (v1), grew a Scout wrapper (v2), then consolidated to 7 gated stages in v3 — RPM extraction folded into Stage 1, and table surgery + bold audit + hyperlinking merged into a single Stage 6 unpack/repack cycle. The published architecture is the hardened 7-stage v3.0; the old 12-stage framing is retired.

A 30K-tokens/min rate limit as a design constraint

The current blocker on fully non-interactive batch runs is the Anthropic API rate limit of 30K input tokens per minute. That constraint is what shaped the split-model routing and the Scout’s aggressive handoff compression — keeping per-call input small is a first-class design pressure here, not an afterthought.

Formatting that couldn’t be unit-tested

Prompt-controlled formatting was non-deterministic across runs and impossible to test reliably. That is precisely why formatting moved into deterministic lxml surgery — a pass either produces the exact XML change or it doesn’t, and that’s checkable.

Two ways to read this, one project

Prefer the value-first version — what it produces and why it’s reliable, minus the XML? That’s the executive summary.

← Back to the executive summary Discuss your use case

Talent Board submission — coming soon