Sean Thimons – Triumph, Trouble, Trainwreck

About This Talk

The project: Serapeum — an R/Shiny research tool with RAG pipelines, citation networks, slide generation, and PDF extraction.

The experiment: Use Claude Code as an autonomous workflow engine via a custom framework called GSD (Get Stuff Done).

The numbers:

256 sessions over 26 days
1,568 messages, ~60/day
91 commits, 587 files touched
+56,491 / -3,594 lines

/gsd-discuss-phase
/gsd-plan-phase
/gsd-execute-phase
/gsd-verify-work
/gsd-audit-milestone
/gsd-complete-milestone

“Don’t prompt Claude to write code. Prompt Claude to run a software team.”

What Is GSD?

A phase-based development framework built entirely from custom Claude Code slash commands.

The Lifecycle

/gsd-discuss-phase

/gsd-plan-phase

/gsd-execute-phase

/gsd-verify-work

/gsd-audit-milestone

/gsd-complete-milestone

What Happens Under the Hood

Each command orchestrates subagents:

Research agents explore the codebase
Pattern mappers find analogous code
Plan checkers validate before execution
Executors work in parallel waves
Verifiers confirm goal achievement

Milestones group phases into versioned releases (v12–v20 in one month).

ACT I: THE TRIUMPH

The Numbers That Worked

Metric	Value
Fully achieved outcomes	73 / 91 (80%)
Multi-file changes	56 sessions
Parallel sessions (multi-clauding)	54 events, 15% of messages
Dissatisfied signals	Only 8

These aren’t rename this variable tasks.

These are multi-file, multi-step, plan-execute-verify pipelines.

Win: Phase Orchestration

Phase 35 example: discussion -> planning -> UI spec -> execution -> 80 passing tests

Multi-wave subagent architecture:

Wave 1: Research + pattern mapping
Wave 2: Planning with checker validation
Wave 3: Parallel execution in isolated tasks
Wave 4: Verification against plan goals

Each wave has gates — the next wave doesn’t start until the previous one passes.

Bug Bash Discipline

plan
  -> fix
    -> Codex cross-audit
      -> Copilot review
        -> merge

The plan artifact does real work — it’s a contract between you and the agent about what “done” means.

The Triumph: Summary

Workflows transform an AI assistant from a pair programmer into a project manager with commit access.

Three things that worked:

Reusable skill templates — encode hard-won knowledge once, reuse forever
Phase gates — verify between stages, prevent compound errors
Parallel orchestration — three things at once, safely

…or so I thought.

ACT II: THE TROUBLE

Friction by the Numbers

Friction Type	Count
Buggy generated code	25
Wrong initial approach	18
Misunderstood request	7
Environment issues	4
User rejected action	3

Error Type	Count
Command failed	222
User rejected tool call	40
File too large	29
File not found	20
Edit failed	7

The workflow doesn’t eliminate mistakes — it changes where they happen.

When Verification Passes and the Answer Is Wrong

The most common bug class: the workflow’s green checkmark and the data’s actual state don’t agree.

Recurring Data Bugs

Column names after janitor::clean_names() — guessed wrong
Proportions (0–1) vs. percentages (0–100) — broken filters
quantile() returns double, sprintf %d expects integer — crash

Wyoming Goes Missing

Wyoming had one hit in the curated dataset.

The dedup step collapsed rows by analyte name and lost the 1:many mapping.

Every workflow step passed:

Pipeline ran clean
Schema validation succeeded
Tests went green
Output committed

WY went from 355 rows to 4. The workflow never noticed.

Structural correctness ≠ semantic correctness.

The workflow ran perfectly. The data was gutted.

I caught it by squinting at a number.

Going Off-Script

When you give an agent autonomy, it uses that autonomy:

Scope creep: Proposed dropping an embedding column without checking downstream usage. The research refiner needed it.

Not listening: Asked about merge strategy. Got a milestone proposal instead.

Premature claims: Claimed 12 issues were closed before finishing. “Did something stop?” became a recurring question.

Guessing: Applied a speculative shinyjs::delay() fix with no evidence. When challenged, admitted it was guessing. “Don’t guess! I’ll start the process over… again.”

Autonomy requires trust. Trust requires boundaries. The more autonomous the workflow, the more damage a wrong autonomous decision causes.

The Trouble: Summary

Workflows solve the process problem but not the judgment problem.

The agent doesn’t say “I’m not sure about this column name” — it just uses the wrong one
The more structured the workflow, the more confidently it executes wrong steps

Better workflows can mask worse judgment.

One Month Is Not a Sigmoid

Before Act III: the trouble is real. The trouble is also one month of data.

Calling a permanent ceiling from one month is the same forecasting error a team at Wharton made when they fit a sigmoid to the METR graph two weeks before the next model release ran the curve right past their projection.

The trouble in this talk is evidence of a correctable problem.

Not a fatal one.

Sigmoid fit: Ge, Bastani & Bastani (2026), arxiv.org/abs/2602.04836. Story via Scott Alexander, “The Sigmoids Won’t Save You,” Astral Codex Ten, May 2026.

ACT III: THE TRAINWRECK

The Setup: Six Parallel Milestones

Mid-March. Six milestones. Six branches. Six Claude sessions. Each running the full GSD pipeline independently.

PR	Milestone	Created	Merged
#156	v12 UX Polish & Onboarding	Mar 13	Mar 20
#161	v13 Search & Discovery	Mar 17	Mar 20
#168	v14 Citation Network Evolution	Mar 19	Mar 20
#162	v15 AI Infrastructure	Mar 18	Mar 20
#221	v16 Content & Output Quality	Mar 21	never merged
#163	v17 PDF Image Pipeline	Mar 19	Mar 21

Each milestone: individually disciplined. Full discuss-plan-execute-verify lifecycle.

All six needed to merge in the same 48-hour window.

The Anatomy of the Disaster

v12 merged into integration — fine.
v13 merged — 8 review issues fixed first, but ok.
v14 merged, then immediately reverted.
- Namespace scope bugs + missing FWCI scaffold discovered post-merge.
- Had to fix and re-merge.
v15 merged — only after a dedicated “resolve conflicts with integration” commit.
v17 had to absorb v12–v15 INTO ITSELF before it could merge.
- The feature branch swallowed four other milestones.
v16 never merged as a PR. PR #221 closed.
- Changes rescued via a separate “uncommitted integration” fix PR the next day.

The Git Graph

*   d834744 merge: v17 + v16 into integration
|\
| *   6beabed v17.0 PDF image pipeline
| |\
| | *   202b90c merge: integration (v12-v15) INTO v17
| | |\
| | |/
| |/|
* | |   0f2ad9a merge: v16 Content & Output Quality
|\ \ \
| |/ /
|/| |
| * | 4322df7 fix: address PR #221 review feedback
* |     4e299ca Merge PR #162 (v15-ai-infrastructure)
| * |   69b9b32 merge: resolve conflicts with integration
|/
*       4c1516b Merge v14 into integration
*       da06085 Revert "Merge PR #168 (v14)"  <-- THE REVERT
*       152ffdb Merge PR #168 (v14-citation-network)
*       3b1fde3 Merge PR #161 (v13-search-discovery)
*       05852b6 Merge v12-ux-polish into integration

135 files. +28,656 lines. -1,261 lines.

This place is not a place of honor. No highly esteemed deed is commemorated here. Nothing valued is here.

The Aftermath

The next week was integration debris:

PR	What Broke	Days After
#233	v16 uncommitted changes salvaged	+1
#237	Document download broken	+2
#241	v18 Bug Bash: 13 issues, 5 fix sessions	+3
#253	Missing community column migration	+4
#262	Citation network bind length mismatch	+4

The v18 Bug Bash wasn’t planned work.

It was cleanup. Five sessions (A through E) to fix what the integration broke.

“The workflows produced code faster than I could integrate it. I was assembling six sets of IKEA furniture that all shared the same screws.”

Why the Workflow Couldn’t Save Me

The GSD framework had no concept of:

Cross-milestone coordination — no shared integration cadence; branches diverged for days
File overlap awareness — rag.R, mod_settings.R, mod_slides.R, db.R all touched by 3+ milestones
Migration collision — numbered migrations from parallel branches guaranteed conflicts
An integration agent — the one role that doesn’t exist in the framework

Each agent was a perfectly disciplined soldier executing a flawless plan. But nobody was the general.

The Windows Tax

And then there’s the unglamorous stuff:

Claude uses /tmp/ paths
Windows R can’t access /tmp/
This happened in 5+ separate sessions
R segfaults on network calls on Windows
File locking blocks git worktree remove

I have this rule in:

My CLAUDE.md
My skill templates
My memory system
My hooks

Claude still uses /tmp/.

Instruction adherence degrades with context length.

By the time Claude is deep in a multi-stage pipeline, the guardrails have scrolled off its priority list.

THE LESSONS

Two Mistakes I’m Trying Not to Make

After the trainwreck, the two tempting conclusions are both wrong.

Mistake 1: “Workflows hit a ceiling”

Reading the trouble as evidence that structured AI development is doomed.

I’ve been doing this for one month. Calling the sigmoid now is the single most common forecasting error in this field.

Mistake 2: “Mythos will fix it”

Waiting for the next frontier model to dissolve the judgment and coordination problems.

Frontier capability keeps climbing. Frontier prices climb with it. For most users, the next model is either out of budget or wildly overkill for the actual task.

The model is not the bottleneck. The harness is. And so am I.

The harness side: context engineering, hooks, verification, coordination.

The human side: patience, judgment, knowing when not to launch the sixth parallel agent.

What Workflows Buy You (And What They Don’t)

Workflows are a floor, not a ceiling.

What They Buy

Repeatability
Parallelism
Institutional knowledge
Scale (56K lines / month)

What They Don’t

Semantic correctness
Cross-workflow coordination
Judgment under ambiguity
Audit at the rate of production

The uncomfortable truth: the maintenance burden of the workflow system is itself a significant time investment. Every mistake adds a new guardrail. Every guardrail adds context length. Context length degrades guardrail adherence.

That cycle isn’t a verdict — it’s a design problem. And it’s the design problem with the most leverage: of the three bottlenecks, the harness is the only one you can grow.

Vibeslop coding is the 3D printer of software. I’m not printing my house with it.

“Of the three bottlenecks” is a deliberate callback to “Two Mistakes I’m Trying Not to Make.” Model is locked (cost, capability). Operator is a hard ceiling (attention doesn’t scale, and we’re producing faster than anyone can read). Harness is what you can actually build today — skill templates, hooks, verification gates, coordination protocol.

Material to riff on for the vibeslop / 3D printer line:

Right tool for some jobs — prototypes, throwaway scripts, the proof-of-concept you’d rewrite anyway. Wrong tool for the load-bearing parts — production systems, data integrity, anything where Wyoming going missing matters. The mistake isn’t using vibeslop; it’s treating it as the default for everything. Same with fast food: fine in a pinch, toxic as a diet. The harness is what makes the distinction operational — it’s how you decide where the slop is acceptable and where the structure has to hold.

What You Actually Need Beyond Workflows

Semantic verification gates — “Is this trend going up or down?” before writing the narrative
Cross-workflow coordination — if two agents touch the same files, they need sequencing, not parallelism
Hooks over instructions — hooks actually block bad behavior; instructions get forgotten at context length
A convergence strategy — regular integration merges, not a big bang at the end
Honest accounting — report net productivity after subtracting the error correction and integration tax

The Maturity Model

Level	Description	Where It Breaks
1. Prompting	Ask the AI to write code	Quality of each prompt
2. Templates	Reusable prompt patterns	Doesn’t adapt to context
3. Skills / Workflows	Multi-step pipelines	Semantic errors, overreach
4. Verification loops	Workflows + runtime validation	Maintenance burden
5. Coordinated fleets	Parallel agents with shared state	Not yet — but not generational

I’m at Level 3–4.

Level 5 requires solving Level 3’s problems first.

Per Scott Alexander’s Lindy argument, AI paradigms turn over on practitioner timescales, not generational ones. Harnesses turn over even faster — this style is two years old. The next level isn’t a decade away.

Which is where the harness investment goes next.

Scott Alexander, “New Paradigms Won’t Save You,” Astral Codex Ten, May 2026.

The Lindy argument is from Alexander’s “New Paradigms Won’t Save You” (linked at bottom of slide).

Sean’s expanded thesis to riff on: As frontier models hit sigmoid limits at the high end and local models commoditize at the low end, the moat isn’t the model itself — it’s the surrounding stuff: RL, harness, IDE integration, more human content to train against. The practitioner-relevant reading of Alexander’s argument is that the action moves to where the leverage is, and right now that’s the harness layer. This is also what the industry is converging on — the moat is the surrounding scaffold, not the weights.

The “harness investment goes next” line points forward to ongoing work — the actionable items on the previous slide (“What You Actually Need Beyond Workflows”) are the concrete shape of that investment.

The Triumph: Workflows made me 5x more productive on a good day.

The Trouble: The agent doesn’t know what it doesn’t know.

The Trainwreck: Six perfect pipelines, one catastrophic merge.

What I’ll Be Changing

What I’m building into GSD before I run six milestones in parallel again:

An integration agent — the role the framework didn’t have. Owns cross-milestone awareness, file overlap, migration collisions
A merge cadence — daily integration during multi-milestone work, not a big-bang at the end
A Windows detection hook — because instructions get forgotten and /tmp/ keeps coming back
A parallel-agent budget — a hard rule that caps how many concurrent sessions I’ll launch
Honest productivity accounting — net lines after correction tax, with the correction tax broken out

The harness is the only thing I can grow. So I’m growing it.

Workflows are not all you need.

Neither is the next model.

The model is locked. The operator is a ceiling. The harness is the only thing you can grow.

So grow it.

With apologies to Scott Alexander, whose “X Won’t Save You” pieces named the rhetorical move I just spent an hour borrowing.

Sean Thimons

github.com/seanthimons