DIL.DEV
LOADING0%
DIL.DEV
← ALL NOTES

Repair Slips, Fail Gaps: What the LLM Markdown Fixes Taught Me

Three thrown executions taught me to separate format slips from missing content. Deterministic repair keeps drafts alive; failing on gaps keeps the output honest.

n8nllmtestingautomation

A terminal showing a test suite with passing repair cases and failing gap cases side by side.

Three thrown executions, two commits and a clearer definition of what "format error" actually means. That is what it took to land on a rule that is simple to explain: if the site can reconstruct intent without guessing, repair it deterministically and carry on. If content is genuinely absent, fail.

The Insight That Changed the Approach

Before these fixes, any validation failure in the Compose Draft node threw immediately and killed the execution. That sounds reasonable until you notice what it actually costs:

  • The queued topic is popped from the queue at the start of the run. A thrown execution means that topic is gone.
  • The paid model call has already happened. The draft exists. The throw discards it.

Two executions illustrated this. Execution 16027 died on a 201-character summary. Execution 17746 died on a closing heading written as ## What I'd Do Differently. Both had usable drafts. Both wasted a call. The learning was that not all failures are equal, and treating them the same way is expensive.

What a Slip Actually Looks Like

A 201-character summary is not a content problem. The model produced a complete, coherent summary that happened to be one character over the limit. Throwing on that is the wrong call.

The clamp introduced to handle it works like this:

  • If the summary is within 200 characters, pass it through unchanged.
  • If it exceeds 200 characters, walk back to the last whole sentence boundary.
  • Keep that shorter version, provided it is at least 60 characters.
  • If no sentence boundary exists above 60 characters, fail. That is a genuine gap.

The 60-character floor matters. A one-sentence clamp that produces "Yes." is not a usable summary. The floor ensures the repair only fires when the result is still meaningful.

The Closing Heading Problem (and What It Taught Me About Normalising)

Every note on this site ends with a section whose exact heading is ## What I'd do differently. The model in execution 17746 produced ## What I'd Do Differently. Title Case on a heading that should be sentence case.

That is a slip. The intent is unambiguous. The normaliser introduced in commit 8ffda02 repairs it by lowercasing the words after the first, then checking the result matches the canonical form.

What still fails:

  • No closing section at all. The heading is simply absent. That is missing content, not a formatting slip.
  • A structurally different heading. Something like ## Lessons learned or ## Final thoughts is a different choice, not a near-miss on the canonical one.

The distinction matters because repair is only safe when there is one plausible correct output. A different heading could mean the model forgot the rule or deliberately chose something else. Failing is the right call.

The Model Keeps Finding New Near-Misses

Execution 18420 threw on ## What I'd do Different. Not wrong case this time. The suffix -ly was simply missing.

Commit 155522c extended the normaliser to cover the full family of near-misses the model had produced across three executions:

  • Wrong capitalisation (Title Case or mixed case)
  • Missing apostrophe in "I'd"
  • "I would" written out in full instead of contracted
  • Missing -ly on "different"
  • One trailing colon or full stop

The saved output from 18420 was replayed through the updated node and passed at 923 words.

What still fails after this extension:

  • ## What I'd do differently next time (extra words, structurally different)
  • A level-3 heading (###) instead of level-2 (##)

Both require a judgement call the normaliser cannot make safely. They still throw.

Recovering Without Paying Again

The most practically useful thing from all three failures: the model output was not lost. n8n saves the output at each node. When a downstream validation node throws, you can re-run from that saved state.

Execution 17746 was retried from Compose Draft as execution 18149. Execution 16027 was retried as 18154. Both reached Send for Review using the original model output, no new draft, no new cost.

The pattern is: fix the validator, then replay the saved output. You only pay for a new model call when the draft itself is the problem, not the validator. That distinction is worth building into how you structure any pipeline that calls a paid model.

Testing Both Sides of the Boundary

The test file at n8n/tests/compose.test.cjs covers both sides of the repair boundary. That is what makes the normaliser safe to extend.

Without tests on both sides, a future change could silently widen what counts as repairable. You might start accepting headings that are genuinely wrong rather than just slightly malformed.

The tests pin:

  • Repair cases: a 201-character summary clamps and passes; each heading variant the normaliser handles reaches Send for Review.
  • Gap cases: a missing closing section throws the expected error; a structurally different heading throws; a level-3 heading throws.

If someone extends the normaliser to cover a new variant, they need to add a repair-side test for that variant and confirm the gap-side tests still fail. The structure makes the boundary visible and keeps it honest.

What I'd do differently

The normaliser grew one execution failure at a time. That is fine for shipping fixes quickly, but it means the repair rules ended up documented in commit messages and a README section rather than in the test file itself.

A cleaner approach would be to write the test cases first when a new slip appears, confirm the test fails, then extend the normaliser. The tests at n8n/tests/compose.test.cjs are close to that now, but the first two fixes predated the test coverage being tight enough to drive the change.

I would also make the canonical heading a single constant imported by both the normaliser and the tests. Right now the string ## What I'd do differently appears in more than one place. If the site's style ever changes, that is a quiet way to get the normaliser and the tests out of sync.

NOTES BY EMAIL

Get the next note in your inbox.

One email per note. No spam. Reply “stop” any time and you are off the list.