Skip to content

TDR-003: Finalization Step Rewrites Entire File to Strip Agent Instructions

Description

All working templates (Refactor Vision, Epic Vision, Feature Vision, Component Architecture, Solution Architecture Spec, Implementation Plan) embed agent instructions as HTML comments (<!-- AGENT INSTRUCTIONS ... -->). During the finalization step, these must be removed to produce a clean, board-ready document.

The current finalization process requires the agent to rewrite the entire file — reading all content, stripping the comment blocks, and writing a new version. For a 2000-line document, this means:

  1. Reading the full file into context (~8K tokens)
  2. Producing a complete rewrite (~8K tokens output)
  3. Risk of accidentally modifying content during the rewrite
  4. Significant token cost and latency for a purely mechanical operation
  5. Git diff shows the entire file as changed, making review harder

This was observed during EPC_009 finalization where both the vision (997 → 502 lines) and architecture (2052 → 964 lines) required full rewrites.

Impact

  • Token waste: Each finalization costs ~16K tokens (input + output) for what is a mechanical find-and-delete operation
  • Error risk: Agent may inadvertently modify content while rewriting (observed: minor formatting changes, section reordering)
  • Review friction: Git diff shows entire file changed, making it hard to verify only comments were removed
  • Time cost: ~30-60 seconds per finalization for the LLM to process, vs. milliseconds for a script

Proposed Resolution

Create a finalize.sh (or Python script) that mechanically strips agent instructions:

#!/bin/bash
# Strip all HTML comment blocks containing "AGENT INSTRUCTIONS"
# Also strips the top-level agent instruction block (<!-- ... -->)
sed -i '/^<!--$/,/^-->$/d' "$1"
# Or more targeted: only strip blocks containing "AGENT INSTRUCTIONS"
python3 -c "
import re, sys
content = open(sys.argv[1]).read()
# Remove top-level agent instruction block
content = re.sub(r'<!--\n={10,}.*?={10,}\n-->\n', '', content, flags=re.DOTALL)
# Remove per-section agent instruction blocks  
content = re.sub(r'<!-- AGENT INSTRUCTIONS.*?-->\n', '', content, flags=re.DOTALL)
open(sys.argv[1], 'w').write(content)
" "$1"

The agent's finalization step then becomes: 1. Run finalize.sh <file> (strips comments) 2. Update YAML frontmatter (version, status) — small targeted edit 3. Move session log to appendix — small targeted edit

Benefits: - Deterministic, no content modification risk - Milliseconds instead of 30-60 seconds - Git diff shows only the removed comments + metadata changes - Can be run as a pre-commit hook or CI check

Option B: Separate Files (Agent Instructions as Sidecar)

Instead of embedding instructions in the document, store them in a separate file:

vision_EPC_009_v1.md                    # Clean content only
vision_EPC_009_v1.instructions.md       # Agent instructions (never committed)

The agent reads both files but only writes to the content file. No finalization stripping needed.

Downside: Agent must manage two files per document. Instructions and content can drift.

Option C: Structured Markers for Targeted Removal

Use consistent, machine-parseable markers:

<!-- @agent-instruction-start section="1" -->
...instructions...
<!-- @agent-instruction-end -->

These are trivially removable with a regex, and the markers are self-documenting.

Recommendation

Option A (script) + Option C (structured markers).

Update all templates to use structured markers (@agent-instruction-start/end). Create a finalize.py script that strips these markers. The agent calls the script during finalization instead of rewriting the file.

Priority

P2 — Quality of life. Not blocking, but every document finalization pays this tax. Will become more painful as document count grows.

Affected Templates

  • templates/documents/refactor-vision.md
  • templates/documents/epic-vision.md
  • templates/documents/feature-vision.md
  • templates/documents/component-architecture.md
  • templates/documents/solution-architecture-spec.md
  • templates/documents/implementation-plan.md