From ee01bd8cf24e78948e0d0482472266eacd4b64f5 Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Fri, 28 Aug 2026 15:49:10 -0700 Subject: [PATCH] docs(diagnosing-superpowers): end-to-end run record and test listing Claude-Session: https://claude.ai/code/session_01DyaGKhTXvHNs2JgPhDktz7 --- docs/testing.md | 1 + skills/diagnosing-superpowers/CREATION-LOG.md | 176 ++++++++++++++++++ 2 files changed, 177 insertions(+) diff --git a/docs/testing.md b/docs/testing.md index 414d69790..3f560a01d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -18,6 +18,7 @@ Live in `tests/`. Currently: - `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions). - `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines). - `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill. +- `tests/diagnosing-superpowers/test-skill-structure.sh` — structural checks for the diagnosing-superpowers skill (frontmatter, referenced files, leak scan, word budget); behavior scenarios live in the skill's `CREATION-LOG.md`. Run plugin tests via the relevant directory's `run-*.sh` or `npm test`. diff --git a/skills/diagnosing-superpowers/CREATION-LOG.md b/skills/diagnosing-superpowers/CREATION-LOG.md index 9be908c52..243257727 100644 --- a/skills/diagnosing-superpowers/CREATION-LOG.md +++ b/skills/diagnosing-superpowers/CREATION-LOG.md @@ -5794,3 +5794,179 @@ that every REQUIRED section is filled and every finding is cited — not that tw runs agree. The skill reliably produces a well-formed, evidenced report. It does not reliably produce the same report twice. That is worth knowing before anyone treats a single run's verdict as settled. + +## End-to-end run + +One full run of the finished skill, on fixture CC-compact (session +`373e29d1-2223-4e81-95e8-976c35c80040`, 9,170 lines / 14.3 MB, 278 subagent +transcripts / 212 MB beside it), driven by the agent that wrote this section +rather than by an eval scenario. Same redaction convention as the rest of +this file: home paths are written `~`, and where an account name would appear +in a path it is dropped or replaced with `USER`. + +The skill was followed by reading `SKILL.md` from the working tree and doing +what it says, step by step. The `Skill` tool could not be used: it resolves +to the installed 6.3.0 plugin, which does not contain this skill. Intake was +pre-answered — the running agent played the human partner and supplied the +problem statement "the session repeated work after a compaction", redaction +level *evidence*, and "search GitHub, do not file". Everything after intake +was done for real: real transcripts, real subagents, real `gh` searches. + +### The checklist + +| Check | Result | +|---|---| +| Workspace at `~/.superpowers/diagnosing-superpowers//`, path printed | PASS, with a deviation. Files went in a `run-task7/` subdirectory of that workspace because earlier eval runs had left `case.md`, `report*.md` and scratch files at its root. The path was printed to the partner at the end of the locate step, as the skill requires. | +| `report.md` has every REQUIRED section filled | PASS. Nine `## n.` headings in template order, eight `### 6.n` subsections, no unfilled `` text. 1,005 lines / 116 KB. | +| §3 lists the superpowers install root, version, and a sha1 table with at least one row | PASS. Install root and version 6.3.0 from the plugin registry; git sha recorded as "not a checkout" because the registry's `gitCommitSha` is null and the install root has no `.git`. One sha1 row: `skills/using-superpowers/SKILL.md`, injected three times as `SessionStart` hook context. That is the only superpowers file that reached the session. | +| §4 lists the main transcript and every subagent transcript with absolute paths | PASS. 279 rows — 1 main plus all 278 subagents — each with an absolute path, line count and byte count, generated from an inventory rather than typed. | +| §6.7 has per-turn token totals | PASS. All 76 per-turn rows inline in §6.7 (input, output, cache read, cache creation, main total, API calls, subagents dispatched, subagent tokens), plus whole-session totals. | +| §6.3 or §6.2 cites the compaction line | PASS. §6.3 cites lines 2555 and 5252; §6.2 cites 2555. §2 cites both. | +| Bundle directory matches `templates/bundle-README.md` | PASS. 14 files: `README.md`, `report.md`, `case.md`, `environment.json`, `timeline.md`, `findings/*.md` (7), `transcripts/.md`, `scrub-log.md`. 884 KB. | +| `scrub-audit` returned CLEAN | PASS on the second pass, not the first. See below. | +| A recursive grep of `bundle/` for absolute home paths returns nothing | PASS, 0 matches. Independent sweeps for the account name, the proprietary terms and their ≥4-character prefixes, email shapes, IPv4 and key/token shapes also came back empty. | +| No fixture file changed | PASS. `find -newer ` returned nothing, checked three times during the run and once at the end. | + +Report path: +`~/.superpowers/diagnosing-superpowers/373e29d1-2223-4e81-95e8-976c35c80040/run-task7/report.md` +(deleted after this section was written; the workspace holds unscrubbed local +data). + +### What the run found + +§2's verdict, in one line: the compaction boundaries did not cause the +session to redo its work at scale — no shell command and no subagent dispatch +recurs across either boundary — but three narrower repetitions are real, and +the session's two largest repetition events (a ~70-minute API 529 outage and +a human-instructed model-budget handoff) have nothing to do with compaction. +Confidence: medium on the overall verdict, high on the individual findings. +§7 returned **possible**, on the evidence that the superpowers bootstrap was +injected three times including at both compaction boundaries, that no +superpowers skill was invoked in the main agent in 9,170 lines, and that the +only superpowers-namespaced invocation anywhere came from a subagent the main +agent had told to use it by name. + +Worth recording because it is the kind of thing the skill exists to catch: +the repeated-work analyst and the plan-adherence analyst disagreed. The first +said nothing recurs across either boundary; the second found a completed RCA +re-commissioned after boundary 1, caught by the duplicate subagent itself +("I think this work may already be done") rather than by its parent. Both are +right. The repeated-work census keys on exact command text and exact dispatch +description, and the two dispatches carried different descriptions. §8 records +that as a method limit rather than smoothing it over, and §2 names the +disagreement instead of picking a winner. + +### Analysts + +All seven dispatched in one parallel batch, all seven returned findings; none +failed to dispatch. Counts of findings returned: skill-timeline 23, +plan-adherence 14, repeated-work 9, stumbles 25, quality-evidence 16, +request-conflicts 13, cost-and-time 17. No returned finding lacked a +`path:line`, so the "discard any finding without a citation" rule never +fired. Every analyst's `Checked:` line named its commands and ranges, and +every one obeyed the read-only and no-whole-line rules. + +Three findings were spot-checked against the fixture by hand — the +`skill_listing` attachment appearing only at line 11, the 75 +`isolation: worktree` dispatches, and the issue #4 re-commission at lines +4819/4844/2213. All three held. + +### Scrub and audit + +Not CLEAN on the first pass. Audit pass 1 returned one miss: + +``` +MISSED +- transcripts/.md:1461 — proprietary term — prime-radi +``` + +The cause is a deviation this run introduced, not a defect in the skill. The +condensed transcript was rendered with assistant text clipped at 450 +characters and tool inputs at 250 (1600/1200 on cited lines) to keep the file +at 620 KB rather than several megabytes. `templates/bundle-README.md` says +the redaction level governs tool-result bodies; it says nothing about +clipping assistant text, so this was an addition, documented in the bundle's +README and in the rendered file's own header. One clip landed mid-token and +left a bare prefix of a proprietary term, which whole-value matching does not +catch. Scrub round 2 swept every ≥4-character prefix of every redactable +value against the text preceding all 1,026 truncation markers, found exactly +that one fragment, and mapped it to the placeholder its full value already +used. Audit pass 2 returned `CLEAN`. + +The loop worked as specified, and it caught something a human reviewer would +plausibly have missed. Two other catches were beyond a plain sweep: a token +carried in a URL query string, and an IPv4 address — neither of which the +dispatching agent had thought to look for. + +### GitHub step + +Search only; nothing was created and nothing was commented on. `gh search +issues` returned empty for every query, apparently a search-index quirk; +`gh issue list --state all --search` against the same repository worked and +was used instead. Worth knowing for the reference files: the skill's step 5 +says "search open and closed issues … (`gh` if installed)", and the obvious +`gh search issues --state all` is rejected outright — `--state` accepts only +`open` or `closed`. Closest matches found, by number and title only: + +- #1220 (closed) — Uninstalled v5.0.7: SessionStart bootstrap re-injection token cost accumulated across long sessions +- #1453 (closed) — SessionStart hook fires on compact, causes LM Studio Qwen3 jinja template crash +- #2051 (open) — using-superpowers' "check any skill before every action" doesn't hold once a workflow step is underway +- #1465 (closed) — superpowers plugin not loaded mid-session — skills missing from manifest, SessionStart hook not fired +- #2203 (open) — Bounded tasks fill the main session context +- #2083 (open) — Pi extension bootstrap only injected on the first turn: agent_end fires per run, not per session + +The suggestion made to the partner was to attach the bundle to #2051, the +closest to what §7 actually evidences. No issue was drafted, filed, or +commented on. + +### Deviations from SKILL.md, and why + +1. **The skill was followed by hand, not invoked.** The `Skill` tool resolves + to the installed 6.3.0 plugin, which does not ship this skill. +2. **Intake was pre-answered.** Step 1 says to ask one question at a time and + stop if the partner is away. Here the running agent played the partner and + supplied the answers, which is a test fixture, not the workflow. Everything + downstream ran normally. +3. **Workspace subdirectory.** `run-task7/` inside the session workspace, + because earlier eval runs' files were already at its root and a reader + could not otherwise tell the runs apart. The skill says + `~/.superpowers/diagnosing-superpowers//` flat. +4. **Analysts were told not to read all 278 subagent transcripts.** 212 MB + does not fit anywhere. They were given an inventory (`subagents.tsv`) and + a dispatch map (`dispatches.tsv`) for counts and identification and told to + open at most ~10 individual transcripts each; the cost analyst streamed all + 278 for token sums without reading bodies. 27 were examined in depth, 7 + more opened incidentally, 251 unopened. §8 records this. +5. **Condensed-transcript clipping**, described above. It caused the audit + miss. +6. **Only the main transcript was rendered into `transcripts/`.** The 278 + subagent transcripts are enumerated in §4 but not rendered; the bundle + README says so. +7. **No archive.** The skill archives only after the partner has seen the + scrub log and file list and approved. Both were shown; no archive was + requested, so none was made. +8. **§6.7's per-turn table was inlined into the report**, not left only in + `findings/cost-and-time.md`, because the report template asks for per-turn + totals in §6.7 itself. + +### Friction worth knowing about + +- **The 278-subagent case is not something the skill's text anticipates.** + Step 2 says "Enumerate subagent transcripts" and the analyst prompts assume + a subagent set you can look at. At 278 files and 212 MB, enumeration has to + become an inventory file, and every analyst has to be told the limit + explicitly or it will try to read them. The run worked, but only because the + dispatching agent added that instruction to all seven dispatches. +- **§4 at this scale is a 279-row table.** It has to be generated, not typed. + Nothing in the skill says so. +- **Three different human-turn counts appeared** — 82, 76 and 74 — depending + on which harness-injected user-shaped lines each analyst filtered. The case + file offered 82, so most findings quote it. §8 reconciles all three + explicitly rather than picking one silently. A line in + `references/claude-code-sessions.md` about ``, + `` and the re-injected compaction summary being + user-shaped but not human-typed would have prevented the divergence. +- **The dispatching agent's `Write` tool refused any filename containing + "report"**, so report sections were drafted as `sec-*.md` fragments and + assembled with a shell script. Harness-specific, but it is what a + controller at depth 1 will hit.