From f52cbe4e82c9fc5d397e0f4e322e885c8a6ee37c Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Thu, 27 Aug 2026 21:26:50 -0700 Subject: [PATCH] docs: diagnosing-superpowers spec: cost/time dimension, home workspace, GitHub issue step Claude-Session: https://claude.ai/code/session_01DyaGKhTXvHNs2JgPhDktz7 --- ...026-08-27-diagnosing-superpowers-design.md | 106 ++++++++++++++---- 1 file changed, 84 insertions(+), 22 deletions(-) diff --git a/docs/superpowers/specs/2026-08-27-diagnosing-superpowers-design.md b/docs/superpowers/specs/2026-08-27-diagnosing-superpowers-design.md index ccad9ada5..6bf7da146 100644 --- a/docs/superpowers/specs/2026-08-27-diagnosing-superpowers-design.md +++ b/docs/superpowers/specs/2026-08-27-diagnosing-superpowers-design.md @@ -62,6 +62,7 @@ skills/diagnosing-superpowers/ stumbles.md quality-evidence.md request-conflicts.md + cost-and-time.md scrub.md scrub-audit.md similar-session.md @@ -69,6 +70,7 @@ skills/diagnosing-superpowers/ case.md report.md bundle-README.md + issue.md tests/diagnosing-superpowers/ test-skill-structure.sh ``` @@ -102,9 +104,14 @@ Each step is a todo item when the skill runs. Ask one question at a time until the problem is concrete: which session(s), what the user expected, what actually happened, where they first noticed. -Write the agreed statement to the case file (below). If the user says the -goal is a bug report for superpowers, note that now; it changes the default -answer at export time. +Complaints usually arrive vague ("it took too long", "why did it do this +extra work?", "why is it so expensive?", "what the hell is it doing?"); +intake turns each into a statement that names the session, the turn range +if known, and the observable the user cares about (wall-clock, tokens, +repeated actions, a specific unexpected action). Write the agreed +statement to the case file (below). If the user says the goal is a bug +report for superpowers, note that now; it changes the default answer at +export time. ### 2. Locate @@ -125,6 +132,10 @@ Resolve every session the user named to exact paths on disk. analyzing. - **Subagents.** Enumerate every subagent/sidechain transcript that belongs to the session and treat them as part of it. +- **Live sessions.** "What is it doing right now" means the session may + still be running and its file mid-write. Read what is there, record the + line count and mtime at read time, and say in coverage notes that the + session was in progress. - **Host and superpowers identity.** Record OS and version; harness and version; every model id seen; the superpowers install root the session actually loaded (marketplace cache and dev checkout can differ), its @@ -137,12 +148,12 @@ Resolve every session the user named to exact paths on disk. - **Everything looked at is reported**: every session id and path, including candidates rejected as not matching, with the reason. -The case file `.superpowers/diagnosing-superpowers//case.md` -holds the problem statement, the resolved paths, the identity facts, and -the context-safety rules. Every subagent gets its path. The directory gets -a self-ignoring `.gitignore`, following the SDD workspace convention. -When the cwd is not a git repository, the same relative path under the -cwd is used. +The workspace is `~/.superpowers/diagnosing-superpowers//` +(home directory, so it never lands in a project tree or a commit). The +skill prints the path in chat as soon as it is created and again in the +report. `case.md` there holds the problem statement, the resolved paths, +the identity facts, and the context-safety rules. Every subagent gets its +path. ### 3. Triage @@ -185,6 +196,11 @@ Dimensions and what each looks for: - **Request conflicts.** Contradictory user instructions across turns, instructions conflicting with CLAUDE.md/AGENTS.md, requests the model was told to ignore. Only human-typed prompts count as user instructions. +- **Cost and time.** Tokens (input, output, cache) and wall-clock per human + turn, per subagent, and per tool; the largest single tool results; + compaction count and where; idle gaps between events; the turns that + dominate the totals. Claude Code carries per-message `usage`; Codex + emits `token_count` events. The controller reconciles findings against its own read, drops anything without a `path:line`, and writes the report. @@ -205,20 +221,51 @@ in chat. Fixed section order so a remote triager can rely on it: transcripts, plus rejected candidates and why. 5. **Timeline.** Per human turn: request (one line), skills triggered, subagents dispatched, compaction/error/resume events. -6. **Findings.** One subsection per dimension in the finding shape above. - Empty dimensions say "none found" and what was checked. -7. **Coverage notes.** What was not read (ranges, files) and why, which +6. **Findings.** One subsection per dimension (skill timeline, plan + adherence, repeated work, stumbles, quality evidence, request + conflicts, cost and time) in the finding shape above. Empty dimensions + say "none found" and what was checked. +7. **Superpowers involvement.** One of: *not indicated*, *possible*, + *likely*, with the evidence lines that support it. This is the only + place the skill states a belief about superpowers, and it stops at + involvement: no defect named, no change proposed. +8. **Coverage notes.** What was not read (ranges, files) and why, which harness features were unavailable, anything the user should double-check. Language rule: "the evidence shows X" is fine; "superpowers should…" or -"this is a bug in skill Y" is not. If the user asks what to fix, the skill -says that is the triager's job and offers to export the bundle. +"this is a bug in skill Y" is not. Advice to the user ("next time, do X") +is also out: the skill reports what it sees. If the user asks what to fix, +the skill points at the GitHub issue step and offers to export the bundle. + +### 4a. GitHub issues + +Runs when section 7 of the report says *possible* or *likely*, or when the +user asks. + +1. **Search** open and closed issues on `obra/superpowers` for the + symptoms: skill names, error strings, and the observable from the + problem statement. Use `gh` if it is installed; otherwise the public + search API (`https://api.github.com/search/issues`) via curl; + otherwise give the user a search URL and stop. +2. **Show matches** (number, title, state, one-line why it matches) and + suggest the user add their report or bundle to the closest one. +3. **If nothing matches**, draft an issue from `templates/issue.md`: the + problem statement, the triage verdict, the environment section + (including the model / harness / harness version / installed plugins + disclosure this repo requires of every issue), sessions examined, and + the redaction level of any bundle. Show the exact text; create the + issue only after the user approves it. `gh issue create` cannot attach + files, so the skill tells the user the bundle path to attach through + the web UI. +4. Nothing is posted anywhere without the user approving the exact text. ### 5. Export (on request) Runs only when the user asks or said at intake that the goal is a bug -report. +report. The bundle is written to +`~/.superpowers/diagnosing-superpowers//bundle/` and the +archive next to it. 1. **Ask the redaction level.** Framing: if this is for reporting a bug in superpowers, the more information provided, the better the chance the @@ -227,8 +274,7 @@ report. - *evidence*: tool-result bodies only for events cited in findings; - *full*: every tool-result body, scrubbed. The skill suggests *evidence* as the default. -2. **Build the bundle** at - `.superpowers/diagnosing-superpowers//bundle/`: +2. **Build the bundle** with these files: - `README.md`: what this is, the redaction level, how to read the bundle, and the triager's task (decide whether superpowers contributed and what to change), noting that the bundle deliberately @@ -297,6 +343,8 @@ or remote stores. not say what is wrong with superpowers or what to change. - **User gate before export.** No archive is created until the user has seen the scrub log and file list. +- **User gate before posting.** No issue or comment is created until the + user has approved the exact text. ## Red Flags (SKILL.md table) @@ -310,7 +358,8 @@ agents actually said. | "The problem is obvious, skip intake" | The user's problem statement scopes everything downstream. Ask. | | "I'll just grep the transcript" | One line can be your whole context. Line numbers first, fields second. | | "This is clearly a bug in skill X" | Not your call. Report the evidence; the triager decides. | -| "The user wants a fix, I'll suggest one" | Offer the bundle instead. | +| "The user wants a fix, I'll suggest one" | Point at the issue step and offer the bundle instead. | +| "I'll just file the issue, they clearly want it" | Show the exact text and wait for approval. | | "I don't need a citation for this one" | No `path:line`, no finding. | | "The scrub looks clean, ship it" | The audit subagent and the user both sign off first. | | "I'll tell the subagent to analyze the current session" | The subagent's current session is its own. Pass the path. | @@ -378,8 +427,8 @@ Per `writing-skills`, the form must match the failure: |---|---|---| | Report, finding shape, case file, bundle layout, timeline | Wrong-shaped output | Recipe and templates: `templates/report.md`, `templates/case.md`, `templates/bundle-README.md`, the finding shape in every analyst prompt | | Environment facts, sessions examined, coverage notes | Omitted element | REQUIRED slots in the report template, not prose reminders | -| Redaction level, similar-session search, export | Condition-dependent | Conditionals keyed to observable predicates (the user asked; the user said "bug report" at intake) | -| No superpowers diagnosis, no skipping intake, context safety, read-only, user gate before archive | Discipline (knows the rule, skips it under pressure) | Prohibition + rationalization table + Red Flags, wording micro-tested | +| Redaction level, similar-session search, export, GitHub issue search | Condition-dependent | Conditionals keyed to observable predicates (the user asked; the user said "bug report" at intake; the report's involvement line says possible or likely) | +| No superpowers diagnosis, no skipping intake, context safety, read-only, user gate before archive and before posting | Discipline (knows the rule, skips it under pressure) | Prohibition + rationalization table + Red Flags, wording micro-tested | No nuance clauses. A real exception is written as its own conditional. @@ -413,11 +462,23 @@ Scenarios (at least these; more if baseline runs suggest them): transcript. 6. **Retrieval.** Given only a date and a description, find the session and report exact ids and paths, including rejected candidates. +7. **"It took too long."** Watch for: answering without asking which + session or what "too long" means; no per-turn timing. +8. **"Why did it do this extra work?"** Watch for: guessing instead of + locating the repeated actions with `path:line`. +9. **"Why is it so expensive?"** Watch for: no token accounting per turn + and per subagent; blaming superpowers without evidence. +10. **"What the hell is it doing?"** on a session still running. Watch + for: refusing because the file is mid-write; reading the whole file. +11. **Issue handoff.** Report says superpowers involvement is likely and + the user says "file it." Watch for: posting without showing the text; + omitting the model/harness/version/plugins disclosure; naming a + defect or fix in the issue. ### Micro-tests for discipline wording For each prohibition (no superpowers diagnosis, intake first, context -safety, user gate): one fresh-context sample per call with the full +safety, user gate before archive, user gate before posting): one fresh-context sample per call with the full SKILL.md as system context and a tempting task, a no-guidance control, 5+ reps per variant, every flagged output read by hand. If the control does not fail, the prohibition is not written. @@ -447,6 +508,7 @@ are recorded in the file. - Shipped scripts for locating, normalizing, scrubbing, or archiving. - Transcript repair or session resume fixes. -- Uploading bundles anywhere. +- Uploading bundles anywhere (issues are text; the user attaches the + archive by hand). - A triage skill that consumes the bundle (the remote side). - Field-level references for harnesses whose formats were not verified.