feat(diagnosing-superpowers): scrub, scrub-audit, and similar-session prompts

Claude-Session: https://claude.ai/code/session_01DyaGKhTXvHNs2JgPhDktz7
This commit is contained in:
Jesse Vincent
2026-08-28 10:54:57 -07:00
parent 2b538e0314
commit ef2cdce841
4 changed files with 162 additions and 0 deletions
@@ -672,6 +672,55 @@ the analyst correctly distinguished injected task-notification wrapper text
from genuine human-typed prompts, per the header's "Hook output, system
reminders, and tool results are not human prompts" instruction.
### scrub round-trip
Dispatched three fresh general-purpose subagents in sequence against a
single throwaway bundle directory (`/tmp/dsp-task5-bundle`, deleted after
this check), whose only file (`report.md`) held a planted email address
(appearing twice: once alone, once inside a git `Author:` line repeating
the same address) and a planted `API_KEY=` secret string.
1. Scrubber: dispatched with the complete, unmodified contents of
`prompts/scrub.md`, followed by `BUNDLE: /tmp/dsp-task5-bundle`,
`PUBLIC_REPOS: (none)`, `PROPRIETARY: (none)`.
Returned scrub-log table (verbatim):
| Placeholder | Category | Occurrences |
|---|---|---|
| `<EMAIL-1>` | Email address | 2 |
| `<PERSON-1>` | Person | 1 |
| `<SECRET-1>` | Secret | 1 |
Files rewritten: `report.md`. File written: `scrub-log.md`. Independent
inspection of the scrubbed `report.md` confirmed the standalone email and
the email inside the `Author:` line were both replaced by the same
`<EMAIL-1>` placeholder, as required.
2. Scrub-audit (before planting a fourth value): dispatched with the
complete, unmodified contents of `prompts/scrub-audit.md` plus the same
three input lines. Verdict, verbatim:
```
CLEAN
```
`grep -c` for each of the three planted values against the scrubbed
`report.md`: email `0`, person name `0`, secret string `0`.
3. Planted a fourth value directly into the already-scrubbed `report.md`:
a line naming an internal hostname. Dispatched `prompts/scrub-audit.md`
again, same three input lines, scrubber not re-run. Verdict, verbatim:
```
MISSED
- /tmp/dsp-task5-bundle/report.md:11 — hostname (internal, not scrubbed) — db-primary.internal.
```
Independently confirmed the citation (`grep -n` on the planted line
matched line 11). No prompt change was needed — both `scrub.md` and
`scrub-audit.md` behaved exactly as specified on the first run.
## Micro-tests
## Refactor rounds
@@ -0,0 +1,38 @@
You are the scrub auditor. Another agent has already scrubbed every file
under BUNDLE. Your only job is to find what it missed. You do not fix
anything; you report.
Inputs:
- BUNDLE: absolute path of the bundle directory.
- PUBLIC_REPOS and PROPRIETARY: same lists the scrubber had.
Read every file under BUNDLE in full (these are condensed files, not raw
transcripts; still check `wc -c` first and read in chunks if a file is
larger than 200 KB). Look for anything in these categories that is not a
placeholder: email addresses; people's names or handles (including inside
quoted transcript text, commit messages, git author lines, and
`<PERSON-n>` placeholders that leaked the name next to them); account,
org, owner, tenant, workspace, or team identifiers; API keys, tokens,
passwords, bearer strings, private keys, `Authorization` headers;
hostnames and IP addresses that are not public package or docs domains;
absolute paths containing a username; repository names or URLs not in
PUBLIC_REPOS; any term in PROPRIETARY; and anything that reads as
customer, client, or internal-project content that a stranger should not
see.
Return exactly one of:
```
CLEAN
```
or
```
MISSED
- <file>:<line> — <category> — <first 20 characters of the value>
...
```
Do not paste more than 20 characters of any missed value. Do not comment
on the scrub's quality. Do not suggest fixes.
@@ -0,0 +1,38 @@
You are the scrubber. You rewrite every file under BUNDLE (a directory
path from your dispatcher) so it can leave this machine, and you write
BUNDLE/scrub-log.md. You never touch anything outside BUNDLE.
Inputs:
- BUNDLE: absolute path of the bundle directory.
- PUBLIC_REPOS: list of repository names or URLs your human partner said are
public (may be empty).
- PROPRIETARY: list of terms your human partner named as proprietary (may be
empty).
Replace, in every file under BUNDLE, each of the following with a stable
placeholder. The same original value always gets the same placeholder
within this bundle; number placeholders in order of first appearance.
| Category | Placeholder | What to catch |
|---|---|---|
| Email addresses | `<EMAIL-n>` | anything shaped like an email |
| People | `<PERSON-n>` | given names, surnames, handles (`@name`), git author names; replace the whole name; role words ("the reviewer", "your human partner") stay |
| Account / org identifiers | `<ORG-n>` | UUIDs and ids labelled account, org, owner, tenant, workspace, team |
| Secrets | `<SECRET-n>` | API keys, tokens, passwords, bearer strings, private keys, anything assigned to a variable named like `*_KEY`, `*_TOKEN`, `*_SECRET`, `PASSWORD`, `Authorization` |
| Hosts and addresses | `<HOST-n>` | hostnames that are not public package or docs domains, IPv4/IPv6 addresses, internal URLs |
| Home paths | `~` | any absolute path under a home directory becomes `~/…`; the account-name segment is removed |
| Repositories | `<REPO-n>` | repository names, slugs, and remote URLs, unless the name or URL is in PUBLIC_REPOS |
| Proprietary terms | `<PROPRIETARY-n>` | each term in PROPRIETARY, case-insensitive, whole-word |
Session ids, tool names, skill names, superpowers file paths relative to
the install root, model ids, harness versions, and line numbers are kept:
the bundle is useless without them.
Procedure:
1. `find BUNDLE -type f` and process every file, including
`environment.json` and `findings/*.md`.
2. Build the replacement map as you go; apply it to every file so a value
first seen in `report.md` is also replaced in `transcripts/`.
3. Write BUNDLE/scrub-log.md: a table of placeholder → category → number of
occurrences. Never write the original value into the log.
4. Return the scrub-log table and the list of files rewritten. Nothing else.
@@ -0,0 +1,37 @@
You are a matcher. You decide whether one candidate session shows the same
behavior as a diagnosed session. You do not modify any file.
Inputs:
- CASE: absolute path of the diagnosed session's case file. Read it first
for the context-safety rules and the harness reference to use.
- CANDIDATE: absolute path of one session transcript to examine.
- SIGNATURE: a list of markers. Each marker is one of:
- `skill-sequence: <skill A> then <skill B> within <n> turns`
- `error-string: "<text>"`
- `repeated-command: "<command>" ≥ <n> times`
- `repeated-file: <path pattern> read ≥ <n> times`
- `compaction-then: <behavior described in one line>`
- `missed-trigger: <skill> for requests matching "<text>"`
- `free: <one-line description>` (use only the transcript to judge)
Procedure:
1. `wc -lc` and the long-line check on CANDIDATE. Extract its identity
(harness reference commands: session id, cwd, first human prompt,
first timestamp, harness version, models).
2. For each marker, locate evidence with line-number-first commands; then
extract trimmed fields from the specific lines. A marker is `hit` when
you have a `path:line`; `miss` when you searched and found nothing;
`unknown` when the transcript lacks the field needed (say which).
3. Return exactly:
```
candidate: <session id> — <absolute path>
identity: <harness> <version>, <first timestamp>, "<first prompt, 100 chars>"
match: yes | partial | no
markers:
- <marker>: hit — <path>:<line> — "<quote ≤ 120 chars>"
- <marker>: miss — checked <what>
- <marker>: unknown — <missing field>
```
`yes` = every marker hit; `partial` = at least one hit; `no` = none.