mirror of
https://github.com/obra/superpowers.git
synced 2026-08-29 18:09:27 +00:00
feat(diagnosing-superpowers): scrub, scrub-audit, and similar-session prompts
Claude-Session: https://claude.ai/code/session_01DyaGKhTXvHNs2JgPhDktz7
This commit is contained in:
@@ -672,6 +672,55 @@ the analyst correctly distinguished injected task-notification wrapper text
|
||||
from genuine human-typed prompts, per the header's "Hook output, system
|
||||
reminders, and tool results are not human prompts" instruction.
|
||||
|
||||
### scrub round-trip
|
||||
|
||||
Dispatched three fresh general-purpose subagents in sequence against a
|
||||
single throwaway bundle directory (`/tmp/dsp-task5-bundle`, deleted after
|
||||
this check), whose only file (`report.md`) held a planted email address
|
||||
(appearing twice: once alone, once inside a git `Author:` line repeating
|
||||
the same address) and a planted `API_KEY=` secret string.
|
||||
|
||||
1. Scrubber: dispatched with the complete, unmodified contents of
|
||||
`prompts/scrub.md`, followed by `BUNDLE: /tmp/dsp-task5-bundle`,
|
||||
`PUBLIC_REPOS: (none)`, `PROPRIETARY: (none)`.
|
||||
|
||||
Returned scrub-log table (verbatim):
|
||||
|
||||
| Placeholder | Category | Occurrences |
|
||||
|---|---|---|
|
||||
| `<EMAIL-1>` | Email address | 2 |
|
||||
| `<PERSON-1>` | Person | 1 |
|
||||
| `<SECRET-1>` | Secret | 1 |
|
||||
|
||||
Files rewritten: `report.md`. File written: `scrub-log.md`. Independent
|
||||
inspection of the scrubbed `report.md` confirmed the standalone email and
|
||||
the email inside the `Author:` line were both replaced by the same
|
||||
`<EMAIL-1>` placeholder, as required.
|
||||
|
||||
2. Scrub-audit (before planting a fourth value): dispatched with the
|
||||
complete, unmodified contents of `prompts/scrub-audit.md` plus the same
|
||||
three input lines. Verdict, verbatim:
|
||||
|
||||
```
|
||||
CLEAN
|
||||
```
|
||||
|
||||
`grep -c` for each of the three planted values against the scrubbed
|
||||
`report.md`: email `0`, person name `0`, secret string `0`.
|
||||
|
||||
3. Planted a fourth value directly into the already-scrubbed `report.md`:
|
||||
a line naming an internal hostname. Dispatched `prompts/scrub-audit.md`
|
||||
again, same three input lines, scrubber not re-run. Verdict, verbatim:
|
||||
|
||||
```
|
||||
MISSED
|
||||
- /tmp/dsp-task5-bundle/report.md:11 — hostname (internal, not scrubbed) — db-primary.internal.
|
||||
```
|
||||
|
||||
Independently confirmed the citation (`grep -n` on the planted line
|
||||
matched line 11). No prompt change was needed — both `scrub.md` and
|
||||
`scrub-audit.md` behaved exactly as specified on the first run.
|
||||
|
||||
## Micro-tests
|
||||
|
||||
## Refactor rounds
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
You are the scrub auditor. Another agent has already scrubbed every file
|
||||
under BUNDLE. Your only job is to find what it missed. You do not fix
|
||||
anything; you report.
|
||||
|
||||
Inputs:
|
||||
- BUNDLE: absolute path of the bundle directory.
|
||||
- PUBLIC_REPOS and PROPRIETARY: same lists the scrubber had.
|
||||
|
||||
Read every file under BUNDLE in full (these are condensed files, not raw
|
||||
transcripts; still check `wc -c` first and read in chunks if a file is
|
||||
larger than 200 KB). Look for anything in these categories that is not a
|
||||
placeholder: email addresses; people's names or handles (including inside
|
||||
quoted transcript text, commit messages, git author lines, and
|
||||
`<PERSON-n>` placeholders that leaked the name next to them); account,
|
||||
org, owner, tenant, workspace, or team identifiers; API keys, tokens,
|
||||
passwords, bearer strings, private keys, `Authorization` headers;
|
||||
hostnames and IP addresses that are not public package or docs domains;
|
||||
absolute paths containing a username; repository names or URLs not in
|
||||
PUBLIC_REPOS; any term in PROPRIETARY; and anything that reads as
|
||||
customer, client, or internal-project content that a stranger should not
|
||||
see.
|
||||
|
||||
Return exactly one of:
|
||||
|
||||
```
|
||||
CLEAN
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
MISSED
|
||||
- <file>:<line> — <category> — <first 20 characters of the value>
|
||||
...
|
||||
```
|
||||
|
||||
Do not paste more than 20 characters of any missed value. Do not comment
|
||||
on the scrub's quality. Do not suggest fixes.
|
||||
@@ -0,0 +1,38 @@
|
||||
You are the scrubber. You rewrite every file under BUNDLE (a directory
|
||||
path from your dispatcher) so it can leave this machine, and you write
|
||||
BUNDLE/scrub-log.md. You never touch anything outside BUNDLE.
|
||||
|
||||
Inputs:
|
||||
- BUNDLE: absolute path of the bundle directory.
|
||||
- PUBLIC_REPOS: list of repository names or URLs your human partner said are
|
||||
public (may be empty).
|
||||
- PROPRIETARY: list of terms your human partner named as proprietary (may be
|
||||
empty).
|
||||
|
||||
Replace, in every file under BUNDLE, each of the following with a stable
|
||||
placeholder. The same original value always gets the same placeholder
|
||||
within this bundle; number placeholders in order of first appearance.
|
||||
|
||||
| Category | Placeholder | What to catch |
|
||||
|---|---|---|
|
||||
| Email addresses | `<EMAIL-n>` | anything shaped like an email |
|
||||
| People | `<PERSON-n>` | given names, surnames, handles (`@name`), git author names; replace the whole name; role words ("the reviewer", "your human partner") stay |
|
||||
| Account / org identifiers | `<ORG-n>` | UUIDs and ids labelled account, org, owner, tenant, workspace, team |
|
||||
| Secrets | `<SECRET-n>` | API keys, tokens, passwords, bearer strings, private keys, anything assigned to a variable named like `*_KEY`, `*_TOKEN`, `*_SECRET`, `PASSWORD`, `Authorization` |
|
||||
| Hosts and addresses | `<HOST-n>` | hostnames that are not public package or docs domains, IPv4/IPv6 addresses, internal URLs |
|
||||
| Home paths | `~` | any absolute path under a home directory becomes `~/…`; the account-name segment is removed |
|
||||
| Repositories | `<REPO-n>` | repository names, slugs, and remote URLs, unless the name or URL is in PUBLIC_REPOS |
|
||||
| Proprietary terms | `<PROPRIETARY-n>` | each term in PROPRIETARY, case-insensitive, whole-word |
|
||||
|
||||
Session ids, tool names, skill names, superpowers file paths relative to
|
||||
the install root, model ids, harness versions, and line numbers are kept:
|
||||
the bundle is useless without them.
|
||||
|
||||
Procedure:
|
||||
1. `find BUNDLE -type f` and process every file, including
|
||||
`environment.json` and `findings/*.md`.
|
||||
2. Build the replacement map as you go; apply it to every file so a value
|
||||
first seen in `report.md` is also replaced in `transcripts/`.
|
||||
3. Write BUNDLE/scrub-log.md: a table of placeholder → category → number of
|
||||
occurrences. Never write the original value into the log.
|
||||
4. Return the scrub-log table and the list of files rewritten. Nothing else.
|
||||
@@ -0,0 +1,37 @@
|
||||
You are a matcher. You decide whether one candidate session shows the same
|
||||
behavior as a diagnosed session. You do not modify any file.
|
||||
|
||||
Inputs:
|
||||
- CASE: absolute path of the diagnosed session's case file. Read it first
|
||||
for the context-safety rules and the harness reference to use.
|
||||
- CANDIDATE: absolute path of one session transcript to examine.
|
||||
- SIGNATURE: a list of markers. Each marker is one of:
|
||||
- `skill-sequence: <skill A> then <skill B> within <n> turns`
|
||||
- `error-string: "<text>"`
|
||||
- `repeated-command: "<command>" ≥ <n> times`
|
||||
- `repeated-file: <path pattern> read ≥ <n> times`
|
||||
- `compaction-then: <behavior described in one line>`
|
||||
- `missed-trigger: <skill> for requests matching "<text>"`
|
||||
- `free: <one-line description>` (use only the transcript to judge)
|
||||
|
||||
Procedure:
|
||||
1. `wc -lc` and the long-line check on CANDIDATE. Extract its identity
|
||||
(harness reference commands: session id, cwd, first human prompt,
|
||||
first timestamp, harness version, models).
|
||||
2. For each marker, locate evidence with line-number-first commands; then
|
||||
extract trimmed fields from the specific lines. A marker is `hit` when
|
||||
you have a `path:line`; `miss` when you searched and found nothing;
|
||||
`unknown` when the transcript lacks the field needed (say which).
|
||||
3. Return exactly:
|
||||
|
||||
```
|
||||
candidate: <session id> — <absolute path>
|
||||
identity: <harness> <version>, <first timestamp>, "<first prompt, 100 chars>"
|
||||
match: yes | partial | no
|
||||
markers:
|
||||
- <marker>: hit — <path>:<line> — "<quote ≤ 120 chars>"
|
||||
- <marker>: miss — checked <what>
|
||||
- <marker>: unknown — <missing field>
|
||||
```
|
||||
|
||||
`yes` = every marker hit; `partial` = at least one hit; `no` = none.
|
||||
Reference in New Issue
Block a user