mirror of
https://github.com/obra/superpowers.git
synced 2026-09-01 03:19:24 +00:00
Jesse Vincent
affa7fa4e2
Evidence-based diagnosis of superpowers sessions: intake with the human partner, safe transcript reading for Claude Code and Codex (discovery procedure for other harnesses), seven analyst subagents, a report with path:line evidence and a bounded superpowers-involvement line, scrubbed export bundles, approval-gated GitHub issue search/draft, and similar-session search. Includes spec, plan, structure test, and README and docs index lines. Developed RED-GREEN-REFACTOR per writing-skills: 46 scored scenario runs across five SKILL.md versions, all twelve scenarios clean against the final version, micro-tests control 5/5 to skill 0/5 on both baseline-failing prohibitions, and one end-to-end run. Eval records are kept by the maintainer outside the repo. Claude-Session: https://claude.ai/code/session_01DyaGKhTXvHNs2JgPhDktz7
37 lines
2.4 KiB
Markdown
37 lines
2.4 KiB
Markdown
# Testing Superpowers
|
|
|
|
Superpowers has two distinct kinds of tests, each in its own directory:
|
|
|
|
- **`tests/`** — does the plugin's non-LLM code work? Bash + node + python integration tests for brainstorm-server JS, OpenCode plugin loading, codex-plugin sync, and analysis utilities.
|
|
- **`evals/`** — do agents behave correctly on real LLM sessions? Python harness driving real tmux sessions of Claude Code / Codex / Gemini CLI, with an LLM actor and verifier judging skill compliance.
|
|
|
|
## Plugin tests
|
|
|
|
Live in `tests/`. Currently:
|
|
|
|
- `tests/brainstorm-server/` — node test suite for the brainstorm server JS code.
|
|
- `tests/opencode/` — bash tests for OpenCode plugin loading, bootstrap caching, and tool registration.
|
|
- `tests/codex-plugin-sync/` — bash sync verification.
|
|
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
|
|
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
|
|
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
|
|
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
|
|
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
|
|
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
|
|
- `tests/diagnosing-superpowers/test-skill-structure.sh` — structural checks for the diagnosing-superpowers skill (frontmatter, referenced files, leak scan, word budget); behavior-scenario eval records are kept by the maintainer outside the repo.
|
|
|
|
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
|
|
|
|
## Skill behavior evals
|
|
|
|
Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start:
|
|
|
|
```bash
|
|
cd evals
|
|
uv sync --extra dev
|
|
export ANTHROPIC_API_KEY=sk-...
|
|
uv run drill run triggering-test-driven-development -b claude
|
|
```
|
|
|
|
Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand).
|