mirror of
https://github.com/obra/superpowers.git
synced 2026-08-15 03:16:44 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 99f9f00869 |
@@ -101,7 +101,7 @@ Skills are not prose — they are code that shapes agent behavior. If you modify
|
||||
|
||||
## Eval harness
|
||||
|
||||
Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Quorum (the harness CLI, one part of that eval lab) drives real coding-agent CLIs — Claude Code, Codex, Gemini, and others — through a Gauntlet QA agent and grades them against scenario acceptance criteria plus deterministic post-checks. Plugin-infrastructure tests still live at `tests/`.
|
||||
Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`.
|
||||
|
||||
## Understand the Project Before Contributing
|
||||
|
||||
|
||||
+9
-10
@@ -14,23 +14,22 @@ Live in `tests/`. Currently:
|
||||
- `tests/codex-plugin-sync/` — bash sync verification.
|
||||
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
|
||||
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
|
||||
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no quorum counterpart; tests description-recall, not behavior).
|
||||
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (quorum covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
|
||||
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (quorum covers the PRESSURE phase; bash also covers RED/GREEN baselines).
|
||||
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by quorum.
|
||||
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
|
||||
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
|
||||
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
|
||||
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
|
||||
|
||||
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
|
||||
|
||||
## Skill behavior evals
|
||||
|
||||
Live in `evals/` (the [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/) eval lab, since renamed from Drill). Quorum is the harness CLI — one part of the system: it drives real coding-agent CLIs through a Gauntlet QA agent and grades them against each scenario's acceptance criteria plus deterministic post-checks. Scenarios live at `evals/scenarios/<name>/`. See `evals/README.md` for setup, the container runtime, and the safety model. Quick start (local break-glass run):
|
||||
Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start:
|
||||
|
||||
```bash
|
||||
cd evals
|
||||
bun install
|
||||
export SUPERPOWERS_ROOT=/path/to/superpowers
|
||||
bun run quorum run scenarios/triggering-test-driven-development --coding-agent claude
|
||||
bun run quorum show <run-dir>
|
||||
uv sync --extra dev
|
||||
export ANTHROPIC_API_KEY=sk-...
|
||||
uv run drill run triggering-test-driven-development -b claude
|
||||
```
|
||||
|
||||
Quorum scenarios are slow (3-30+ minutes each) and run real LLM sessions in permissive modes — read `evals/README.md`'s Live Eval Risk section first. Only the static gates (`bun run check`, `bun run quorum check`) are safe for public CI; the natural follow-up remains a tiered model (static gates on PR, live sweep nightly + on-demand).
|
||||
Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand).
|
||||
|
||||
@@ -22,6 +22,11 @@ head=$3
|
||||
git rev-parse --verify --quiet "$base" >/dev/null || { echo "bad BASE: $base" >&2; exit 2; }
|
||||
git rev-parse --verify --quiet "$head" >/dev/null || { echo "bad HEAD: $head" >&2; exit 2; }
|
||||
|
||||
# Range guards (exit 3): a wrong-branch HEAD yields a range that is empty or
|
||||
# not rooted at BASE; either would silently produce a bogus review package.
|
||||
git merge-base --is-ancestor "$base" "$head" || { echo "HEAD is not a descendant of BASE: ${base}..${head}" >&2; exit 3; }
|
||||
[ "$(git rev-list --count "${base}..${head}")" -gt 0 ] || { echo "empty commit range: ${base}..${head}" >&2; exit 3; }
|
||||
|
||||
if [ $# -eq 4 ]; then
|
||||
out=$4
|
||||
else
|
||||
|
||||
@@ -165,6 +165,30 @@ PLAN
|
||||
echo " got: $rp_explicit"
|
||||
fi
|
||||
|
||||
# --- range guards: BASE must be an ancestor of HEAD, range must be non-empty ---
|
||||
local divergent
|
||||
divergent="$(cd "$repo" && git "${git_id[@]}" commit-tree 'HEAD~1^{tree}' -p 'HEAD~1' -m divergent)"
|
||||
rc=0
|
||||
local guard_err
|
||||
guard_err="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md "$divergent" HEAD 2>&1 >/dev/null)" || rc=$?
|
||||
if [[ "$rc" -eq 3 && "$guard_err" == *"not a descendant"* ]]; then
|
||||
pass "review-package rejects a BASE that is not an ancestor of HEAD with exit 3"
|
||||
else
|
||||
fail "review-package rejects a BASE that is not an ancestor of HEAD with exit 3"
|
||||
echo " exit: $rc"
|
||||
echo " stderr: $guard_err"
|
||||
fi
|
||||
|
||||
rc=0
|
||||
guard_err="$(cd "$repo" && "$SDD_SCRIPTS/review-package" plan-a.md HEAD HEAD 2>&1 >/dev/null)" || rc=$?
|
||||
if [[ "$rc" -eq 3 && "$guard_err" == *"empty commit range"* ]]; then
|
||||
pass "review-package rejects an empty BASE..HEAD range with exit 3"
|
||||
else
|
||||
fail "review-package rejects an empty BASE..HEAD range with exit 3"
|
||||
echo " exit: $rc"
|
||||
echo " stderr: $guard_err"
|
||||
fi
|
||||
|
||||
# --- Worktree isolation: a linked worktree resolves its own workspace ---
|
||||
local wt="$TEST_ROOT/wt"
|
||||
( cd "$repo" && git worktree add -q "$wt" -b wt-feature )
|
||||
|
||||
Reference in New Issue
Block a user