Compare commits

..

1 Commits

Author SHA1 Message Date
Jesse Vincent 8a96c967c9 docs(testing): describe the Quorum eval lab accurately, replacing stale Drill references
The evals harness was renamed Drill -> Quorum and rewritten from
Python/uv to Bun/TypeScript; docs/testing.md and CLAUDE.md still
described the old tool. Beyond the rename, the old text also
misdescribed the system: quorum is the harness CLI, one part of the
eval lab — it drives real coding-agent CLIs through a Gauntlet QA
agent and grades against scenario acceptance criteria plus
deterministic post-checks. The quick start now matches the eval
repo's actual commands (bun install / bun run quorum run
scenarios/<name> --coding-agent claude; scenarios are directories,
not *.yaml) and points at the Live Eval Risk section before anyone
runs a permissive-mode session.

Drift reported in closed PR #2121 (@JFWaskin); that PR's replacement
quick start kept the uv commands, so this rewrite goes from the eval
repo's README instead.
2026-08-13 00:29:43 +00:00
5 changed files with 24 additions and 24 deletions
+1 -1
View File
@@ -101,7 +101,7 @@ Skills are not prose — they are code that shapes agent behavior. If you modify
## Eval harness ## Eval harness
Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`. Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Quorum (the harness CLI, one part of that eval lab) drives real coding-agent CLIs — Claude Code, Codex, Gemini, and others — through a Gauntlet QA agent and grades them against scenario acceptance criteria plus deterministic post-checks. Plugin-infrastructure tests still live at `tests/`.
## Understand the Project Before Contributing ## Understand the Project Before Contributing
+10 -9
View File
@@ -14,22 +14,23 @@ Live in `tests/`. Currently:
- `tests/codex-plugin-sync/` — bash sync verification. - `tests/codex-plugin-sync/` — bash sync verification.
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring. - `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests. - `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior). - `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no quorum counterpart; tests description-recall, not behavior).
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions). - `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (quorum covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines). - `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (quorum covers the PRESSURE phase; bash also covers RED/GREEN baselines).
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill. - `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by quorum.
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`. Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
## Skill behavior evals ## Skill behavior evals
Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start: Live in `evals/` (the [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/) eval lab, since renamed from Drill). Quorum is the harness CLI — one part of the system: it drives real coding-agent CLIs through a Gauntlet QA agent and grades them against each scenario's acceptance criteria plus deterministic post-checks. Scenarios live at `evals/scenarios/<name>/`. See `evals/README.md` for setup, the container runtime, and the safety model. Quick start (local break-glass run):
```bash ```bash
cd evals cd evals
uv sync --extra dev bun install
export ANTHROPIC_API_KEY=sk-... export SUPERPOWERS_ROOT=/path/to/superpowers
uv run drill run triggering-test-driven-development -b claude bun run quorum run scenarios/triggering-test-driven-development --coding-agent claude
bun run quorum show <run-dir>
``` ```
Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand). Quorum scenarios are slow (3-30+ minutes each) and run real LLM sessions in permissive modes — read `evals/README.md`'s Live Eval Risk section first. Only the static gates (`bun run check`, `bun run quorum check`) are safe for public CI; the natural follow-up remains a tiered model (static gates on PR, live sweep nightly + on-demand).
+2 -3
View File
@@ -15,9 +15,8 @@ run_claude() {
cmd+=(--allowed-tools="$allowed_tools") cmd+=(--allowed-tools="$allowed_tools")
fi fi
# Run Claude in headless mode with timeout. Redirect stdin from # Run Claude in headless mode with timeout
# /dev/null so the CLI can't block waiting for input and hang the suite. if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1; then
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1 < /dev/null; then
cat "$output_file" cat "$output_file"
rm -f "$output_file" rm -f "$output_file"
return 0 return 0
@@ -23,7 +23,7 @@ echo "========================================"
echo "" echo ""
echo "This test executes a real plan using the skill and verifies:" echo "This test executes a real plan using the skill and verifies:"
echo " 1. Plan is read once (not per task)" echo " 1. Plan is read once (not per task)"
echo " 2. Task requirements routed to subagents via brief files" echo " 2. Full task text provided to subagents"
echo " 3. Subagents perform self-review" echo " 3. Subagents perform self-review"
echo " 4. Spec compliance review before code quality" echo " 4. Spec compliance review before code quality"
echo " 5. Review loops when issues found" echo " 5. Review loops when issues found"
@@ -136,7 +136,7 @@ I want you to execute the implementation plan at docs/superpowers/plans/implemen
IMPORTANT: Follow the skill exactly. I will be verifying that you: IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning 1. Read the plan once at the beginning
2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan) 2. Provide full task text to subagents (don't make them read files)
3. Ensure subagents do self-review before reporting 3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review 4. Run spec compliance review before code quality review
5. Use review loops when issues are found 5. Use review loops when issues are found
@@ -150,7 +150,7 @@ PROMPT="Execute the implementation plan at docs/superpowers/plans/implementation
IMPORTANT: Follow the skill exactly. I will be verifying that you: IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning 1. Read the plan once at the beginning
2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan) 2. Provide full task text to subagents (don't make them read files)
3. Ensure subagents do self-review before reporting 3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review 4. Run spec compliance review before code quality review
5. Use review loops when issues are found 5. Use review loops when issues are found
@@ -164,7 +164,7 @@ PLUGIN_DIR=$(cd "$SCRIPT_DIR/../.." && pwd)
# other concurrent claude sessions. # other concurrent claude sessions.
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..." echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
echo "================================================================================" echo "================================================================================"
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions < /dev/null 2>&1 | tee "$OUTPUT_FILE" || { cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
echo "" echo ""
echo "================================================================================" echo "================================================================================"
echo "EXECUTION FAILED (exit code: $?)" echo "EXECUTION FAILED (exit code: $?)"
@@ -316,7 +316,7 @@ if [ $FAILED -eq 0 ]; then
echo "" echo ""
echo "The subagent-driven-development skill correctly:" echo "The subagent-driven-development skill correctly:"
echo " ✓ Reads plan once at start" echo " ✓ Reads plan once at start"
echo " ✓ Routes task requirements via brief files" echo " ✓ Provides full task text to subagents"
echo " ✓ Enforces self-review" echo " ✓ Enforces self-review"
echo " ✓ Runs spec compliance before code quality" echo " ✓ Runs spec compliance before code quality"
echo " ✓ Spec reviewer verifies independently" echo " ✓ Spec reviewer verifies independently"
@@ -4,7 +4,7 @@
# #
# No drill coverage: this test asks the agent to *describe* SDD (string- # No drill coverage: this test asks the agent to *describe* SDD (string-
# matches its verbal explanation against expected keywords like # matches its verbal explanation against expected keywords like
# "self-review", "skeptical", "worktree", "setup", "loop"). Drill scenarios # "self-review", "skeptical", "worktree", "Step 1", "loop"). Drill scenarios
# test behavior (real subagent dispatch, plan-following, review loops), # test behavior (real subagent dispatch, plan-following, review loops),
# not description-recall. Kept by design. # not description-recall. Kept by design.
set -euo pipefail set -euo pipefail
@@ -83,7 +83,7 @@ else
exit 1 exit 1
fi fi
if assert_contains "$output" "beginning\|start\|setup\|before.*dispatch\|before.*task" "Read at beginning"; then if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then
: # pass : # pass
else else
exit 1 exit 1
@@ -133,16 +133,16 @@ echo ""
echo "Test 7: Task context provision..." echo "Test 7: Task context provision..."
output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure: output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure:
Controller provides: <brief file or whole plan file> Controller provides: <directly or by file>
Implementer must read whole plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT") Implementer must read plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
if assert_contains "$output" "task-brief\|brief file\|brief.*path\|Controller provides:.*brief" "Provides task brief file"; then if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then
: # pass : # pass
else else
exit 1 exit 1
fi fi
if assert_contains "$output" "Implementer must read whole plan file:.*no" "Doesn't make subagent read whole plan"; then if assert_contains "$output" "Implementer must read plan file:.*no" "Doesn't make subagent read file"; then
: # pass : # pass
else else
exit 1 exit 1