mirror of
https://github.com/obra/superpowers.git
synced 2026-08-17 12:26:46 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0e798e1dc2 |
@@ -101,7 +101,7 @@ Skills are not prose — they are code that shapes agent behavior. If you modify
|
||||
|
||||
## Eval harness
|
||||
|
||||
Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Quorum (the harness CLI, one part of that eval lab) drives real coding-agent CLIs — Claude Code, Codex, Gemini, and others — through a Gauntlet QA agent and grades them against scenario acceptance criteria plus deterministic post-checks. Plugin-infrastructure tests still live at `tests/`.
|
||||
Skill-behavior evals live in [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/), cloned into `evals/` — see `evals/README.md` for setup. Drill (the harness) drives real tmux sessions of Claude Code / Codex / Gemini CLI and judges skill compliance with an LLM verifier. Plugin-infrastructure tests still live at `tests/`.
|
||||
|
||||
## Understand the Project Before Contributing
|
||||
|
||||
|
||||
+9
-10
@@ -14,23 +14,22 @@ Live in `tests/`. Currently:
|
||||
- `tests/codex-plugin-sync/` — bash sync verification.
|
||||
- `tests/kimi/` — bash/Python checks for Kimi plugin manifest wiring.
|
||||
- `tests/claude-code/test-helpers.sh`, `analyze-token-usage.py` — utilities used by remaining bash tests.
|
||||
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no quorum counterpart; tests description-recall, not behavior).
|
||||
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (quorum covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
|
||||
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (quorum covers the PRESSURE phase; bash also covers RED/GREEN baselines).
|
||||
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by quorum.
|
||||
- `tests/claude-code/test-subagent-driven-development.sh` — agent-can-describe-SDD test (no drill counterpart; tests description-recall, not behavior).
|
||||
- `tests/claude-code/test-subagent-driven-development-integration.sh` — extended SDD integration with token analysis (drill covers the YAGNI subset; bash adds commit-count, Claude Code task-tracking, and token telemetry assertions).
|
||||
- `tests/claude-code/test-worktree-native-preference.sh` — RED-GREEN-REFACTOR validation for worktree skill (drill covers the PRESSURE phase; bash also covers RED/GREEN baselines).
|
||||
- `tests/explicit-skill-requests/` — Haiku-specific, multi-turn, and skill-name-prompted tests not covered by drill.
|
||||
|
||||
Run plugin tests via the relevant directory's `run-*.sh` or `npm test`.
|
||||
|
||||
## Skill behavior evals
|
||||
|
||||
Live in `evals/` (the [superpowers-evals](https://github.com/prime-radiant-inc/superpowers-evals/) eval lab, since renamed from Drill). Quorum is the harness CLI — one part of the system: it drives real coding-agent CLIs through a Gauntlet QA agent and grades them against each scenario's acceptance criteria plus deterministic post-checks. Scenarios live at `evals/scenarios/<name>/`. See `evals/README.md` for setup, the container runtime, and the safety model. Quick start (local break-glass run):
|
||||
Live in `evals/`. Drill is the harness; scenarios live at `evals/scenarios/*.yaml`. See `evals/README.md` for setup. Quick start:
|
||||
|
||||
```bash
|
||||
cd evals
|
||||
bun install
|
||||
export SUPERPOWERS_ROOT=/path/to/superpowers
|
||||
bun run quorum run scenarios/triggering-test-driven-development --coding-agent claude
|
||||
bun run quorum show <run-dir>
|
||||
uv sync --extra dev
|
||||
export ANTHROPIC_API_KEY=sk-...
|
||||
uv run drill run triggering-test-driven-development -b claude
|
||||
```
|
||||
|
||||
Quorum scenarios are slow (3-30+ minutes each) and run real LLM sessions in permissive modes — read `evals/README.md`'s Live Eval Risk section first. Only the static gates (`bun run check`, `bun run quorum check`) are safe for public CI; the natural follow-up remains a tiered model (static gates on PR, live sweep nightly + on-demand).
|
||||
Drill scenarios are slow (3-30+ minutes each) and run real LLM sessions. They are not part of CI today; the natural follow-up is a tiered model (fast subset on PR, full sweep nightly + on-demand).
|
||||
|
||||
@@ -135,7 +135,7 @@ a ledger file, not only in todos.
|
||||
|
||||
- Each plan owns a workspace: at skill start, run this skill's
|
||||
`scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored
|
||||
directory (`<repo-root>/.superpowers/sdd/<plan-basename>/`), home to
|
||||
directory (under `<repo-root>/.superpowers/sdd/`), home to
|
||||
every artifact for THIS plan: ledger, briefs, reports, review packages.
|
||||
Another plan's directory is never yours to read or write.
|
||||
- Check for this plan's ledger at `<workspace>/progress.md`. If its first
|
||||
|
||||
@@ -8,6 +8,16 @@
|
||||
# artifacts. A stale ledger misread as current progress makes controllers
|
||||
# skip whole task sequences — plan-scoping removes that failure structurally.
|
||||
#
|
||||
# Basename slugs collide when two plans share a filename (docs/alpha/plan.md
|
||||
# vs docs/beta/plan.md), so each workspace records its owning plan's path in
|
||||
# a plan-path marker (repo-relative in-repo, absolute outside). A workspace
|
||||
# owned by a different plan is skipped and the slug disambiguated with the
|
||||
# plan's parent-directory name, then a counter. A workspace with no marker
|
||||
# predates the marker scheme and is adopted for the current plan so in-flight
|
||||
# workspaces keep resolving — which means the first collision on such a
|
||||
# legacy workspace adopts instead of detecting; acceptable, marker-less
|
||||
# workspaces age out as plans finish.
|
||||
#
|
||||
# The workspace lives in the working tree (not under .git/) because Claude Code
|
||||
# treats .git/ as a protected path and denies agent writes there — which blocks
|
||||
# an implementer subagent from writing its report file. A self-ignoring
|
||||
@@ -34,7 +44,39 @@ slug=$(basename "$plan" .md)
|
||||
|
||||
root=$(git rev-parse --show-toplevel)
|
||||
base="$root/.superpowers/sdd"
|
||||
|
||||
# Normalize the plan path (physical directory, so relative/absolute/../
|
||||
# spellings of one plan compare equal) and express it as the marker value:
|
||||
# repo-relative when the plan lives under the repo root, absolute otherwise.
|
||||
plan_dir=$(CDPATH= cd -- "$(dirname "$plan")" && pwd -P)
|
||||
plan_abs="$plan_dir/$(basename "$plan")"
|
||||
case "$plan_abs" in
|
||||
"$root"/*) plan_id=${plan_abs#"$root"/} ;;
|
||||
*) plan_id=$plan_abs ;;
|
||||
esac
|
||||
|
||||
# True when the workspace at $1 is (or becomes) this plan's: an existing
|
||||
# marker must name this plan; a missing marker means a new workspace or a
|
||||
# pre-marker legacy one, and either way the plan claims it by writing one.
|
||||
owns() {
|
||||
if [ -e "$1/plan-path" ]; then
|
||||
[ "$(cat "$1/plan-path")" = "$plan_id" ]
|
||||
else
|
||||
mkdir -p "$1"
|
||||
printf '%s\n' "$plan_id" > "$1/plan-path"
|
||||
fi
|
||||
}
|
||||
|
||||
dir="$base/$slug"
|
||||
mkdir -p "$dir"
|
||||
if ! owns "$dir"; then
|
||||
parent=$(basename "$plan_dir")
|
||||
dir="$base/$slug-$parent"
|
||||
if ! owns "$dir"; then
|
||||
n=2
|
||||
while ! owns "$base/$slug-$parent-$n"; do n=$((n + 1)); done
|
||||
dir="$base/$slug-$parent-$n"
|
||||
fi
|
||||
fi
|
||||
|
||||
printf '*\n' > "$base/.gitignore"
|
||||
cd "$dir" && pwd
|
||||
CDPATH= cd -- "$dir" && pwd
|
||||
|
||||
@@ -189,6 +189,128 @@ PLAN
|
||||
echo " status: $wt_status"
|
||||
fi
|
||||
|
||||
# --- Ownership markers: two plans with the same basename (#2045) ---
|
||||
mkdir -p "$repo/docs/alpha" "$repo/docs/beta"
|
||||
cat > "$repo/docs/alpha/plan.md" <<'PLAN'
|
||||
# Alpha Plan
|
||||
|
||||
## Task 1: Alpha work
|
||||
|
||||
Alpha-only requirement text.
|
||||
PLAN
|
||||
cat > "$repo/docs/beta/plan.md" <<'PLAN'
|
||||
# Beta Plan
|
||||
|
||||
## Task 1: Beta work
|
||||
|
||||
Beta-only requirement text.
|
||||
PLAN
|
||||
|
||||
local dir_alpha dir_beta
|
||||
dir_alpha="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
|
||||
dir_beta="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/beta/plan.md)"
|
||||
if [[ "$dir_alpha" != "$dir_beta" ]]; then
|
||||
pass "same-basename plans resolve to distinct workspaces"
|
||||
else
|
||||
fail "same-basename plans resolve to distinct workspaces"
|
||||
echo " alpha: $dir_alpha"
|
||||
echo " beta: $dir_beta"
|
||||
fi
|
||||
|
||||
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/alpha/plan.md 1 >/dev/null )
|
||||
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/beta/plan.md 1 >/dev/null )
|
||||
if grep -q "Alpha-only requirement text." "$dir_alpha/task-1-brief.md" 2>/dev/null \
|
||||
&& grep -q "Beta-only requirement text." "$dir_beta/task-1-brief.md" 2>/dev/null; then
|
||||
pass "same-basename plans keep both task briefs intact"
|
||||
else
|
||||
fail "same-basename plans keep both task briefs intact"
|
||||
echo " alpha brief: $(cat "$dir_alpha/task-1-brief.md" 2>/dev/null)"
|
||||
echo " beta brief: $(cat "$dir_beta/task-1-brief.md" 2>/dev/null)"
|
||||
fi
|
||||
|
||||
# --- Legacy adoption: pre-existing workspace without a marker ---
|
||||
printf '# Foo\n\n## Task 1: Foo\n\nFoo.\n' > "$repo/foo.md"
|
||||
mkdir -p "$repo/.superpowers/sdd/foo"
|
||||
printf 'ledger\n' > "$repo/.superpowers/sdd/foo/progress.md"
|
||||
local dir_foo
|
||||
dir_foo="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" foo.md)"
|
||||
if [[ "$dir_foo" == "$repo/.superpowers/sdd/foo" \
|
||||
&& -f "$dir_foo/progress.md" \
|
||||
&& "$(cat "$dir_foo/plan-path" 2>/dev/null)" == "foo.md" ]]; then
|
||||
pass "legacy markerless workspace is adopted in place and marked"
|
||||
else
|
||||
fail "legacy markerless workspace is adopted in place and marked"
|
||||
echo " dir: $dir_foo"
|
||||
echo " marker: $(cat "$dir_foo/plan-path" 2>/dev/null)"
|
||||
fi
|
||||
|
||||
# --- Ownership conflict: marker names a different plan ---
|
||||
printf '# Bar\n\n## Task 1: Bar\n\nBar.\n' > "$repo/bar.md"
|
||||
mkdir -p "$repo/.superpowers/sdd/bar"
|
||||
printf 'somewhere-else/bar.md\n' > "$repo/.superpowers/sdd/bar/plan-path"
|
||||
printf 'other ledger\n' > "$repo/.superpowers/sdd/bar/progress.md"
|
||||
local dir_bar
|
||||
dir_bar="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" bar.md)"
|
||||
if [[ "$dir_bar" == "$repo/.superpowers/sdd/bar-repo" \
|
||||
&& "$(cat "$dir_bar/plan-path" 2>/dev/null)" == "bar.md" ]]; then
|
||||
pass "owned workspace disambiguates with parent-dir suffix"
|
||||
else
|
||||
fail "owned workspace disambiguates with parent-dir suffix"
|
||||
echo " got: $dir_bar"
|
||||
fi
|
||||
if [[ "$(cat "$repo/.superpowers/sdd/bar/plan-path")" == "somewhere-else/bar.md" \
|
||||
&& "$(cat "$repo/.superpowers/sdd/bar/progress.md")" == "other ledger" ]]; then
|
||||
pass "conflicting plan leaves the original workspace untouched"
|
||||
else
|
||||
fail "conflicting plan leaves the original workspace untouched"
|
||||
fi
|
||||
|
||||
# --- Counter fallback: parent-suffixed workspace is owned too ---
|
||||
printf '# Baz\n\n## Task 1: Baz\n\nBaz.\n' > "$repo/baz.md"
|
||||
mkdir -p "$repo/.superpowers/sdd/baz" "$repo/.superpowers/sdd/baz-repo"
|
||||
printf 'one/baz.md\n' > "$repo/.superpowers/sdd/baz/plan-path"
|
||||
printf 'two/baz.md\n' > "$repo/.superpowers/sdd/baz-repo/plan-path"
|
||||
local dir_baz
|
||||
dir_baz="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" baz.md)"
|
||||
if [[ "$dir_baz" == "$repo/.superpowers/sdd/baz-repo-2" \
|
||||
&& "$(cat "$dir_baz/plan-path" 2>/dev/null)" == "baz.md" ]]; then
|
||||
pass "double conflict falls back to a counter suffix"
|
||||
else
|
||||
fail "double conflict falls back to a counter suffix"
|
||||
echo " got: $dir_baz"
|
||||
fi
|
||||
|
||||
# --- Same plan spelled differently resolves to one workspace ---
|
||||
local dir_rel dir_abs dir_dotdot
|
||||
dir_rel="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
|
||||
dir_abs="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$repo/docs/alpha/plan.md")"
|
||||
dir_dotdot="$(cd "$repo/docs/beta" && "$SDD_SCRIPTS/sdd-workspace" ../alpha/plan.md)"
|
||||
if [[ "$dir_rel" == "$dir_abs" && "$dir_rel" == "$dir_dotdot" \
|
||||
&& "$(cat "$dir_rel/plan-path" 2>/dev/null)" == "docs/alpha/plan.md" ]]; then
|
||||
pass "relative, absolute, and ../ spellings share one workspace and marker"
|
||||
else
|
||||
fail "relative, absolute, and ../ spellings share one workspace and marker"
|
||||
echo " rel: $dir_rel"
|
||||
echo " abs: $dir_abs"
|
||||
echo " dotdot: $dir_dotdot"
|
||||
echo " marker: $(cat "$dir_rel/plan-path" 2>/dev/null)"
|
||||
fi
|
||||
|
||||
# --- Out-of-repo plans keep working, marker holds the absolute path ---
|
||||
mkdir -p "$TEST_ROOT/outside"
|
||||
printf '# Remote\n\n## Task 1: Remote\n\nRemote.\n' > "$TEST_ROOT/outside/remote-plan.md"
|
||||
local outside_abs dir_out
|
||||
outside_abs="$(cd "$TEST_ROOT/outside" && pwd -P)/remote-plan.md"
|
||||
dir_out="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$TEST_ROOT/outside/remote-plan.md")"
|
||||
if [[ "$dir_out" == "$repo/.superpowers/sdd/remote-plan" \
|
||||
&& "$(cat "$dir_out/plan-path" 2>/dev/null)" == "$outside_abs" ]]; then
|
||||
pass "out-of-repo plan gets a basename slug and an absolute-path marker"
|
||||
else
|
||||
fail "out-of-repo plan gets a basename slug and an absolute-path marker"
|
||||
echo " dir: $dir_out"
|
||||
echo " marker: $(cat "$dir_out/plan-path" 2>/dev/null)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "$FAILURES" -ne 0 ]]; then
|
||||
echo "FAILED: $FAILURES assertion(s)."
|
||||
|
||||
Reference in New Issue
Block a user