Compare commits

..

2 Commits

Author SHA1 Message Date
Jesse Vincent c7b456dd0e tests: update SDD assertions and prompts to current skill behavior
The SDD skill tests still asserted on pre-rename skill text, so correct
model answers failed the suite:

- "Read at beginning" asserted "Step 1|beginning|start|Load Plan";
  the current skill has no numbered steps or Load Plan phase (setup
  covers it: "Read the plan once" during Setup). A live run today
  failed this assertion when the model correctly answered "during
  setup". Pattern now accepts setup/before-dispatch paraphrases while
  still requiring an at-the-start answer.
- "Provides text directly" asserted the removed provide-full-task-text
  behavior; SDD now routes task requirements through brief files
  (scripts/task-brief). The test now asks brief-file-vs-whole-plan and
  asserts the brief-based flow.
- The integration test's prompt and summary told the agent to
  "provide full task text to subagents (don't make them read files)",
  contradicting the skill it verifies; reworded to the task-brief flow.
  Also gave its direct `timeout 1800 claude -p` the same </dev/null
  stdin guard as run_claude.

Live-LLM tests; verified with bash -n on every touched file.

Part of #2130; defects documented in PR #2071 by @ericyen97903-lab.
2026-08-13 00:32:49 +00:00
Jesse Vincent 72ee5bbc5e tests: redirect stdin from /dev/null when spawning claude CLI
run_claude ran `timeout "$timeout" "${cmd[@]}"` with the suite's stdin
inherited by the spawned CLI. When the suite is run from a terminal (or
any open stdin), claude -p can block reading stdin and each test stalls
for its full timeout instead of completing.

Verified deterministically with a stub `claude` that reads stdin (cat):
with an open stdin pipe the old helper blocks until timeout kills it
(exit 124); with the redirect it exits immediately with output.

Part of #2130; defect documented in PR #2071 by @ericyen97903-lab.
2026-08-13 00:32:39 +00:00
6 changed files with 17 additions and 180 deletions
+1 -1
View File
@@ -135,7 +135,7 @@ a ledger file, not only in todos.
- Each plan owns a workspace: at skill start, run this skill's - Each plan owns a workspace: at skill start, run this skill's
`scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored
directory (under `<repo-root>/.superpowers/sdd/`), home to directory (`<repo-root>/.superpowers/sdd/<plan-basename>/`), home to
every artifact for THIS plan: ledger, briefs, reports, review packages. every artifact for THIS plan: ledger, briefs, reports, review packages.
Another plan's directory is never yours to read or write. Another plan's directory is never yours to read or write.
- Check for this plan's ledger at `<workspace>/progress.md`. If its first - Check for this plan's ledger at `<workspace>/progress.md`. If its first
@@ -8,16 +8,6 @@
# artifacts. A stale ledger misread as current progress makes controllers # artifacts. A stale ledger misread as current progress makes controllers
# skip whole task sequences — plan-scoping removes that failure structurally. # skip whole task sequences — plan-scoping removes that failure structurally.
# #
# Basename slugs collide when two plans share a filename (docs/alpha/plan.md
# vs docs/beta/plan.md), so each workspace records its owning plan's path in
# a plan-path marker (repo-relative in-repo, absolute outside). A workspace
# owned by a different plan is skipped and the slug disambiguated with the
# plan's parent-directory name, then a counter. A workspace with no marker
# predates the marker scheme and is adopted for the current plan so in-flight
# workspaces keep resolving — which means the first collision on such a
# legacy workspace adopts instead of detecting; acceptable, marker-less
# workspaces age out as plans finish.
#
# The workspace lives in the working tree (not under .git/) because Claude Code # The workspace lives in the working tree (not under .git/) because Claude Code
# treats .git/ as a protected path and denies agent writes there — which blocks # treats .git/ as a protected path and denies agent writes there — which blocks
# an implementer subagent from writing its report file. A self-ignoring # an implementer subagent from writing its report file. A self-ignoring
@@ -44,39 +34,7 @@ slug=$(basename "$plan" .md)
root=$(git rev-parse --show-toplevel) root=$(git rev-parse --show-toplevel)
base="$root/.superpowers/sdd" base="$root/.superpowers/sdd"
# Normalize the plan path (physical directory, so relative/absolute/../
# spellings of one plan compare equal) and express it as the marker value:
# repo-relative when the plan lives under the repo root, absolute otherwise.
plan_dir=$(CDPATH= cd -- "$(dirname "$plan")" && pwd -P)
plan_abs="$plan_dir/$(basename "$plan")"
case "$plan_abs" in
"$root"/*) plan_id=${plan_abs#"$root"/} ;;
*) plan_id=$plan_abs ;;
esac
# True when the workspace at $1 is (or becomes) this plan's: an existing
# marker must name this plan; a missing marker means a new workspace or a
# pre-marker legacy one, and either way the plan claims it by writing one.
owns() {
if [ -e "$1/plan-path" ]; then
[ "$(cat "$1/plan-path")" = "$plan_id" ]
else
mkdir -p "$1"
printf '%s\n' "$plan_id" > "$1/plan-path"
fi
}
dir="$base/$slug" dir="$base/$slug"
if ! owns "$dir"; then mkdir -p "$dir"
parent=$(basename "$plan_dir")
dir="$base/$slug-$parent"
if ! owns "$dir"; then
n=2
while ! owns "$base/$slug-$parent-$n"; do n=$((n + 1)); done
dir="$base/$slug-$parent-$n"
fi
fi
printf '*\n' > "$base/.gitignore" printf '*\n' > "$base/.gitignore"
CDPATH= cd -- "$dir" && pwd cd "$dir" && pwd
+3 -2
View File
@@ -15,8 +15,9 @@ run_claude() {
cmd+=(--allowed-tools="$allowed_tools") cmd+=(--allowed-tools="$allowed_tools")
fi fi
# Run Claude in headless mode with timeout # Run Claude in headless mode with timeout. Redirect stdin from
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1; then # /dev/null so the CLI can't block waiting for input and hang the suite.
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1 < /dev/null; then
cat "$output_file" cat "$output_file"
rm -f "$output_file" rm -f "$output_file"
return 0 return 0
-122
View File
@@ -189,128 +189,6 @@ PLAN
echo " status: $wt_status" echo " status: $wt_status"
fi fi
# --- Ownership markers: two plans with the same basename (#2045) ---
mkdir -p "$repo/docs/alpha" "$repo/docs/beta"
cat > "$repo/docs/alpha/plan.md" <<'PLAN'
# Alpha Plan
## Task 1: Alpha work
Alpha-only requirement text.
PLAN
cat > "$repo/docs/beta/plan.md" <<'PLAN'
# Beta Plan
## Task 1: Beta work
Beta-only requirement text.
PLAN
local dir_alpha dir_beta
dir_alpha="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
dir_beta="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/beta/plan.md)"
if [[ "$dir_alpha" != "$dir_beta" ]]; then
pass "same-basename plans resolve to distinct workspaces"
else
fail "same-basename plans resolve to distinct workspaces"
echo " alpha: $dir_alpha"
echo " beta: $dir_beta"
fi
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/alpha/plan.md 1 >/dev/null )
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/beta/plan.md 1 >/dev/null )
if grep -q "Alpha-only requirement text." "$dir_alpha/task-1-brief.md" 2>/dev/null \
&& grep -q "Beta-only requirement text." "$dir_beta/task-1-brief.md" 2>/dev/null; then
pass "same-basename plans keep both task briefs intact"
else
fail "same-basename plans keep both task briefs intact"
echo " alpha brief: $(cat "$dir_alpha/task-1-brief.md" 2>/dev/null)"
echo " beta brief: $(cat "$dir_beta/task-1-brief.md" 2>/dev/null)"
fi
# --- Legacy adoption: pre-existing workspace without a marker ---
printf '# Foo\n\n## Task 1: Foo\n\nFoo.\n' > "$repo/foo.md"
mkdir -p "$repo/.superpowers/sdd/foo"
printf 'ledger\n' > "$repo/.superpowers/sdd/foo/progress.md"
local dir_foo
dir_foo="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" foo.md)"
if [[ "$dir_foo" == "$repo/.superpowers/sdd/foo" \
&& -f "$dir_foo/progress.md" \
&& "$(cat "$dir_foo/plan-path" 2>/dev/null)" == "foo.md" ]]; then
pass "legacy markerless workspace is adopted in place and marked"
else
fail "legacy markerless workspace is adopted in place and marked"
echo " dir: $dir_foo"
echo " marker: $(cat "$dir_foo/plan-path" 2>/dev/null)"
fi
# --- Ownership conflict: marker names a different plan ---
printf '# Bar\n\n## Task 1: Bar\n\nBar.\n' > "$repo/bar.md"
mkdir -p "$repo/.superpowers/sdd/bar"
printf 'somewhere-else/bar.md\n' > "$repo/.superpowers/sdd/bar/plan-path"
printf 'other ledger\n' > "$repo/.superpowers/sdd/bar/progress.md"
local dir_bar
dir_bar="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" bar.md)"
if [[ "$dir_bar" == "$repo/.superpowers/sdd/bar-repo" \
&& "$(cat "$dir_bar/plan-path" 2>/dev/null)" == "bar.md" ]]; then
pass "owned workspace disambiguates with parent-dir suffix"
else
fail "owned workspace disambiguates with parent-dir suffix"
echo " got: $dir_bar"
fi
if [[ "$(cat "$repo/.superpowers/sdd/bar/plan-path")" == "somewhere-else/bar.md" \
&& "$(cat "$repo/.superpowers/sdd/bar/progress.md")" == "other ledger" ]]; then
pass "conflicting plan leaves the original workspace untouched"
else
fail "conflicting plan leaves the original workspace untouched"
fi
# --- Counter fallback: parent-suffixed workspace is owned too ---
printf '# Baz\n\n## Task 1: Baz\n\nBaz.\n' > "$repo/baz.md"
mkdir -p "$repo/.superpowers/sdd/baz" "$repo/.superpowers/sdd/baz-repo"
printf 'one/baz.md\n' > "$repo/.superpowers/sdd/baz/plan-path"
printf 'two/baz.md\n' > "$repo/.superpowers/sdd/baz-repo/plan-path"
local dir_baz
dir_baz="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" baz.md)"
if [[ "$dir_baz" == "$repo/.superpowers/sdd/baz-repo-2" \
&& "$(cat "$dir_baz/plan-path" 2>/dev/null)" == "baz.md" ]]; then
pass "double conflict falls back to a counter suffix"
else
fail "double conflict falls back to a counter suffix"
echo " got: $dir_baz"
fi
# --- Same plan spelled differently resolves to one workspace ---
local dir_rel dir_abs dir_dotdot
dir_rel="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
dir_abs="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$repo/docs/alpha/plan.md")"
dir_dotdot="$(cd "$repo/docs/beta" && "$SDD_SCRIPTS/sdd-workspace" ../alpha/plan.md)"
if [[ "$dir_rel" == "$dir_abs" && "$dir_rel" == "$dir_dotdot" \
&& "$(cat "$dir_rel/plan-path" 2>/dev/null)" == "docs/alpha/plan.md" ]]; then
pass "relative, absolute, and ../ spellings share one workspace and marker"
else
fail "relative, absolute, and ../ spellings share one workspace and marker"
echo " rel: $dir_rel"
echo " abs: $dir_abs"
echo " dotdot: $dir_dotdot"
echo " marker: $(cat "$dir_rel/plan-path" 2>/dev/null)"
fi
# --- Out-of-repo plans keep working, marker holds the absolute path ---
mkdir -p "$TEST_ROOT/outside"
printf '# Remote\n\n## Task 1: Remote\n\nRemote.\n' > "$TEST_ROOT/outside/remote-plan.md"
local outside_abs dir_out
outside_abs="$(cd "$TEST_ROOT/outside" && pwd -P)/remote-plan.md"
dir_out="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$TEST_ROOT/outside/remote-plan.md")"
if [[ "$dir_out" == "$repo/.superpowers/sdd/remote-plan" \
&& "$(cat "$dir_out/plan-path" 2>/dev/null)" == "$outside_abs" ]]; then
pass "out-of-repo plan gets a basename slug and an absolute-path marker"
else
fail "out-of-repo plan gets a basename slug and an absolute-path marker"
echo " dir: $dir_out"
echo " marker: $(cat "$dir_out/plan-path" 2>/dev/null)"
fi
echo "" echo ""
if [[ "$FAILURES" -ne 0 ]]; then if [[ "$FAILURES" -ne 0 ]]; then
echo "FAILED: $FAILURES assertion(s)." echo "FAILED: $FAILURES assertion(s)."
@@ -23,7 +23,7 @@ echo "========================================"
echo "" echo ""
echo "This test executes a real plan using the skill and verifies:" echo "This test executes a real plan using the skill and verifies:"
echo " 1. Plan is read once (not per task)" echo " 1. Plan is read once (not per task)"
echo " 2. Full task text provided to subagents" echo " 2. Task requirements routed to subagents via brief files"
echo " 3. Subagents perform self-review" echo " 3. Subagents perform self-review"
echo " 4. Spec compliance review before code quality" echo " 4. Spec compliance review before code quality"
echo " 5. Review loops when issues found" echo " 5. Review loops when issues found"
@@ -136,7 +136,7 @@ I want you to execute the implementation plan at docs/superpowers/plans/implemen
IMPORTANT: Follow the skill exactly. I will be verifying that you: IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning 1. Read the plan once at the beginning
2. Provide full task text to subagents (don't make them read files) 2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan)
3. Ensure subagents do self-review before reporting 3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review 4. Run spec compliance review before code quality review
5. Use review loops when issues are found 5. Use review loops when issues are found
@@ -150,7 +150,7 @@ PROMPT="Execute the implementation plan at docs/superpowers/plans/implementation
IMPORTANT: Follow the skill exactly. I will be verifying that you: IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning 1. Read the plan once at the beginning
2. Provide full task text to subagents (don't make them read files) 2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan)
3. Ensure subagents do self-review before reporting 3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review 4. Run spec compliance review before code quality review
5. Use review loops when issues are found 5. Use review loops when issues are found
@@ -164,7 +164,7 @@ PLUGIN_DIR=$(cd "$SCRIPT_DIR/../.." && pwd)
# other concurrent claude sessions. # other concurrent claude sessions.
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..." echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
echo "================================================================================" echo "================================================================================"
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || { cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions < /dev/null 2>&1 | tee "$OUTPUT_FILE" || {
echo "" echo ""
echo "================================================================================" echo "================================================================================"
echo "EXECUTION FAILED (exit code: $?)" echo "EXECUTION FAILED (exit code: $?)"
@@ -316,7 +316,7 @@ if [ $FAILED -eq 0 ]; then
echo "" echo ""
echo "The subagent-driven-development skill correctly:" echo "The subagent-driven-development skill correctly:"
echo " ✓ Reads plan once at start" echo " ✓ Reads plan once at start"
echo " ✓ Provides full task text to subagents" echo " ✓ Routes task requirements via brief files"
echo " ✓ Enforces self-review" echo " ✓ Enforces self-review"
echo " ✓ Runs spec compliance before code quality" echo " ✓ Runs spec compliance before code quality"
echo " ✓ Spec reviewer verifies independently" echo " ✓ Spec reviewer verifies independently"
@@ -4,7 +4,7 @@
# #
# No drill coverage: this test asks the agent to *describe* SDD (string- # No drill coverage: this test asks the agent to *describe* SDD (string-
# matches its verbal explanation against expected keywords like # matches its verbal explanation against expected keywords like
# "self-review", "skeptical", "worktree", "Step 1", "loop"). Drill scenarios # "self-review", "skeptical", "worktree", "setup", "loop"). Drill scenarios
# test behavior (real subagent dispatch, plan-following, review loops), # test behavior (real subagent dispatch, plan-following, review loops),
# not description-recall. Kept by design. # not description-recall. Kept by design.
set -euo pipefail set -euo pipefail
@@ -83,7 +83,7 @@ else
exit 1 exit 1
fi fi
if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then if assert_contains "$output" "beginning\|start\|setup\|before.*dispatch\|before.*task" "Read at beginning"; then
: # pass : # pass
else else
exit 1 exit 1
@@ -133,16 +133,16 @@ echo ""
echo "Test 7: Task context provision..." echo "Test 7: Task context provision..."
output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure: output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure:
Controller provides: <directly or by file> Controller provides: <brief file or whole plan file>
Implementer must read plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT") Implementer must read whole plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then if assert_contains "$output" "task-brief\|brief file\|brief.*path\|Controller provides:.*brief" "Provides task brief file"; then
: # pass : # pass
else else
exit 1 exit 1
fi fi
if assert_contains "$output" "Implementer must read plan file:.*no" "Doesn't make subagent read file"; then if assert_contains "$output" "Implementer must read whole plan file:.*no" "Doesn't make subagent read whole plan"; then
: # pass : # pass
else else
exit 1 exit 1