Compare commits

..

2 Commits

Author SHA1 Message Date
Jesse Vincent c7b456dd0e tests: update SDD assertions and prompts to current skill behavior
The SDD skill tests still asserted on pre-rename skill text, so correct
model answers failed the suite:

- "Read at beginning" asserted "Step 1|beginning|start|Load Plan";
  the current skill has no numbered steps or Load Plan phase (setup
  covers it: "Read the plan once" during Setup). A live run today
  failed this assertion when the model correctly answered "during
  setup". Pattern now accepts setup/before-dispatch paraphrases while
  still requiring an at-the-start answer.
- "Provides text directly" asserted the removed provide-full-task-text
  behavior; SDD now routes task requirements through brief files
  (scripts/task-brief). The test now asks brief-file-vs-whole-plan and
  asserts the brief-based flow.
- The integration test's prompt and summary told the agent to
  "provide full task text to subagents (don't make them read files)",
  contradicting the skill it verifies; reworded to the task-brief flow.
  Also gave its direct `timeout 1800 claude -p` the same </dev/null
  stdin guard as run_claude.

Live-LLM tests; verified with bash -n on every touched file.

Part of #2130; defects documented in PR #2071 by @ericyen97903-lab.
2026-08-13 00:32:49 +00:00
Jesse Vincent 72ee5bbc5e tests: redirect stdin from /dev/null when spawning claude CLI
run_claude ran `timeout "$timeout" "${cmd[@]}"` with the suite's stdin
inherited by the spawned CLI. When the suite is run from a terminal (or
any open stdin), claude -p can block reading stdin and each test stalls
for its full timeout instead of completing.

Verified deterministically with a stub `claude` that reads stdin (cat):
with an open stdin pipe the old helper blocks until timeout kills it
(exit 124); with the redirect it exits immediately with output.

Part of #2130; defect documented in PR #2071 by @ericyen97903-lab.
2026-08-13 00:32:39 +00:00
6 changed files with 17 additions and 180 deletions
+1 -1
View File
@@ -135,7 +135,7 @@ a ledger file, not only in todos.
- Each plan owns a workspace: at skill start, run this skill's
`scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored
directory (under `<repo-root>/.superpowers/sdd/`), home to
directory (`<repo-root>/.superpowers/sdd/<plan-basename>/`), home to
every artifact for THIS plan: ledger, briefs, reports, review packages.
Another plan's directory is never yours to read or write.
- Check for this plan's ledger at `<workspace>/progress.md`. If its first
@@ -8,16 +8,6 @@
# artifacts. A stale ledger misread as current progress makes controllers
# skip whole task sequences — plan-scoping removes that failure structurally.
#
# Basename slugs collide when two plans share a filename (docs/alpha/plan.md
# vs docs/beta/plan.md), so each workspace records its owning plan's path in
# a plan-path marker (repo-relative in-repo, absolute outside). A workspace
# owned by a different plan is skipped and the slug disambiguated with the
# plan's parent-directory name, then a counter. A workspace with no marker
# predates the marker scheme and is adopted for the current plan so in-flight
# workspaces keep resolving — which means the first collision on such a
# legacy workspace adopts instead of detecting; acceptable, marker-less
# workspaces age out as plans finish.
#
# The workspace lives in the working tree (not under .git/) because Claude Code
# treats .git/ as a protected path and denies agent writes there — which blocks
# an implementer subagent from writing its report file. A self-ignoring
@@ -44,39 +34,7 @@ slug=$(basename "$plan" .md)
root=$(git rev-parse --show-toplevel)
base="$root/.superpowers/sdd"
# Normalize the plan path (physical directory, so relative/absolute/../
# spellings of one plan compare equal) and express it as the marker value:
# repo-relative when the plan lives under the repo root, absolute otherwise.
plan_dir=$(CDPATH= cd -- "$(dirname "$plan")" && pwd -P)
plan_abs="$plan_dir/$(basename "$plan")"
case "$plan_abs" in
"$root"/*) plan_id=${plan_abs#"$root"/} ;;
*) plan_id=$plan_abs ;;
esac
# True when the workspace at $1 is (or becomes) this plan's: an existing
# marker must name this plan; a missing marker means a new workspace or a
# pre-marker legacy one, and either way the plan claims it by writing one.
owns() {
if [ -e "$1/plan-path" ]; then
[ "$(cat "$1/plan-path")" = "$plan_id" ]
else
mkdir -p "$1"
printf '%s\n' "$plan_id" > "$1/plan-path"
fi
}
dir="$base/$slug"
if ! owns "$dir"; then
parent=$(basename "$plan_dir")
dir="$base/$slug-$parent"
if ! owns "$dir"; then
n=2
while ! owns "$base/$slug-$parent-$n"; do n=$((n + 1)); done
dir="$base/$slug-$parent-$n"
fi
fi
mkdir -p "$dir"
printf '*\n' > "$base/.gitignore"
CDPATH= cd -- "$dir" && pwd
cd "$dir" && pwd
+3 -2
View File
@@ -15,8 +15,9 @@ run_claude() {
cmd+=(--allowed-tools="$allowed_tools")
fi
# Run Claude in headless mode with timeout
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1; then
# Run Claude in headless mode with timeout. Redirect stdin from
# /dev/null so the CLI can't block waiting for input and hang the suite.
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1 < /dev/null; then
cat "$output_file"
rm -f "$output_file"
return 0
-122
View File
@@ -189,128 +189,6 @@ PLAN
echo " status: $wt_status"
fi
# --- Ownership markers: two plans with the same basename (#2045) ---
mkdir -p "$repo/docs/alpha" "$repo/docs/beta"
cat > "$repo/docs/alpha/plan.md" <<'PLAN'
# Alpha Plan
## Task 1: Alpha work
Alpha-only requirement text.
PLAN
cat > "$repo/docs/beta/plan.md" <<'PLAN'
# Beta Plan
## Task 1: Beta work
Beta-only requirement text.
PLAN
local dir_alpha dir_beta
dir_alpha="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
dir_beta="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/beta/plan.md)"
if [[ "$dir_alpha" != "$dir_beta" ]]; then
pass "same-basename plans resolve to distinct workspaces"
else
fail "same-basename plans resolve to distinct workspaces"
echo " alpha: $dir_alpha"
echo " beta: $dir_beta"
fi
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/alpha/plan.md 1 >/dev/null )
( cd "$repo" && "$SDD_SCRIPTS/task-brief" docs/beta/plan.md 1 >/dev/null )
if grep -q "Alpha-only requirement text." "$dir_alpha/task-1-brief.md" 2>/dev/null \
&& grep -q "Beta-only requirement text." "$dir_beta/task-1-brief.md" 2>/dev/null; then
pass "same-basename plans keep both task briefs intact"
else
fail "same-basename plans keep both task briefs intact"
echo " alpha brief: $(cat "$dir_alpha/task-1-brief.md" 2>/dev/null)"
echo " beta brief: $(cat "$dir_beta/task-1-brief.md" 2>/dev/null)"
fi
# --- Legacy adoption: pre-existing workspace without a marker ---
printf '# Foo\n\n## Task 1: Foo\n\nFoo.\n' > "$repo/foo.md"
mkdir -p "$repo/.superpowers/sdd/foo"
printf 'ledger\n' > "$repo/.superpowers/sdd/foo/progress.md"
local dir_foo
dir_foo="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" foo.md)"
if [[ "$dir_foo" == "$repo/.superpowers/sdd/foo" \
&& -f "$dir_foo/progress.md" \
&& "$(cat "$dir_foo/plan-path" 2>/dev/null)" == "foo.md" ]]; then
pass "legacy markerless workspace is adopted in place and marked"
else
fail "legacy markerless workspace is adopted in place and marked"
echo " dir: $dir_foo"
echo " marker: $(cat "$dir_foo/plan-path" 2>/dev/null)"
fi
# --- Ownership conflict: marker names a different plan ---
printf '# Bar\n\n## Task 1: Bar\n\nBar.\n' > "$repo/bar.md"
mkdir -p "$repo/.superpowers/sdd/bar"
printf 'somewhere-else/bar.md\n' > "$repo/.superpowers/sdd/bar/plan-path"
printf 'other ledger\n' > "$repo/.superpowers/sdd/bar/progress.md"
local dir_bar
dir_bar="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" bar.md)"
if [[ "$dir_bar" == "$repo/.superpowers/sdd/bar-repo" \
&& "$(cat "$dir_bar/plan-path" 2>/dev/null)" == "bar.md" ]]; then
pass "owned workspace disambiguates with parent-dir suffix"
else
fail "owned workspace disambiguates with parent-dir suffix"
echo " got: $dir_bar"
fi
if [[ "$(cat "$repo/.superpowers/sdd/bar/plan-path")" == "somewhere-else/bar.md" \
&& "$(cat "$repo/.superpowers/sdd/bar/progress.md")" == "other ledger" ]]; then
pass "conflicting plan leaves the original workspace untouched"
else
fail "conflicting plan leaves the original workspace untouched"
fi
# --- Counter fallback: parent-suffixed workspace is owned too ---
printf '# Baz\n\n## Task 1: Baz\n\nBaz.\n' > "$repo/baz.md"
mkdir -p "$repo/.superpowers/sdd/baz" "$repo/.superpowers/sdd/baz-repo"
printf 'one/baz.md\n' > "$repo/.superpowers/sdd/baz/plan-path"
printf 'two/baz.md\n' > "$repo/.superpowers/sdd/baz-repo/plan-path"
local dir_baz
dir_baz="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" baz.md)"
if [[ "$dir_baz" == "$repo/.superpowers/sdd/baz-repo-2" \
&& "$(cat "$dir_baz/plan-path" 2>/dev/null)" == "baz.md" ]]; then
pass "double conflict falls back to a counter suffix"
else
fail "double conflict falls back to a counter suffix"
echo " got: $dir_baz"
fi
# --- Same plan spelled differently resolves to one workspace ---
local dir_rel dir_abs dir_dotdot
dir_rel="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" docs/alpha/plan.md)"
dir_abs="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$repo/docs/alpha/plan.md")"
dir_dotdot="$(cd "$repo/docs/beta" && "$SDD_SCRIPTS/sdd-workspace" ../alpha/plan.md)"
if [[ "$dir_rel" == "$dir_abs" && "$dir_rel" == "$dir_dotdot" \
&& "$(cat "$dir_rel/plan-path" 2>/dev/null)" == "docs/alpha/plan.md" ]]; then
pass "relative, absolute, and ../ spellings share one workspace and marker"
else
fail "relative, absolute, and ../ spellings share one workspace and marker"
echo " rel: $dir_rel"
echo " abs: $dir_abs"
echo " dotdot: $dir_dotdot"
echo " marker: $(cat "$dir_rel/plan-path" 2>/dev/null)"
fi
# --- Out-of-repo plans keep working, marker holds the absolute path ---
mkdir -p "$TEST_ROOT/outside"
printf '# Remote\n\n## Task 1: Remote\n\nRemote.\n' > "$TEST_ROOT/outside/remote-plan.md"
local outside_abs dir_out
outside_abs="$(cd "$TEST_ROOT/outside" && pwd -P)/remote-plan.md"
dir_out="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace" "$TEST_ROOT/outside/remote-plan.md")"
if [[ "$dir_out" == "$repo/.superpowers/sdd/remote-plan" \
&& "$(cat "$dir_out/plan-path" 2>/dev/null)" == "$outside_abs" ]]; then
pass "out-of-repo plan gets a basename slug and an absolute-path marker"
else
fail "out-of-repo plan gets a basename slug and an absolute-path marker"
echo " dir: $dir_out"
echo " marker: $(cat "$dir_out/plan-path" 2>/dev/null)"
fi
echo ""
if [[ "$FAILURES" -ne 0 ]]; then
echo "FAILED: $FAILURES assertion(s)."
@@ -23,7 +23,7 @@ echo "========================================"
echo ""
echo "This test executes a real plan using the skill and verifies:"
echo " 1. Plan is read once (not per task)"
echo " 2. Full task text provided to subagents"
echo " 2. Task requirements routed to subagents via brief files"
echo " 3. Subagents perform self-review"
echo " 4. Spec compliance review before code quality"
echo " 5. Review loops when issues found"
@@ -136,7 +136,7 @@ I want you to execute the implementation plan at docs/superpowers/plans/implemen
IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning
2. Provide full task text to subagents (don't make them read files)
2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan)
3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review
5. Use review loops when issues are found
@@ -150,7 +150,7 @@ PROMPT="Execute the implementation plan at docs/superpowers/plans/implementation
IMPORTANT: Follow the skill exactly. I will be verifying that you:
1. Read the plan once at the beginning
2. Provide full task text to subagents (don't make them read files)
2. Route each task's requirements to subagents via a task brief file (don't make them read the whole plan)
3. Ensure subagents do self-review before reporting
4. Run spec compliance review before code quality review
5. Use review loops when issues are found
@@ -164,7 +164,7 @@ PLUGIN_DIR=$(cd "$SCRIPT_DIR/../.." && pwd)
# other concurrent claude sessions.
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
echo "================================================================================"
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions < /dev/null 2>&1 | tee "$OUTPUT_FILE" || {
echo ""
echo "================================================================================"
echo "EXECUTION FAILED (exit code: $?)"
@@ -316,7 +316,7 @@ if [ $FAILED -eq 0 ]; then
echo ""
echo "The subagent-driven-development skill correctly:"
echo " ✓ Reads plan once at start"
echo " ✓ Provides full task text to subagents"
echo " ✓ Routes task requirements via brief files"
echo " ✓ Enforces self-review"
echo " ✓ Runs spec compliance before code quality"
echo " ✓ Spec reviewer verifies independently"
@@ -4,7 +4,7 @@
#
# No drill coverage: this test asks the agent to *describe* SDD (string-
# matches its verbal explanation against expected keywords like
# "self-review", "skeptical", "worktree", "Step 1", "loop"). Drill scenarios
# "self-review", "skeptical", "worktree", "setup", "loop"). Drill scenarios
# test behavior (real subagent dispatch, plan-following, review loops),
# not description-recall. Kept by design.
set -euo pipefail
@@ -83,7 +83,7 @@ else
exit 1
fi
if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then
if assert_contains "$output" "beginning\|start\|setup\|before.*dispatch\|before.*task" "Read at beginning"; then
: # pass
else
exit 1
@@ -133,16 +133,16 @@ echo ""
echo "Test 7: Task context provision..."
output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure:
Controller provides: <directly or by file>
Implementer must read plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
Controller provides: <brief file or whole plan file>
Implementer must read whole plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then
if assert_contains "$output" "task-brief\|brief file\|brief.*path\|Controller provides:.*brief" "Provides task brief file"; then
: # pass
else
exit 1
fi
if assert_contains "$output" "Implementer must read plan file:.*no" "Doesn't make subagent read file"; then
if assert_contains "$output" "Implementer must read whole plan file:.*no" "Doesn't make subagent read whole plan"; then
: # pass
else
exit 1