Skip to content

Commit bc8fd20

Browse files
authored
feat: add npx CLI with monorepo, CI/CD, and ephemeral worker architecture (#256)
* feat: integrate npx CLI, CI/CD, and ephemeral worker architecture Bring in changes from shannon-npx: npx-distributable CLI package (cli/), semantic-release CI/CD workflows, ephemeral per-scan worker containers, TOML config support, setup wizard, and workspace management. Preserves all shannon-only changes: security hardening (localhost-bound ports, MCP env allowlist, path traversal guard), updated benchmarks (XBEN 19/31/35/44), README assets, and prompt injection disclaimer. Applies security hardening to cli/infra/compose.yml as well. * refactor: migrate to Turborepo + pnpm + Biome monorepo Restructure into apps/worker, apps/cli, packages/mcp-server with Turborepo task orchestration, pnpm workspaces, Biome linting/formatting, and tsdown CLI bundling. Key changes: - src/ -> apps/worker/src/, cli/ -> apps/cli/, mcp-server/ -> packages/mcp-server/ - prompts/ and configs/ moved into apps/worker/ - npm replaced with pnpm, package-lock.json replaced with pnpm-lock.yaml - Dockerfile updated for pnpm-based builds - CLI logs command rewritten with chokidar for cross-platform reliability - Router health checking added for auto-detected router mode - Centralized path resolution via apps/worker/src/paths.ts * fix: resolve all biome warnings and formatting issues - Remove unnecessary non-null assertions where values are guaranteed - Replace array index access with .at() for safer element retrieval - Use local variables to avoid repeated process.env lookups - Replace any types with unknown in functional utilities - Use nullish coalescing for TOTP hash byte access - Auto-format security patches to match biome config * fix: pin pnpm to 10.12.1 in Dockerfile for catalog support * fix: handle Esc cancellation in Bedrock setup flow Replace p.group() with individual prompts and per-field cancel checks, matching the pattern used by all other provider setup flows. * feat: add optional model customization to Anthropic setup * fix: resolve Docker bind mount permission errors on Linux Use entrypoint-based UID remapping instead of --user flag so the container's pentest user matches the host UID/GID, keeping bind-mounted volumes writable. Git config moved to --system level to survive remapping. * fix: show resumed workflow ID in splash screen URL When resuming a workflow, the Temporal Web UI link pointed to the old (terminated) workflow ID. Now extracts "New Workflow ID" from the resume header in workflow.log, falling back to the original ID for fresh scans. * style: fix biome formatting in docker.ts * fix: align TypeScript config types with JSON Schema - SuccessCondition.type: use schema values (url_contains, element_present, url_equals_exactly, text_contains) instead of stale values (url, cookie, element, redirect) - Authentication.login_flow: mark optional to match schema which does not require it * feat: mark GitHub release as latest during rollback * fix: use native ARM64 runners for Docker multi-platform builds Replace QEMU emulation with parallel native builds using a matrix strategy (ubuntu-latest for amd64, ubuntu-24.04-arm for arm64). Each platform pushes by digest, then a merge job creates the multi-arch manifest list before signing with cosign. * fix: resolve SessionMutex race condition with 3+ concurrent waiters * fix: skip POSIX permission check on Windows writeFileSync mode option is ignored on Windows, so config.toml gets 0o666 and the guard rejects it. * fix: resolve unsubstituted placeholders in report prompt Remove unused {{GITHUB_URL}} placeholder and wire up {{AUTH_CONTEXT}} with structured auth context (login type, username, URL, MFA status). * fix: remove duplicate environment gate from merge-docker job Move DOCKERHUB_USERNAME from vars to secrets so merge-docker can access credentials without its own environment scope. This eliminates the redundant double approval since build-docker already gates on release-publish. * fix: replace POSIX sleep binary with cross-platform async sleep execFileSync('sleep') is unavailable on Windows. Use node:timers/promises setTimeout instead, making ensureInfra async. * fix: use session.json for workflow ID on resume instead of parsing workflow.log On resume, workflow.log already exists with stale headers from the previous run. The CLI poll found '====' immediately and extracted the old workflow ID, producing a wrong Temporal Web UI URL. Read the workflow ID from session.json instead — the worker writes resume attempts there atomically. For fresh runs, poll until originalWorkflowId appears. For resumes, poll until a new resumeAttempts entry is appended. * feat: add custom base URL support for Anthropic-compatible proxies Support ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN to route SDK requests through LiteLLM or any Anthropic-compatible proxy. Adds TUI wizard option, TOML config mapping, credential validation, and preflight endpoint reachability check via SDK query. * fix: remove environment gates and add NPM_TOKEN to publish step * feat: add beta release and rollback workflows with cosign signing * fix: remove redundant checkout and pnpm steps from beta release workflow * docs: normalize README commands to mode-neutral shorthand Add a substitution note after Quick Start sections so all subsequent examples use bare `shannon` instead of mixing `./shannon` and `npx @keygraph/shannon`. Mode-specific commands (build, update, uninstall) get inline annotations. Also fixes a broken command in the Custom Base URL section. * fix: remove redundant `update` command Image is already auto-pulled by `ensureImage()` during `start` when the pinned version tag is missing locally. Manual `update` was unnecessary. * docs: add CLI package README stub * docs: update README setup instructions for dual CLI modes * docs: update announcement banner to npx availability * feat: migrate from MCP tools to CLI based tools (#252) * feat: migrate from MCP tools to CLI tools * fix: restore browser action emoji formatters for CLI output Adapt formatBrowserAction for playwright-cli commands, replacing the old mcp__playwright__browser_* tool name matching removed during migration. * fix: mount credential file to fixed container path for Vertex AI GOOGLE_APPLICATION_CREDENTIALS was forwarded as-is to the container, causing the relative host path to resolve against the repo mount instead of the credentials mount. Now both local and npx modes mount the resolved file to /app/credentials/google-sa-key.json and rewrite the env var to match. * feat: add git awareness and optional description field to config * fix: drop redundant --ipc host flag from worker container * fix: align announcement banner URL with main branch * feat: add target URL reachability preflight check (#254) * Moving asset benchmark graph image to this folder * Move benchmark results to benchmark repo Windows Defender flags exploit code in the pentest reports as false positives, forcing every Windows user to add a Defender exclusion just to clone Shannon. * Updated README * fix: case-insensitive grep for semantic-release version probe * fix: harden supply chain security (#255) * fix: patch smol-toml and tsdown vulnerabilities Update smol-toml 1.6.0→1.6.1 (DoS via recursive comment parsing) and tsdown 0.21.2→0.21.5 (picomatch ReDoS + method injection). * fix: pin all unpinned dependency versions in Dockerfile Pins subfinder v2.13.0, WhatWeb v0.6.3 (switched from git clone to release tarball), schemathesis 4.13.0, addressable 2.8.9, claude-code 2.1.84, and playwright-cli 0.1.1 for reproducible builds. * fix: pin GitHub Actions to commit SHAs for supply chain security * fix: pin GitHub Actions to commit SHAs in beta and rollback workflows
1 parent 0d172f5 commit bc8fd20

4,079 files changed

Lines changed: 13773 additions & 1195079 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/commands/debug.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,21 +22,21 @@ You are debugging an issue. Follow this structured approach to avoid spinning in
2222
**Session audit logs:**
2323
```bash
2424
# Find most recent session
25-
ls -lt audit-logs/ | head -5
25+
ls -lt workspaces/ | head -5
2626

2727
# Check session metrics and errors
28-
cat audit-logs/<session>/session.json | jq '.errors, .agentMetrics'
28+
cat workspaces/<session>/session.json | jq '.errors, .agentMetrics'
2929

3030
# Check agent execution logs
31-
ls -lt audit-logs/<session>/agents/
32-
cat audit-logs/<session>/agents/<latest>.log
31+
ls -lt workspaces/<session>/agents/
32+
cat workspaces/<session>/agents/<latest>.log
3333
```
3434

3535
## Step 3: Trace the Call Path
3636

3737
For Shannon, trace through these layers:
3838

39-
1. **Temporal Client**`src/temporal/client.ts` - Workflow initiation
39+
1. **Worker + Client**`src/temporal/worker.ts` - Combined worker + workflow submission
4040
2. **Workflow**`src/temporal/workflows.ts` - Pipeline orchestration
4141
3. **Activities**`src/temporal/activities.ts` - Thin wrappers: heartbeat, error classification
4242
4. **Container**`src/services/container.ts` - Per-workflow DI
@@ -72,7 +72,7 @@ For Shannon, trace through these layers:
7272
npx playwright install chromium
7373

7474
# Check MCP server startup (look for connection errors)
75-
grep -i "mcp\|playwright" audit-logs/<session>/agents/*.log
75+
grep -i "mcp\|playwright" workspaces/<session>/agents/*.log
7676
```
7777

7878
**Git State Issues:**

.dockerignore

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,14 @@ temp/
4646
ehthumbs.db
4747
Thumbs.db
4848

49+
# CLI package (runs on host, not in container)
50+
# Keep apps/cli/package.json so pnpm workspaces resolve
51+
apps/cli/src/
52+
apps/cli/dist/
53+
apps/cli/infra/
54+
apps/cli/tsconfig.json
55+
apps/cli/tsdown.config.ts
56+
4957
# Docker files (avoid recursive copying)
5058
Dockerfile*
5159
docker-compose*.yml

.env.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ ANTHROPIC_API_KEY=your-api-key-here
7171
# CLAUDE_CODE_USE_VERTEX=1
7272
# CLOUD_ML_REGION=us-east5
7373
# ANTHROPIC_VERTEX_PROJECT_ID=your-gcp-project-id
74-
# GOOGLE_APPLICATION_CREDENTIALS=./credentials/gcp-sa-key.json
74+
# GOOGLE_APPLICATION_CREDENTIALS=./credentials/google-sa-key.json
7575

7676
# =============================================================================
7777
# Available Models

.github/ISSUE_TEMPLATE/bug_report.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ body:
6969
7070
Issues without this information may be difficult to triage.
7171
72-
- Check the audit logs at: `./audit-logs/target_url_shannon-123/workflow.log`
72+
- Check the logs at: `./workspaces/target_url_shannon-123/workflow.log`
7373
Use `grep` or search to identify errors.
7474
Paste the relevant error output below.
7575
- Temporal:

.github/workflows/release-beta.yml

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ jobs:
1919

2020
steps:
2121
- name: Setup Node.js
22-
uses: actions/setup-node@v6
22+
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
2323
with:
2424
node-version: 24
2525
registry-url: https://registry.npmjs.org
@@ -61,20 +61,20 @@ jobs:
6161

6262
steps:
6363
- name: Checkout
64-
uses: actions/checkout@v6
64+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
6565

6666
- name: Set up Docker Buildx
67-
uses: docker/setup-buildx-action@v4
67+
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
6868

6969
- name: Log in to Docker Hub
70-
uses: docker/login-action@v4
70+
uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0
7171
with:
7272
username: ${{ secrets.DOCKERHUB_USERNAME }}
7373
password: ${{ secrets.DOCKERHUB_TOKEN }}
7474

7575
- name: Build and push by digest
7676
id: build
77-
uses: docker/build-push-action@v7
77+
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
7878
with:
7979
context: .
8080
platforms: ${{ matrix.platform }}
@@ -89,7 +89,7 @@ jobs:
8989
touch "/tmp/digests/${digest#sha256:}"
9090
9191
- name: Upload digest
92-
uses: actions/upload-artifact@v6
92+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
9393
with:
9494
name: digests-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
9595
path: /tmp/digests/*
@@ -108,17 +108,17 @@ jobs:
108108

109109
steps:
110110
- name: Download digests
111-
uses: actions/download-artifact@v6
111+
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
112112
with:
113113
path: /tmp/digests
114114
pattern: digests-*
115115
merge-multiple: true
116116

117117
- name: Set up Docker Buildx
118-
uses: docker/setup-buildx-action@v4
118+
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
119119

120120
- name: Log in to Docker Hub
121-
uses: docker/login-action@v4
121+
uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0
122122
with:
123123
username: ${{ secrets.DOCKERHUB_USERNAME }}
124124
password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -138,7 +138,7 @@ jobs:
138138
echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
139139
140140
- name: Install cosign
141-
uses: sigstore/cosign-installer@v4.1.0
141+
uses: sigstore/cosign-installer@ba7bc0a3fef59531c69a25acd34668d6d3fe6f22 # v4.1.0
142142

143143
- name: Sign Docker image
144144
run: cosign sign --yes "keygraph/shannon@${{ steps.inspect.outputs.digest }}"
@@ -161,13 +161,13 @@ jobs:
161161

162162
steps:
163163
- name: Checkout
164-
uses: actions/checkout@v6
164+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
165165

166166
- name: Install pnpm
167-
uses: pnpm/action-setup@v4
167+
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
168168

169169
- name: Configure npm registry
170-
uses: actions/setup-node@v6
170+
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
171171
with:
172172
node-version: 24
173173
registry-url: https://registry.npmjs.org

0 commit comments

Comments
 (0)