Skip to content

Commit 9b8b6bd

Browse files
authored
Deterministic test infrastructure with injectable clocks and project reorganization (#119)
## Summary Introduce an injectable `ClockFn` abstraction and in-memory `ChannelSocket` transport to enable fully deterministic, fast, and platform-independent integration tests — replacing real UDP sockets, `thread::sleep()`, and `#[serial]` test ordering throughout the test suite. Also reorganizes scripts and LLM skills into domain-based subdirectories. ### Clock injection & deterministic time control - Add `ClockFn` type alias (`Arc<dyn Fn() -> Instant + Send + Sync>`) and `ProtocolConfig::clock` field for injectable time sources - Add `ChaosSocket::with_clock()` builder method for deterministic chaos/network-condition testing - Replace all `Instant::now()` calls in `Protocol` and `ChaosSocket` with the injected clock (falling back to system time) - **Breaking:** `ProtocolConfig` no longer implements `Copy` (due to `Arc`-based clock field) ### New test infrastructure - `ChannelSocket`: in-memory `NonBlockingSocket` using `mpsc` channels — eliminates all real UDP I/O from tests - `TestClock`: manually-advanceable virtual clock with `advance(duration)`, `as_protocol_clock()`, and `as_chaos_clock()` helpers - Deterministic test utilities: `synchronize_sessions_deterministic()`, `poll_with_advance()`, `run_p2p_frame_advancement_test_deterministic()` ### Test suite migration - Converted all resilience, p2p, p2p_enum, and spectator tests from real sockets + `thread::sleep` + `#[serial]` to `ChannelSocket` + `TestClock` - Removed `serial_test` dependency and all `#[serial]` attributes from migrated tests - Removed `#[cfg_attr(miri, ignore)]` from timing-dependent chaos socket tests (now virtual-time-based) ### Scripts & skills reorganization - Reorganized `scripts/` into `build/`, `ci/`, `docs/`, `verification/` subdirectories - Reorganized `.llm/skills/` into 8 category subdirectories (rust-language, testing, formal-verification, etc.) - Added 8 new workflow skills (code-review, adversarial-review, dev-pipeline, investigation, etc.) - Added `.llm/design-history/` and `.llm/templates/ask-user-question.md` ### Documentation & CI - Updated user guide with custom clock documentation, new config presets, and expanded API reference - Updated all wiki pages with corrected examples, API signatures, and verification status - Updated all CI workflows and pre-commit hooks for new script paths - Expanded devcontainer with additional VS Code extensions and editor settings - Added logo banner SVG ## Test plan - [ ] `cargo nextest run --no-capture` — all tests pass with deterministic infrastructure - [ ] `cargo clippy --all-targets --features tokio,json` — no warnings - [ ] CI workflows resolve scripts at new paths - [ ] Pre-commit hooks work with reorganized script layout - [ ] Verify no `#[serial]` tests remain in migrated test files 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent dd673ce commit 9b8b6bd

158 files changed

Lines changed: 6259 additions & 2549 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.config/nextest.toml

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,6 @@
66
# it's either a production bug or a test bug that must be fixed.
77
# Do NOT add retries to mask flakiness.
88

9-
# Test groups for tests that share UDP ports
10-
# These tests must run serially to avoid port conflicts
11-
[test-groups]
12-
# Tests in sessions::p2p, sessions::p2p_enum, and some spectator tests use ports 7777/8888
13-
port-7777 = { max-threads = 1 }
14-
159
[profile.default]
1610
# Zero retries - tests must pass first time
1711
retries = 0
@@ -33,16 +27,6 @@ status-level = "pass"
3327
failure-output = "immediate-final" # Show failure output immediately AND at end
3428
success-output = "never" # Hide output for passing tests (reduce noise)
3529

36-
# Serialize tests that use port 7777 (and 8888)
37-
[[profile.default.overrides]]
38-
filter = 'test(sessions::p2p::) | test(sessions::p2p_enum::)'
39-
test-group = 'port-7777'
40-
41-
# Spectator tests using 7777/8888 (test_start_session and test_synchronize_with_host)
42-
[[profile.default.overrides]]
43-
filter = 'test(sessions::spectator::test_start_session) | test(sessions::spectator::test_synchronize_with_host)'
44-
test-group = 'port-7777'
45-
4630
# Multi-process network tests need longer timeouts because they:
4731
# 1. Spawn external peer processes
4832
# 2. Simulate hostile network conditions (high packet loss, burst loss)

.devcontainer/README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ This devcontainer provides a complete environment for developing, testing, and f
5656
After the container starts, verify all tools:
5757

5858
```bash
59-
./scripts/check-tools.sh
59+
./scripts/ci/check-tools.sh
6060
```
6161

6262
## Build Performance
@@ -96,19 +96,19 @@ means subsequent rebuilds only reprocess changed layers.
9696
### All Verifiers
9797

9898
```bash
99-
./scripts/verify-all.sh
99+
./scripts/verification/verify-all.sh
100100
```
101101

102102
### TLA+ Only
103103

104104
```bash
105-
./scripts/verify-tla.sh
105+
./scripts/verification/verify-tla.sh
106106
```
107107

108108
### Kani Only
109109

110110
```bash
111-
./scripts/verify-kani.sh
111+
./scripts/verification/verify-kani.sh
112112
# Or directly:
113113
cargo kani
114114
```

.devcontainer/devcontainer.json

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
// README at: https://github.com/devcontainers/templates/tree/main/src/rust
33
{
44
"name": "Fortress Rollback - Full Verification Environment",
5-
65
// ===== BUILD OPTIONS (choose one) =====
76
//
87
// Option A: Build locally from Dockerfile (first build ~3-5 min, rebuilds use cache)
@@ -15,7 +14,6 @@
1514
// Local Dockerfile builds remain the default until the registry path is validated across environments.
1615
// Uncomment the line below and comment out the "build" block above to use it.
1716
// "image": "ghcr.io/wallstop/fortress-rollback/devcontainer:latest",
18-
1917
// Additional features (on top of what's in Dockerfile)
2018
"features": {
2119
"ghcr.io/devcontainers/features/powershell:1": {}
@@ -28,7 +26,7 @@
2826
}
2927
],
3028
// Post-create setup: only workspace-specific tasks (tools are pre-installed in image)
31-
"postCreateCommand": "bash -c '\n set -e\n echo \"=== Setting up Fortress Rollback Development Environment ===\"\n \n # Ensure TLA+ tools are in the workspace\n mkdir -p .tla-tools\n if [ -f /opt/tla/tla2tools.jar ]; then\n cp /opt/tla/tla2tools.jar .tla-tools/\n elif [ ! -f .tla-tools/tla2tools.jar ]; then\n echo \"Downloading TLA+ tools...\"\n curl --proto \"=https\" --tlsv1.2 -fsSL --retry 5 --retry-delay 2 --retry-all-errors --max-time 120 https://github.com/tlaplus/tlaplus/releases/download/v1.8.0/tla2tools.jar -o .tla-tools/tla2tools.jar\n fi\n \n # Sync Vale styles (prose linter)\n if command -v vale >/dev/null 2>&1 && [ -f .vale.ini ]; then\n echo \"Syncing Vale packages...\"\n vale sync || echo \"Vale sync failed, run vale sync manually\"\n fi\n \n # Build project to warm dependency cache\n echo \"Warming dependency cache...\"\n if ! cargo build; then\n echo \"WARNING: Initial cargo build failed. Run cargo build manually to see errors.\"\n fi\n \n echo \"=== Development environment ready! ===\"\n echo \"Run ./scripts/check-tools.sh to verify all tools are installed.\"\n'",
29+
"postCreateCommand": "bash -c '\n set -e\n echo \"=== Setting up Fortress Rollback Development Environment ===\"\n \n # Ensure TLA+ tools are in the workspace\n mkdir -p .tla-tools\n if [ -f /opt/tla/tla2tools.jar ]; then\n cp /opt/tla/tla2tools.jar .tla-tools/\n elif [ ! -f .tla-tools/tla2tools.jar ]; then\n echo \"Downloading TLA+ tools...\"\n curl --proto \"=https\" --tlsv1.2 -fsSL --retry 5 --retry-delay 2 --retry-all-errors --max-time 120 https://github.com/tlaplus/tlaplus/releases/download/v1.8.0/tla2tools.jar -o .tla-tools/tla2tools.jar\n fi\n \n # Sync Vale styles (prose linter)\n if command -v vale >/dev/null 2>&1 && [ -f .vale.ini ]; then\n echo \"Syncing Vale packages...\"\n vale sync || echo \"Vale sync failed, run vale sync manually\"\n fi\n \n # Build project to warm dependency cache\n echo \"Warming dependency cache...\"\n if ! cargo build; then\n echo \"WARNING: Initial cargo build failed. Run cargo build manually to see errors.\"\n fi\n \n echo \"=== Development environment ready! ===\"\n echo \"Run ./scripts/ci/check-tools.sh to verify all tools are installed.\"\n'",
3230
// Environment variables for tools
3331
"containerEnv": {
3432
"TLA_TOOLS_JAR": "${containerWorkspaceFolder}/.tla-tools/tla2tools.jar",
@@ -53,11 +51,91 @@
5351
"ms-python.vscode-pylance",
5452
"ryanluker.vscode-coverage-gutters",
5553
"streetsidesoftware.code-spell-checker",
54+
"PKief.material-icon-theme",
55+
"GitHub.github-vscode-theme",
56+
"redhat.vscode-yaml",
57+
"markdownlint.markdownlint",
58+
"eamodio.gitlens",
59+
"serayuzgur.crates",
60+
"EditorConfig.EditorConfig",
61+
"yzhang.markdown-all-in-one",
62+
"ms-vscode.hexeditor",
63+
"dracula-theme.theme-dracula",
64+
"enkia.tokyo-night",
65+
"catppuccin.catppuccin-vsc",
66+
"zhuangtongfa.material-theme",
67+
"akamud.vscode-theme-onedark",
68+
"sdras.night-owl",
69+
"teabyii.ayu",
70+
"arcticicestudio.nord-visual-studio-code",
71+
"jdinhlife.gruvbox",
72+
"mvllow.rose-pine",
73+
"whizkydee.material-palenight-theme",
74+
"alexandernanberg.horizon-theme-vscode",
75+
"brittanychiang.halcyon-vscode",
76+
"robbowen.synthwave-vscode",
77+
"pmndrs.pmndrs",
78+
"qufiwefefwoyn.kanagawa",
79+
"azemoh.one-monokai",
5680
"GitHub.copilot",
5781
"GitHub.copilot-chat",
5882
"anthropic.claude-code"
5983
],
6084
"settings": {
85+
"workbench.iconTheme": "material-icon-theme",
86+
"workbench.colorTheme": "GitHub Dark Default",
87+
"workbench.preferredDarkColorTheme": "GitHub Dark Default",
88+
"workbench.preferredLightColorTheme": "GitHub Light Default",
89+
"explorer.excludeGitIgnore": false,
90+
"explorer.compactFolders": false,
91+
"explorer.confirmDelete": false,
92+
"files.exclude": {
93+
"**/.git": true,
94+
"**/.claude": true,
95+
"**/.ruff_cache": true,
96+
"**/.vale": true,
97+
"**/.vscode": true,
98+
"**/target": true,
99+
"**/node_modules": true,
100+
"**/__pycache__": true,
101+
"**/specs/tla/states*": true,
102+
"**/mutants.out": true,
103+
"**/.tmp": true,
104+
"**/.idea": true,
105+
"**/site": true,
106+
"**/states": true,
107+
"loom-tests/target": true,
108+
"progress": true,
109+
"fuzz/corpus": true,
110+
"fuzz/artifacts": true,
111+
"fuzz/coverage": true
112+
},
113+
"files.watcherExclude": {
114+
"**/.git/objects/**": true,
115+
"**/target/**": true,
116+
"**/node_modules/**": true
117+
},
118+
"search.exclude": {
119+
"**/.git": true,
120+
"**/target": true,
121+
"**/node_modules": true,
122+
"**/__pycache__": true,
123+
"**/mutants.out": true
124+
},
125+
"git.autofetch": true,
126+
"scm.diffDecorations": "all",
127+
"editor.bracketPairColorization.enabled": true,
128+
"editor.guides.bracketPairs": true,
129+
"editor.inlineSuggest.enabled": true,
130+
"editor.smoothScrolling": true,
131+
"editor.stickyScroll.enabled": true,
132+
"editor.minimap.enabled": false,
133+
"editor.wordWrap": "on",
134+
"terminal.integrated.defaultProfile.linux": "bash",
135+
"yaml.schemas": {
136+
"https://json.schemastore.org/github-workflow.json": ".github/workflows/*.yml"
137+
},
138+
"markdownlint.run": "onSave",
61139
"rust-analyzer.cargo.features": "all",
62140
"rust-analyzer.check.command": "clippy",
63141
"rust-analyzer.check.allTargets": true,

.devcontainer/welcome.sh

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,13 @@ echo ""
1717
echo " TLA+ Model Checker:"
1818
echo " tlc <spec.tla> # Run TLC model checker"
1919
echo " sany <spec.tla> # Parse/check TLA+ spec"
20-
echo " ./scripts/verify-tla.sh # Verify all TLA+ specs"
20+
echo " ./scripts/verification/verify-tla.sh # Verify all TLA+ specs"
2121
echo ""
2222
echo " Kani (Bounded Model Checker for Rust):"
2323
echo " cargo kani # Run all Kani proofs"
2424
echo " cargo kani --harness <name> # Run specific proof"
25-
echo " ./scripts/verify-kani.sh # Verify with helper script"
26-
echo " ./scripts/check-kani-coverage.sh # Validate proof coverage"
25+
echo " ./scripts/verification/verify-kani.sh # Verify with helper script"
26+
echo " ./scripts/verification/check-kani-coverage.sh # Validate proof coverage"
2727
echo ""
2828
echo " Miri (Undefined Behavior Detection):"
2929
echo " cargo +nightly miri test # Run tests under Miri"
@@ -63,7 +63,7 @@ echo " Profiling:"
6363
echo " cargo flamegraph # Generate flame graphs"
6464
echo ""
6565
echo " All-in-one verification:"
66-
echo " ./scripts/verify-all.sh # Run all verifiers"
66+
echo " ./scripts/verification/verify-all.sh # Run all verifiers"
6767
echo ""
6868
echo " High-Performance CLI Tools (aliased, use instead of standard):"
6969
echo " rg (ripgrep) → grep fd → find"

.github/copilot-instructions.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
# GitHub Copilot Instructions for Fortress Rollback
22

33
**Read and follow [`.llm/context.md`](../.llm/context.md)** — the canonical source of truth for all project context, development policies, testing guidelines, and coding standards. You must read it before making any changes.
4+
5+
When clarifying questions are needed, follow [`.llm/templates/ask-user-question.md`](../.llm/templates/ask-user-question.md) to keep questions concise and actionable.

.github/workflows/ci-benchmarks.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ jobs:
5959
- name: Verify sccache is working
6060
id: sccache-check
6161
if: steps.sccache.outcome == 'success'
62-
run: ./scripts/verify-sccache.sh
62+
run: ./scripts/ci/verify-sccache.sh
6363
continue-on-error: true
6464

6565
- name: Clear sccache env on failure
@@ -203,7 +203,7 @@ jobs:
203203
- name: Verify sccache is working
204204
id: sccache-check
205205
if: steps.sccache.outcome == 'success'
206-
run: ./scripts/verify-sccache.sh
206+
run: ./scripts/ci/verify-sccache.sh
207207
continue-on-error: true
208208

209209
- name: Clear sccache env on failure

.github/workflows/ci-docs.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ jobs:
157157
python-version: '3.12'
158158

159159
- name: Validate wiki consistency
160-
run: python scripts/check-wiki-consistency.py
160+
run: python scripts/docs/check-wiki-consistency.py
161161

162162
# ============================================================================
163163
# SCRIPT TESTS & WIKI DRY-RUN - Run all script tests and validate wiki generation
@@ -181,10 +181,10 @@ jobs:
181181
run: python -m pytest scripts/tests/ -v
182182

183183
- name: Generate wiki (dry-run)
184-
run: python scripts/sync-wiki.py --dest wiki-test-output
184+
run: python scripts/docs/sync-wiki.py --dest wiki-test-output
185185

186186
- name: Validate generated wiki format
187-
run: python scripts/validate-wiki-output.py --wiki-dir wiki-test-output
187+
run: python scripts/docs/validate-wiki-output.py --wiki-dir wiki-test-output
188188

189189
markdown-lint:
190190
name: Markdown Lint
@@ -223,7 +223,7 @@ jobs:
223223
echo ""
224224
} >> "$GITHUB_STEP_SUMMARY"
225225
226-
if ./scripts/check-code-fence-syntax.sh docs/ 2>&1 | tee code-fence-output.txt; then
226+
if ./scripts/docs/check-code-fence-syntax.sh docs/ 2>&1 | tee code-fence-output.txt; then
227227
echo "✅ No rustdoc-style code fence attributes found" >> "$GITHUB_STEP_SUMMARY"
228228
else
229229
{
@@ -272,7 +272,7 @@ jobs:
272272
continue-on-error: true # External links may have rate limits
273273

274274
- name: Check local file links
275-
run: ./scripts/check-links.sh
275+
run: ./scripts/docs/check-links.sh
276276

277277
# ============================================================================
278278
# LYCHEE LINK CHECK - Fast link checking with caching
@@ -571,7 +571,7 @@ jobs:
571571
} >> "$GITHUB_STEP_SUMMARY"
572572
573573
# Run verification on docs/ directory with fail-fast for CI
574-
if ./scripts/verify-markdown-code.sh --fail-fast docs/ 2>&1 | tee markdown-code-output.txt; then
574+
if ./scripts/docs/verify-markdown-code.sh --fail-fast docs/ 2>&1 | tee markdown-code-output.txt; then
575575
echo "✅ All markdown code samples compile successfully" >> "$GITHUB_STEP_SUMMARY"
576576
else
577577
{

.github/workflows/ci-llm-lint.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,4 @@ jobs:
2020
- name: Verify skills index is up-to-date
2121
run: python scripts/hooks/regenerate-skills-index.py --check
2222
- name: Check skill code example quality
23-
run: bash scripts/check-llm-skills.sh
23+
run: bash scripts/docs/check-llm-skills.sh

.github/workflows/ci-network.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ on:
88
- '**/Cargo.toml'
99
- '**/Cargo.lock'
1010
- 'docker/**'
11-
- 'scripts/docker-network-tests.sh'
11+
- 'scripts/ci/docker-network-tests.sh'
1212
- '.github/workflows/ci-network.yml'
1313
pull_request:
1414
branches: [main]
@@ -17,7 +17,7 @@ on:
1717
- '**/Cargo.toml'
1818
- '**/Cargo.lock'
1919
- 'docker/**'
20-
- 'scripts/docker-network-tests.sh'
20+
- 'scripts/ci/docker-network-tests.sh'
2121
- '.github/workflows/ci-network.yml'
2222
workflow_dispatch:
2323

@@ -56,14 +56,14 @@ jobs:
5656
- name: Verify sccache is working (Unix)
5757
id: sccache-check-unix
5858
if: steps.sccache.outcome == 'success' && runner.os != 'Windows'
59-
run: ./scripts/verify-sccache.sh
59+
run: ./scripts/ci/verify-sccache.sh
6060
continue-on-error: true
6161

6262
- name: Verify sccache is working (Windows)
6363
id: sccache-check-windows
6464
if: steps.sccache.outcome == 'success' && runner.os == 'Windows'
6565
shell: bash
66-
run: ./scripts/verify-sccache.sh
66+
run: ./scripts/ci/verify-sccache.sh
6767
continue-on-error: true
6868

6969
- name: Set sccache working status
@@ -131,7 +131,7 @@ jobs:
131131
run: docker compose -f docker/docker-compose.yml build
132132

133133
- name: Run Docker network tests (quick)
134-
run: ./scripts/docker-network-tests.sh --quick
134+
run: ./scripts/ci/docker-network-tests.sh --quick
135135
timeout-minutes: 15
136136

137137
- name: Upload test results

.github/workflows/ci-rust.yml

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ on:
1212
- 'Cross.toml'
1313
- 'clippy.toml'
1414
- 'rustfmt.toml'
15-
- 'scripts/verify-kani.sh'
16-
- 'scripts/check-kani-coverage.sh'
15+
- 'scripts/verification/verify-kani.sh'
16+
- 'scripts/verification/check-kani-coverage.sh'
1717
- '.github/workflows/ci-rust.yml'
1818
pull_request:
1919
branches: [main]
@@ -26,8 +26,8 @@ on:
2626
- 'Cross.toml'
2727
- 'clippy.toml'
2828
- 'rustfmt.toml'
29-
- 'scripts/verify-kani.sh'
30-
- 'scripts/check-kani-coverage.sh'
29+
- 'scripts/verification/verify-kani.sh'
30+
- 'scripts/verification/check-kani-coverage.sh'
3131
- '.github/workflows/ci-rust.yml'
3232
workflow_dispatch:
3333

@@ -45,7 +45,7 @@ jobs:
4545
- uses: actions/checkout@v6
4646

4747
- name: Validate Kani proof coverage
48-
run: ./scripts/check-kani-coverage.sh
48+
run: ./scripts/verification/check-kani-coverage.sh
4949
# Ensures all #[kani::proof] functions are in tier lists in verify-kani.sh
5050

5151
# Primary build and test job (cross-platform matrix)
@@ -79,14 +79,14 @@ jobs:
7979
- name: Verify sccache is working (Unix)
8080
id: sccache-check-unix
8181
if: steps.sccache.outcome == 'success' && runner.os != 'Windows'
82-
run: ./scripts/verify-sccache.sh
82+
run: ./scripts/ci/verify-sccache.sh
8383
continue-on-error: true
8484

8585
- name: Verify sccache is working (Windows)
8686
id: sccache-check-windows
8787
if: steps.sccache.outcome == 'success' && runner.os == 'Windows'
8888
shell: bash
89-
run: ./scripts/verify-sccache.sh
89+
run: ./scripts/ci/verify-sccache.sh
9090
continue-on-error: true
9191

9292
- name: Set sccache working status
@@ -433,7 +433,7 @@ jobs:
433433
- name: Verify sccache is working
434434
id: sccache-check
435435
if: steps.sccache.outcome == 'success'
436-
run: ./scripts/verify-sccache.sh
436+
run: ./scripts/ci/verify-sccache.sh
437437
continue-on-error: true
438438

439439
- name: Clear sccache env on failure
@@ -522,7 +522,7 @@ jobs:
522522
- name: Verify sccache is working
523523
id: sccache-check
524524
if: steps.sccache.outcome == 'success'
525-
run: ./scripts/verify-sccache.sh
525+
run: ./scripts/ci/verify-sccache.sh
526526
continue-on-error: true
527527

528528
- name: Clear sccache env on failure

0 commit comments

Comments
 (0)