Skip to content

feat: enhance CI and smoke testing with checksum verification and modular job structure #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
amichne wants to merge 2 commits into from
Closed

feat: enhance CI and smoke testing with checksum verification and modular job structure #61

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eea3c82
feat: enhance CI and smoke testing with checksum verification and mod…
amichne Apr 15, 2026
89b449e
Upgrading CI for robustness
amichne Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/scripts/smoke-installer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,19 +83,21 @@ trap cleanup EXIT
cp "$portable_zip" "$asset_path"

python3 - "$metadata_path" "$asset_path" <<'PY'
import hashlib
import json
import sys
from pathlib import Path

metadata_path = Path(sys.argv[1])
asset_path = Path(sys.argv[2])
digest = hashlib.sha256(asset_path.read_bytes()).hexdigest()
payload = {
"tag_name": "v0.0.0-smoke",
"assets": [
{
"name": asset_path.name,
"browser_download_url": asset_path.as_uri(),
"digest": "",
"digest": f"sha256:{digest}",
Comment on lines 85 to +100
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

This still doesn't prove checksum enforcement.

The digest is computed from the same ZIP the installer later consumes, so this path stays green even if install.sh ignores assets[0].digest completely. Please add a negative case that tampers with the digest or asset and asserts the install fails before creating the target install tree.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/scripts/smoke-installer.sh around lines 85 - 100, Add a negative
tamper test to the smoke-installer workflow: after computing the real digest
(variable digest) and building payload, create a second test case that either
corrupts the asset (modify asset_path bytes) or sets a bogus digest (e.g.,
tampered_digest) in the payload, then run the installer (install.sh) against
that tampered metadata and assert it fails and does not create the target
install tree; specifically, reuse metadata_path and asset_path variables but
produce a payload where "assets[0].digest" is the wrong sha256 and verify
install.sh exits non‑zero and the target install directory is absent before
exiting.

}
],
}
Expand Down
38 changes: 38 additions & 0 deletions .github/scripts/smoke-kast-cli.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,44 @@ assert diagnostics["diagnostics"][0]["code"] == "UNRESOLVED_REFERENCE"
edit_files = {Path(edit["filePath"]).name for edit in rename["edits"]}
assert edit_files == {"Greeter.kt", "Use.kt", "SecondaryUse.kt"}
assert set(Path(path).name for path in rename["affectedFiles"]) == edit_files

# Build an ApplyEditsQuery from the rename result
apply_edits_query = {
"edits": rename["edits"],
"fileHashes": rename["fileHashes"],
"fileOperations": [],
}
(tmp_dir / "apply-edits-request.json").write_text(
json.dumps(apply_edits_query), encoding="utf-8"
)
PY

KAST_CONFIG_HOME="$instance_dir" \
"$KAST_CMD" apply-edits \
--workspace-root="$workspace_dir" \
--request-file="${tmp_dir}/apply-edits-request.json" \
--wait-timeout-ms=180000 >"${tmp_dir}/apply-edits.json"

python3 - "$tmp_dir" "$workspace_dir" <<'PY'
import json
import sys
from pathlib import Path

tmp_dir = Path(sys.argv[1])
workspace_dir = Path(sys.argv[2])
source_root = workspace_dir / "src/main/kotlin/sample"

apply_result = json.loads((tmp_dir / "apply-edits.json").read_text(encoding="utf-8"))
assert len(apply_result.get("applied", [])) > 0, f"expected edits to be applied: {apply_result}"

greeter_text = (source_root / "Greeter.kt").read_text(encoding="utf-8")
use_text = (source_root / "Use.kt").read_text(encoding="utf-8")
secondary_text = (source_root / "SecondaryUse.kt").read_text(encoding="utf-8")

assert "welcome" in greeter_text, f"Greeter.kt should contain 'welcome' after apply-edits: {greeter_text}"
assert "greet" not in greeter_text, f"Greeter.kt should not contain 'greet' after rename: {greeter_text}"
assert "welcome" in use_text, f"Use.kt should contain 'welcome' after apply-edits: {use_text}"
assert "welcome" in secondary_text, f"SecondaryUse.kt should contain 'welcome' after apply-edits: {secondary_text}"
PY

KAST_CONFIG_HOME="$instance_dir" \
Expand Down
91 changes: 77 additions & 14 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true

jobs:
cli-smoke:
name: CLI smoke (${{ matrix.os }})
build-and-test:
name: Build & test (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -25,12 +25,10 @@ jobs:
steps:
- uses: actions/checkout@v5

- uses: graalvm/setup-graalvm@v1
- uses: actions/setup-java@v4
with:
distribution: graalvm-community
distribution: temurin
java-version: "21"
github-token: ${{ secrets.GITHUB_TOKEN }}
native-image-job-reports: "true"

- uses: gradle/actions/setup-gradle@v4

Expand All @@ -39,18 +37,72 @@ jobs:
path: ~/.gradle/kast/intellij-distributions
key: idea-dist-${{ hashFiles('gradle/libs.versions.toml') }}

- name: Build and test Kast
- name: Test and build portable distribution
run: >
./gradlew
./gradlew -PjvmOnly
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 JVM-only build breaks smoke-installer.sh: installer expects native binary absent from distribution

The CI build step now uses -PjvmOnly (line 42), which produces a portable distribution without bin/kast (the native binary is only included when jvmOnly property is absent — see kast/build.gradle.kts:87-92). However, the downstream "Smoke installer" step (line 136) runs smoke-installer.sh, which invokes install.sh without --jvm-only. Since install.sh defaults jvm_only="false" (install.sh:424), it checks [[ -f "${release_dir}/bin/kast" ]] at install.sh:653 and dies with "Installed archive did not contain the kast native binary". Even if that check were bypassed, smoke-installer.sh:125 independently asserts [[ -x "${installed_root}/bin/kast" ]]. Both checks fail because the JVM-only zip never contains bin/kast.

The two failing assertions

install.sh:652-653 (reached first):

if [[ "$jvm_only" != "true" ]]; then
  [[ -f "${release_dir}/bin/kast" ]] || die "Installed archive did not contain the kast native binary"

smoke-installer.sh:125 (would also fail):

[[ -x "${installed_root}/bin/kast" ]] || die "Installed kast native binary is missing from ${installed_root}/bin"
Prompt for agents 
The build step uses -PjvmOnly to skip native compilation (since GraalVM was replaced with Temurin), but the smoke-installer.sh test still expects a non-JVM-only distribution. There are two places that need fixing:

1. smoke-installer.sh line 117: The /bin/bash -c invocation of install.sh needs --jvm-only appended, e.g.: /bin/bash -c "$installer_content" -- --jvm-only

2. smoke-installer.sh line 125: The assertion [[ -x "${installed_root}/bin/kast" ]] needs to be removed or made conditional on whether the distribution is JVM-only.

Alternatively, if the intent is to test the full native path in CI, revert to GraalVM setup and remove -PjvmOnly. But given the deliberate switch to Temurin, the smoke-installer.sh should be updated to match the JVM-only distribution.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

:analysis-api:test
:analysis-server:test
:backend-standalone:test
:kast:test
:kast:portableDistZip
Comment thread
coderabbitai[bot] marked this conversation as resolved.

- name: Upload portable distribution
uses: actions/upload-artifact@v4
with:
name: kast-portable-${{ matrix.os }}
path: kast/build/distributions/kast-*-portable.zip
if-no-files-found: error
retention-days: 1

test-intellij-plugin:
name: IntelliJ plugin
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5

- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"

- uses: gradle/actions/setup-gradle@v4

- uses: actions/cache@v4
with:
path: ~/.gradle/kast/intellij-distributions
key: idea-dist-${{ hashFiles('gradle/libs.versions.toml') }}

- name: Test and verify IntelliJ plugin
run: >
./gradlew
:backend-intellij:test
:backend-intellij:buildPlugin
:backend-intellij:verifyPluginStructure
:backend-intellij:verifyPluginXmlPresent
:kast:test
:kast:portableDistZip

smoke-kast-cli:
name: Smoke CLI (${{ matrix.os }})
runs-on: ${{ matrix.os }}
needs: build-and-test
strategy:
fail-fast: false
matrix:
os:
- ubuntu-latest
- macos-latest
steps:
- uses: actions/checkout@v5

- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"

- name: Download portable distribution
uses: actions/download-artifact@v4
with:
name: kast-portable-${{ matrix.os }}
path: kast/build/distributions

- name: Smoke portable Kast distribution
shell: bash
Expand All @@ -77,7 +129,7 @@ jobs:
eval-agent-routing:
name: Eval agent routing
runs-on: ubuntu-latest
needs: cli-smoke
needs: smoke-kast-cli
steps:
- uses: actions/checkout@v5

Expand All @@ -86,10 +138,21 @@ jobs:
distribution: temurin
java-version: "21"

- uses: gradle/actions/setup-gradle@v4
- name: Download portable distribution
uses: actions/download-artifact@v4
with:
name: kast-portable-ubuntu-latest
path: kast/build/distributions

- name: Build kast
run: ./gradlew --no-daemon :kast:installDist
- name: Prepare kast binary
shell: bash
run: |
set -euo pipefail
dist_dir="$RUNNER_TEMP/kast-dist"
rm -rf "$dist_dir"
mkdir -p "$dist_dir"
unzip -q kast/build/distributions/kast-*-portable.zip -d "$dist_dir"
echo "KAST_BIN=$dist_dir/kast/kast" >> "$GITHUB_ENV"
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

- name: Run kast-routing evals
run: bash evals/harness/run-evals.sh --suite=kast-routing --format=json
Expand Down
Loading