[release/v26.1.x] Parallelize acceptance and integration tests (#1407) (#1416)

* Parallelize acceptance and integration tests (#1407)

## Summary

- Parallelize acceptance and integration tests, reducing CI wall time from ~90+ minutes to ~22 minutes
- Add test infrastructure for parallel execution: file-based locking, per-test namespaces, shared operator install, image import caching
- Fix flaky tests exposed by parallel execution
- Add debugging tooling: diagnostics on failure, feature-name namespaces, timing markers
- Speed up CI builds by only compiling for the host architecture and parallelizing Docker image pulls

## Details

### Acceptance test parallelization

Refactored the harpoon test framework to run BDD features in parallel. Features are partitioned by a `@serial` tag — features without it run concurrently via `t.Run` + `t.Parallel()`, while `@serial` features (decommissioning, helm-chart) run sequentially afterward since they perform k3d node operations.

- Moved the Redpanda operator from per-feature helm install to a single shared instance installed during `BeforeSuite`
- Features that need their own operator (upgrade tests) run in isolated vclusters (`@vcluster` tag)
- Setup and teardown are separated so the cluster isn't torn down between parallel and serial phases

### Integration test parallelization

- Refactored `RedpandaControllerSuite` from a testify suite with shared mutable state to parallel subtests with per-test namespaces
- Added `WatchAllNamespaces` option to `testenv.Env` so a single controller manager serves all test namespaces
- Parallelized `charts/redpanda` integration test subtests
- Parallelized `TestLicense` subtests across 4 image variants
- Removed `-p=1` gate on integration tests (now uses Go's default parallelism)

### CI build speedups

- Only compile `linux/amd64` binaries in CI (was building all 4 os/arch combos)
- Parallelize Docker image pre-pulls (~25 images pulled concurrently instead of sequentially)
- Pre-pull k3d infrastructure images (`rancher/k3s`, `k3d-tools`, `k3d-proxy`)
- Batch `k3d image import` calls (one command with all images instead of N separate calls)
- Cache imported images with marker files to avoid redundant imports across parallel test packages

### Cross-process coordination

- Added file-based locking (`flock`) around k3d cluster creation and image imports to prevent conflicts when multiple test packages run simultaneously
- Made CRD installation idempotent (tolerates "already exists" errors)
- Made cert-manager installation idempotent in `helmtest.Setup`
- Added cert-manager webhook readiness wait after k3s manifest installation

### Flaky test fixes

- Fixed `require.FailNow` inside `assert.Eventually` goroutine causing panics in `rpk.go`
- Added namespace scoping to Redpanda list queries to avoid cross-feature interference
- Added `waitForStatefulSetReady` after cluster availability checks (looks up StatefulSet by label, not name)
- Bumped k3d cluster creation timeout from 3 to 5 minutes
- Bumped vcluster creation timeout from 3 to 5 minutes
- Bumped PVC unbinder test timeouts
- Added retry for schema registry readiness in `FactoryOperatorV1`
- Overrode upgrade test operator images to use Docker Hub (pre-loaded into k3d) instead of `docker.redpanda.com`
- Added `--rerun-fails` to gotestsum for automatic retry of flaky tests
- Disabled k3s servicelb (unused in tests)

### Debugging improvements

- Feature namespaces now include the feature name (e.g. `test-basic-cluster-tests-abc123`)
- Added `=== FEATURE START/END/FAILED ===` log markers with timing
- Added `DumpDiagnostics` on feature failure: pod statuses, events, resource descriptions, pod logs
- Diagnostics written to `ACCEPTANCE_ARTIFACTS_DIR` when set (uploaded as CI artifacts)
- Added `dumpDiagnostics` to `testenv.Env` for integration test failures
- Added `DumpContainerLogsOnFailure` helper for testcontainer-based tests
- Added vcluster creation failure diagnostics (pod state + events from host namespace)
- Added `[TestName]` prefix to `waitFor` log messages for traceability in parallel output
- CI test output uses `testname` format (per-test lines with timing, verbose on failure)

### Pre-loaded images

Ensured all Docker images used by tests are pre-pulled and imported into k3d clusters to avoid in-test pulls:
- Added operator images for upgrade tests (`v25.1.3`, `v25.2.2`, `v25.3.1`)
- Added Redpanda images for license tests (`v24.2.9`, `v24.3.1-rc4`)
- Added `redpanda-nightly`, `redpanda-operator-nightly` for topic controller and factory tests

(cherry picked from commit ca83446)

# Conflicts:
#	acceptance/steps/defaults.go
#	acceptance/steps/multicluster.go
#	charts/redpanda/testdata/template-cases.golden.txtar
#	ci/scripts/run-in-nix-docker.sh
#	harpoon/suite.go
#	operator/internal/controller/redpanda/redpanda_controller_test.go
#	pkg/vcluster/vcluster.go

* Fix up merge conflicts and remove non-backported multicluster acceptance test

* fix bad merge

* fix testing t usage

* fix test expectation

* try to fix up backport some more

* update to 26.1.2 for test runs

* regen golden files

* dump logs on cluster config failure

* fix configuration test by backport

* license constant swap

---------

Co-authored-by: Andrew Stucki <andrew.stucki@redpanda.com>

Author: github-actions[bot]

.buildkite/testsuite.yml

@@ -28,6 +28,8 @@ steps:
       queue: k8s-m6id12xlarge
     command: ./ci/scripts/run-in-nix-docker.sh task ci:configure ci:test:unit
     env:
+      BUILD_GOARCH: amd64
+      BUILD_GOOS: linux
       LOG_LEVEL: trace
       OTLP_DIR: /work/artifacts
       OTLP_METRIC_INTERVAL: 5s
@@ -73,6 +75,8 @@ steps:
       queue: k8s-m6id12xlarge
     command: ./ci/scripts/run-in-nix-docker.sh task ci:configure ci:test:integration
     env:
+      BUILD_GOARCH: amd64
+      BUILD_GOOS: linux
       LOG_LEVEL: trace
       OTLP_DIR: /work/artifacts
       OTLP_METRIC_INTERVAL: 5s
@@ -121,6 +125,8 @@ steps:
       queue: k8s-m6id12xlarge
     command: ./ci/scripts/run-in-nix-docker.sh task ci:configure ci:test:acceptance
     env:
+      BUILD_GOARCH: amd64
+      BUILD_GOOS: linux
       LOG_LEVEL: trace
       OTLP_DIR: /work/artifacts
       OTLP_METRIC_INTERVAL: 5s
@@ -169,6 +175,8 @@ steps:
       queue: k8s-m6id12xlarge
     command: ./ci/scripts/run-in-nix-docker.sh task ci:configure ci:test:kuttl-v1
     env:
+      BUILD_GOARCH: amd64
+      BUILD_GOOS: linux
       LOG_LEVEL: trace
       OTLP_DIR: /work/artifacts
       OTLP_METRIC_INTERVAL: 5s
@@ -218,6 +226,8 @@ steps:
       queue: k8s-m6id12xlarge
     command: ./ci/scripts/run-in-nix-docker.sh task ci:configure ci:test:kuttl-v1-nodepools
     env:
+      BUILD_GOARCH: amd64
+      BUILD_GOOS: linux
       LOG_LEVEL: trace
       OTLP_DIR: /work/artifacts
       OTLP_METRIC_INTERVAL: 5s

Taskfile.yml

@@ -264,8 +264,16 @@ tasks:
       _PKG:
         sh: go work edit -json | jq -j '.Use.[].DiskPath + "/... "'
       PKG: '{{ .PKG | default ._PKG }}'
+      # When using gotestsum with --packages, packages must be a single
+      # quoted arg and -- separates gotestsum flags from go test args.
+      _USE_PACKAGES: '{{if contains "--packages" .GO_TEST_RUNNER}}true{{end}}'
     cmds:
-    - '{{.GO_TEST_RUNNER}} {{.PKG}} {{.CLI_ARGS}}'
+    - |
+      {{- if ._USE_PACKAGES}}
+      {{.GO_TEST_RUNNER}} "{{.PKG}}" -- {{.CLI_ARGS}}
+      {{- else}}
+      {{.GO_TEST_RUNNER}} {{.PKG}} {{.CLI_ARGS}}
+      {{- end}}
 
   test:integration:
     desc: "Run all integration tests (~90m)"
@@ -284,7 +292,7 @@ tasks:
       vars:
         GO_TEST_RUNNER:
           ref: .GO_TEST_RUNNER
-        CLI_ARGS: '{{.CLI_ARGS}} -p=1 -run {{.RUN}} -timeout 60m -tags integration'
+        CLI_ARGS: '{{.CLI_ARGS}} -run {{.RUN}} -timeout 60m -tags integration'
 
   test:acceptance:
     desc: "Run all acceptance tests (~90m)"
@@ -330,6 +338,10 @@ tasks:
       TEST_KUBE_VERSION: '{{ .TEST_KUBE_VERSION | default .DEFAULT_TEST_KUBE_VERSION }}'
       TEST_COREDNS_VERSION: '{{ .TEST_COREDNS_VERSION | default .DEFAULT_TEST_COREDNS_VERSION }}'
       IMAGES:
+      # k3d infrastructure images — pre-pulling avoids slow pulls during cluster creation.
+      - rancher/k3s:v1.32.13-k3s1
+      - ghcr.io/k3d-io/k3d-tools:5.8.3
+      - ghcr.io/k3d-io/k3d-proxy:5.8.3
       - quay.io/jetstack/cert-manager-controller:{{.TEST_CERTMANAGER_VERSION}}
       - quay.io/jetstack/cert-manager-cainjector:{{.TEST_CERTMANAGER_VERSION}}
       - quay.io/jetstack/cert-manager-startupapicheck:{{.TEST_CERTMANAGER_VERSION}}
@@ -339,20 +351,38 @@ tasks:
       - quay.io/jetstack/cert-manager-webhook:{{.SECOND_TEST_CERTMANAGER_VERSION}}
       - '{{.TEST_REDPANDA_REPO}}:{{.TEST_REDPANDA_VERSION}}'
       - '{{.DEFAULT_TEST_UPGRADE_REDPANDA_REPO}}:{{.TEST_UPGRADE_REDPANDA_VERSION}}'
+      - redpandadata/redpanda-operator:v25.1.3
+      - redpandadata/redpanda-operator:v25.2.2
       - redpandadata/redpanda-operator:v25.3.1
       - redpandadata/redpanda-operator:{{.TEST_UPGRADE_OPERATOR_VERSION}}
       - ghcr.io/loft-sh/vcluster-pro:{{.TEST_VCLUSTER_VERSION}}
       - registry.k8s.io/kube-controller-manager:{{.TEST_KUBE_VERSION}}
       - registry.k8s.io/kube-apiserver:{{.TEST_KUBE_VERSION}}
       - coredns/coredns:{{.TEST_COREDNS_VERSION}}
+      - redpandadata/redpanda-unstable:v24.3.1-rc4
       - redpandadata/redpanda-unstable:v24.3.1-rc8
+      - redpandadata/redpanda-unstable:v25.2.1-rc7
       - redpandadata/redpanda-unstable:v25.3.1-rc2
+      - redpandadata/redpanda-unstable:v25.3.1-rc4
       - redpandadata/redpanda-unstable:v26.1.1-rc5
+      - redpandadata/redpanda-nightly:v0.0.0-20260330git0d4187b
+      - redpandadata/redpanda-operator-nightly:v0.0.0-20250129gita89e202
+      - redpandadata/redpanda:v23.2.8
+      - redpandadata/redpanda:v24.2.9
+      - redpandadata/redpanda:v25.1.1
+      - redpandadata/redpanda:v25.2.1
+      - redpandadata/redpanda:v25.2.11
       - redpandadata/redpanda:v26.1.1
+      - redpandadata/redpanda:v26.1.2
 
     cmds:
-      - for: {var: IMAGES}
-        cmd: docker inspect {{.ITEM}} > /dev/null || docker pull {{.ITEM}}
+      - |
+        pids=""
+        {{range .IMAGES}}
+        (docker inspect "{{.}}" > /dev/null 2>&1 || docker pull -q "{{.}}") &
+        pids="$pids $!"
+        {{end}}
+        for pid in $pids; do wait "$pid" || true; done
 
   pending-prs:
     desc: "Get all pending PRs for watched branches"

acceptance/features/cluster.feature

@@ -3,7 +3,7 @@
 Feature: Basic cluster tests
   @skip:gke @skip:aks @skip:eks
   Scenario: Updating admin ports
-    # replaces e2e-v2 "upgrade-values-check"   
+    # replaces e2e-v2 "upgrade-values-check"
     Given I apply Kubernetes manifest:
     """
     ---
@@ -44,32 +44,3 @@ Feature: Basic cluster tests
     Then cluster "upgrade" is stable with 1 nodes
     And service "upgrade-external" should have named port "admin-default" with value 9640
     And rpk is configured correctly in "upgrade" cluster
-
-
-  @skip:gke @skip:aks @skip:eks
-  Scenario: Rack Awareness
-    Given I apply Kubernetes manifest:
-    # NB: You wouldn't actually use kubernetes.io/os for the value of rack,
-    # it's just a value that we know is both present and deterministic for the
-    # purpose of testing.
-    """
-    ---
-    apiVersion: cluster.redpanda.com/v1alpha2
-    kind: Redpanda
-    metadata:
-      name: rack-awareness
-    spec:
-      clusterSpec:
-        console:
-          enabled: false
-        statefulset:
-          replicas: 1
-        rackAwareness:
-          enabled: true
-          nodeAnnotation: 'kubernetes.io/os'
-    """
-    And cluster "rack-awareness" is stable with 1 nodes
-    Then running `cat /etc/redpanda/redpanda.yaml | grep -o 'rack: .*$'` will output:
-    """
-    rack: linux
-    """

acceptance/features/console-upgrades.feature

@@ -1,9 +1,13 @@
-@operator:none
+@vcluster
 Feature: Upgrading the operator with Console installed
   @skip:gke @skip:aks @skip:eks
   Scenario: Console v2 to v3 no warnings
     Given I helm install "redpanda-operator" "redpanda/operator" --version v25.1.3 with values:
     """
+    image:
+      repository: redpandadata/redpanda-operator
+    crds:
+      enabled: true
     """
     And I apply Kubernetes manifest:
     """
@@ -47,6 +51,10 @@ Feature: Upgrading the operator with Console installed
   Scenario: Console v2 to v3 with warnings
     Given I helm install "redpanda-operator" "redpanda/operator" --version v25.1.3 with values:
     """
+    image:
+      repository: redpandadata/redpanda-operator
+    crds:
+      enabled: true
     """
     And I apply Kubernetes manifest:
     """

acceptance/features/decommissioning.feature

@@ -1,6 +1,7 @@
+@serial
 Feature: Decommissioning brokers
   # note that this test requires both the decommissioner and pvc unbinder
-  # run in order to pass  
+  # run in order to pass
   @skip:gke @skip:aks @skip:eks
   Scenario: Pruning brokers on failed nodes
     Given I create a basic cluster "decommissioning" with 3 nodes

acceptance/features/helm-chart.feature

@@ -1,4 +1,4 @@
-@operator:none
+@serial
 Feature: Redpanda Helm Chart
 
   Scenario: Tolerating Node Failure

acceptance/features/multicluster.feature

@@ -1,9 +1,11 @@
-@operator:none @vcluster
+@vcluster
 Feature: Upgrading the operator
   @skip:gke @skip:aks @skip:eks
   Scenario: Operator upgrade from 25.2.2
     Given I helm install "redpanda-operator" "redpanda/operator" --version v25.2.2 with values:
     """
+    image:
+      repository: redpandadata/redpanda-operator
     crds:
       enabled: true
     """

acceptance/features/operator-upgrades.feature

@@ -0,0 +1,28 @@
+Feature: Rack Awareness
+  @skip:gke @skip:aks @skip:eks
+  Scenario: Rack Awareness
+    Given I apply Kubernetes manifest:
+    # NB: You wouldn't actually use kubernetes.io/os for the value of rack,
+    # it's just a value that we know is both present and deterministic for the
+    # purpose of testing.
+    """
+    ---
+    apiVersion: cluster.redpanda.com/v1alpha2
+    kind: Redpanda
+    metadata:
+      name: rack-awareness
+    spec:
+      clusterSpec:
+        console:
+          enabled: false
+        statefulset:
+          replicas: 1
+        rackAwareness:
+          enabled: true
+          nodeAnnotation: 'kubernetes.io/os'
+    """
+    And cluster "rack-awareness" is stable with 1 nodes
+    Then running `cat /etc/redpanda/redpanda.yaml | grep -o 'rack: .*$'` will output:
+    """
+    rack: linux
+    """

acceptance/features/rack-awareness.feature

@@ -5,10 +5,3 @@ Feature: Scaling down broker nodes
     And cluster "scaledown" is stable with 5 nodes
     When I scale "scaledown" to 3 nodes
     Then cluster "scaledown" should be stable with 3 nodes
-
-  @skip:gke @skip:aks @skip:eks
-  Scenario: Scaling up nodes
-    Given I create a basic cluster "scaleup" with 1 nodes
-    And cluster "scaleup" is stable with 1 nodes
-    When I scale "scaleup" to 3 nodes
-    Then cluster "scaleup" should be stable with 3 nodes