[Flaky Test] github.com/elastic/beats/v7/x-pack/metricbeat/module/cockroachdb/status TestFetch

[elastic-vault-github-plugin-prod]

Flaky Test

• Test Name: TestFetch
• Scope: github.com/elastic/beats/v7/x-pack/metricbeat/module/cockroachdb/status
• File: N/A
• Location: N/A
• Buildkite Link: https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/247467ab-aaf7-8dff-9a0b-0329e049568d
• Flaky Instances: 5
• Latest Occurrence: 2026-05-06T20:55:10.853Z

Details

{
  "id": "247467ab-aaf7-8dff-9a0b-0329e049568d",
  "web_url": "https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/247467ab-aaf7-8dff-9a0b-0329e049568d",
  "scope": "github.com/elastic/beats/v7/x-pack/metricbeat/module/cockroachdb/status",
  "name": "TestFetch",
  "location": null,
  "file_name": null,
  "instances": 5,
  "latest_occurrence_at": "2026-05-06T20:55:10.853Z",
  "most_recent_instance_at": "2026-05-06T20:55:10.853Z",
  "last_resolved_at": null,
  "ownership_team_ids": [],
  "failure_examples_count": 1
}

Failure Examples

Example 1:
Run: https://api.buildkite.com/v2/analytics/organizations/elastic/suites/beats/runs/70800beb-d49b-88aa-af94-86b8cd3fc353
Time: 2026-05-06T19:55:01.229Z
Stacktrace:

=== RUN   TestFetch
    logger.go:146: 2026-05-06T20:03:07.746Z	DEBUG	docker	Docker client will negotiate the API version on the first request.
    logger.go:146: 2026-05-06T20:03:08.271Z	WARN	pull failed for cockroachdb (will build if needed): exit status 18
 cockroachdb Pulling 
 cockroachdb Warning failed to resolve reference "docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1": docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1: not found
#0 building with "default" instance using docker driver

#1 [cockroachdb internal] load build definition from Dockerfile
#1 transferring dockerfile: 242B done
#1 DONE 0.0s

#2 [cockroachdb internal] load metadata for docker.io/cockroachdb/cockroach:v22.1.19
#2 DONE 0.5s

#3 [cockroachdb internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [cockroachdb 1/1] FROM docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22
#4 resolve docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22 done
#4 DONE 0.1s

#5 [cockroachdb] exporting to image
#5 exporting layers done
#5 exporting manifest sha256:6edfe5558ed493d7ac8f1f50fd143be1e5bd494085f68c919846aaeff87e2da0 done
#5 exporting config sha256:04a0f26dd13c201688695d00f92e6674f514e42d62e30d1cab13959c766996ae done
#5 exporting attestation manifest sha256:06bf427678b2ff2fea89e08956f0cc3a40c8a6b8ec8d0c145e0702f3a541594f 0.0s done
#5 exporting manifest list sha256:2f807c05a1bdc85e7ef3548b1daeb12bb3a56e19464bb8842c138d38bc3b3d25 done
#5 naming to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1 done
#5 unpacking to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1
#5 ...

#4 [cockroachdb 1/1] FROM docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22
#4 sha256:a49367d57626b909254c52a27854d7d4163acce34b18157d7d5b08a60b897e34 36.94MB / 36.94MB 0.4s done
#4 sha256:5986e9939b973d0fa55d854b99ee34af7b3955c4bad75c5ad6fa96622897496b 3.39MB / 3.39MB 0.2s done
#4 sha256:94760643ed57a812121e88215c17527f63d5503d08e5dec44fb04bb28eae2537 45.54kB / 45.54kB 0.1s done
#4 sha256:dc52d919f1ed7200c0589cec55e0232fcdbdc97bdee68bf40a08f0800a5fddf4 833.94kB / 833.94kB 0.2s done
#4 sha256:9b4bb62f8c565a804d1ae6cbb7ab495adece603c38ed5c38f936bc9c6d34b8ff 220B / 220B 0.1s done
#4 sha256:973efc01749c95df5a6661cc9d32e5851046942539560d4bd0904cbf4b511315 87.16MB / 87.16MB 0.9s done
#4 DONE 1.2s

#5 [cockroachdb] exporting to image
#5 unpacking to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1 2.8s done
#5 DONE 2.8s
time="2026-05-06T20:03:12Z" level=warning msg="Found orphan containers ([metricbeat_9_5_0_9860cceb44-snapshot-beat-run-d9ee9d016a64 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-48f1ee4299d3 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-06d8c012568e metricbeat_9_5_0_9860cceb44-snapshot-beat-run-fe14a7be7ba3 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-63e921e4f4d1 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-3ac0b78e6157 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-a36ac7d413b1 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-b2384523cb97 metricbeat_9_5_0_9860cceb44-snapshot-beat-run-065dae3584d8]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up."
 Container metricbeat_9_5_0_9860cceb44-snapshot-cockroachdb-1  Creating
 Container metricbeat_9_5_0_9860cceb44-snapshot-cockroachdb-1  Created
 Container metricbeat_9_5_0_9860cceb44-snapshot-cockroachdb-1  Starting
 Container metricbeat_9_5_0_9860cceb44-snapshot-cockroachdb-1  Started
    logger.go:146: 2026-05-06T20:03:17.224Z	DEBUG	cockroachdb.status	Fetching families from 172.18.0.2:8080
    logger.go:146: 2026-05-06T20:03:17.224Z	DEBUG	cockroachdb.status	Making HTTP request: GET http://172.18.0.2:8080/_status/vars
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	Completed dialing successfully	{"network.transport": "tcp", "server.address": "172.18.0.2:8080"}
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	GET http://172.18.0.2:8080/_status/vars request failed with status code: 404
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	error received from prometheus endpoint: 404 page not found
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	Error getting families: unable to decode response from prometheus endpoint: unexpected status code 404 from server
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	Processed 1 families (0 skipped), generated 1 events
    logger.go:146: 2026-05-06T20:03:17.225Z	DEBUG	cockroachdb.status	Fetch completed: reported 1/1 events
    status_integration_test.go:37: Expected 0 error, had 1. [unable to decode response from prometheus endpoint: unexpected status code 404 from server]
--- FAIL: TestFetch (9.48s)

[github-actions]

tl;dr: This looks like the same unresolved flake pattern as #49601 after the endpoint-typo fix—TestFetch still does a single immediate fetch after container health and fails on the first transient HTTP error.

Recommendation

Make x-pack/metricbeat/module/cockroachdb/status/status_integration_test.go resilient to startup timing by using bounded retry before failing (prefer existing helper mbtest.PeriodicReportingFetchV2Error), and set metrics_path: "/_status/vars" explicitly in getConfig so the test does not rely on implicit defaults.

Findings

1. The integration test is one-shot and fail-fast:

// x-pack/metricbeat/module/cockroachdb/status/status_integration_test.go:32-37
service := compose.EnsureUp(t, "cockroachdb")
f := mbtest.NewFetcher(t, getConfig(service.Host()))
events, errs := f.FetchEvents()
if len(errs) > 0 {
    t.Fatalf("Expected 0 error, had %d. %v\n", len(errs), errs)
}

2. EnsureUp gates on Docker health, not metric fetch success:

// libbeat/tests/compose/compose.go:87-89
// Wait for health
err = compose.Wait(upOptions.Timeout, service)

3. Prometheus helper converts any 4xx/5xx into a hard fetch error:

// metricbeat/helper/prometheus/prometheus.go:102-107
if resp.StatusCode > 399 {
    ...
    return nil, fmt.Errorf("unexpected status code %d from server", resp.StatusCode)
}

4. Endpoint mismatch fix from #49601 is already present, so this is not a reintroduced typo:

• x-pack/metricbeat/module/cockroachdb/_meta/Dockerfile:4 → HEALTHCHECK ... /_status/vars
• x-pack/metricbeat/module/cockroachdb/status/manifest.yml:6 → metrics_path: /_status/vars

5. Existing retry helper is available but unused in this test:

• metricbeat/mb/testing/modules.go:269-294 (PeriodicReportingFetchV2Error)
• Example usage pattern: metricbeat/module/prometheus/collector/collector_integration_test.go:29-35

Verification

I ran targeted local checks in this workspace:

go test -v -run TestEventMapping ./x-pack/metricbeat/module/cockroachdb/status
PASS

go test -tags=integration -v -run TestFetch ./x-pack/metricbeat/module/cockroachdb/status
=== RUN   TestFetch
status_integration_test.go:32: listing containers to be killed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
--- FAIL: TestFetch (0.00s)

So I could not reproduce runtime flakiness here due missing Docker daemon, but the code-path evidence above is consistent with intermittent startup-readiness failures.

Detailed Action Plan

1. Update x-pack/metricbeat/module/cockroachdb/status/status_integration_test.go to replace one-shot f.FetchEvents() with bounded retry using mbtest.NewReportingMetricSetV2Error + mbtest.PeriodicReportingFetchV2Error (short period + finite timeout).
2. In the same file, make getConfig explicit with "metrics_path": "/_status/vars".
3. Keep current Docker healthcheck as-is (x-pack/metricbeat/module/cockroachdb/_meta/Dockerfile:4) since it already matches the intended endpoint.
4. Validate in CI Docker environment with repeated runs (e.g. go test -tags=integration -run TestFetch -count=20 ./x-pack/metricbeat/module/cockroachdb/status) to confirm flake reduction.

Related Items

Type	Link	Relevance
Issue	#49601	Same flaky test, prior 404 evidence on /_status/vars.
PR	#49605	Fixed /_stats/vars → /_status/vars healthcheck typo.
Issue	#35211	Older CockroachDB test instability (different root cause).
File	x-pack/metricbeat/module/cockroachdb/status/status_integration_test.go:31-49	Current fail-fast test logic and config helper.
File	libbeat/tests/compose/compose.go:87-89	Health-based readiness gate.
File	metricbeat/helper/prometheus/prometheus.go:102-107	4xx/5xx treated as hard error.
File	metricbeat/mb/testing/modules.go:269-294	Existing retry helper to use.

Note

🔒 Integrity filter blocked 12 items

The following items were blocked because they don't meet the GitHub integrity level.

• #50539 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #35279 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #35249 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #34589 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #28153 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #28127 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #28147 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #28113 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #28084 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50539 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50447 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #43252 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elastic-vault-github-plugin-prod]

Flaky Test Still Occurring

• Latest Occurrence: 2026-05-07T16:23:56.551Z
• Total Instances: 3
• Buildkite Link: https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/247467ab-aaf7-8dff-9a0b-0329e049568d

Details

{
  "id": "247467ab-aaf7-8dff-9a0b-0329e049568d",
  "web_url": "https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/247467ab-aaf7-8dff-9a0b-0329e049568d",
  "scope": "github.com/elastic/beats/v7/x-pack/metricbeat/module/cockroachdb/status",
  "name": "TestFetch",
  "location": null,
  "file_name": null,
  "instances": 3,
  "latest_occurrence_at": "2026-05-07T16:23:56.551Z",
  "most_recent_instance_at": "2026-05-07T16:23:56.551Z",
  "last_resolved_at": null,
  "ownership_team_ids": [],
  "failure_examples_count": 1
}

Failure Examples

Example 1:
Run: https://api.buildkite.com/v2/analytics/organizations/elastic/suites/beats/runs/a8e8f09c-7f15-8dba-824b-e0f96483dd70
Time: 2026-05-07T15:24:09.412Z
Stacktrace:

=== RUN   TestFetch
    logger.go:146: 2026-05-07T15:30:50.626Z	DEBUG	docker	Docker client will negotiate the API version on the first request.
    logger.go:146: 2026-05-07T15:30:51.147Z	WARN	pull failed for cockroachdb (will build if needed): exit status 18
 cockroachdb Pulling 
 cockroachdb Warning failed to resolve reference "docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1": docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1: not found
#0 building with "default" instance using docker driver

#1 [cockroachdb internal] load build definition from Dockerfile
#1 transferring dockerfile: 242B done
#1 DONE 0.0s

#2 [cockroachdb internal] load metadata for docker.io/cockroachdb/cockroach:v22.1.19
#2 DONE 0.5s

#3 [cockroachdb internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [cockroachdb 1/1] FROM docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22
#4 resolve docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22 0.0s done
#4 DONE 0.1s

#5 [cockroachdb] exporting to image
#5 exporting layers done
#5 exporting manifest sha256:52f728461fe5a4abb17a6d9024df9b72e8d62d2279772a9a2ba7679bfac25b0b done
#5 exporting config sha256:63b9ca45e6cfb33ce1316d8c1e92b4dea00b508cab7a925877f9b1e0bce1f9cd done
#5 exporting attestation manifest sha256:c9de31d5514ea0fec7afe36ed7983e12aaa37022dfb3f4402ef4cccf2f1b356a 0.0s done
#5 exporting manifest list sha256:a79a38254a2d5f7ea882dc3ed61dcdc88d1c49d2d6d1ad52f4d6577bb31f617b done
#5 naming to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1 done
#5 unpacking to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1
#5 ...

#4 [cockroachdb 1/1] FROM docker.io/cockroachdb/cockroach:v22.1.19@sha256:0f11b3306212e279f7f6f68ffeef57522fe7b95b3b5adf7190a16e4c48b0dd22
#4 sha256:dc52d919f1ed7200c0589cec55e0232fcdbdc97bdee68bf40a08f0800a5fddf4 833.94kB / 833.94kB 0.1s done
#4 sha256:94760643ed57a812121e88215c17527f63d5503d08e5dec44fb04bb28eae2537 45.54kB / 45.54kB 0.1s done
#4 sha256:5986e9939b973d0fa55d854b99ee34af7b3955c4bad75c5ad6fa96622897496b 3.39MB / 3.39MB 0.2s done
#4 sha256:973efc01749c95df5a6661cc9d32e5851046942539560d4bd0904cbf4b511315 87.16MB / 87.16MB 1.8s done
#4 sha256:9b4bb62f8c565a804d1ae6cbb7ab495adece603c38ed5c38f936bc9c6d34b8ff 220B / 220B 0.1s done
#4 sha256:a49367d57626b909254c52a27854d7d4163acce34b18157d7d5b08a60b897e34 36.94MB / 36.94MB 0.4s done
#4 DONE 1.8s

#5 [cockroachdb] exporting to image
#5 unpacking to docker.elastic.co/integrations-ci/beats-cockroachdb:22.1.19-1 3.5s done
#5 DONE 3.5s
time="2026-05-07T15:30:55Z" level=warning msg="Found orphan containers ([metricbeat_9_5_0_1f653432ee-snapshot-beat-run-d4bbfe4df478 metricbeat_9_5_0_1f653432ee-snapshot-beat-run-c478413219fa metricbeat_9_5_0_1f653432ee-snapshot-beat-run-fd37781043ba metricbeat_9_5_0_1f653432ee-snapshot-beat-run-24cd4fdb15d8 metricbeat_9_5_0_1f653432ee-snapshot-beat-run-cbfd4bbcfadf metricbeat_9_5_0_1f653432ee-snapshot-beat-run-fe4281a8238e metricbeat_9_5_0_1f653432ee-snapshot-beat-run-10ff28843f8a metricbeat_9_5_0_1f653432ee-snapshot-beat-run-911eb9421110 metricbeat_9_5_0_1f653432ee-snapshot-beat-run-d522e71f7bf7]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up."
 Container metricbeat_9_5_0_1f653432ee-snapshot-cockroachdb-1  Creating
 Container metricbeat_9_5_0_1f653432ee-snapshot-cockroachdb-1  Created
 Container metricbeat_9_5_0_1f653432ee-snapshot-cockroachdb-1  Starting
 Container metricbeat_9_5_0_1f653432ee-snapshot-cockroachdb-1  Started
    logger.go:146: 2026-05-07T15:31:00.691Z	DEBUG	cockroachdb.status	Fetching families from 172.18.0.2:8080
    logger.go:146: 2026-05-07T15:31:00.691Z	DEBUG	cockroachdb.status	Making HTTP request: GET http://172.18.0.2:8080/_status/vars
    logger.go:146: 2026-05-07T15:31:00.691Z	DEBUG	cockroachdb.status	Completed dialing successfully	{"network.transport": "tcp", "server.address": "172.18.0.2:8080"}
    logger.go:146: 2026-05-07T15:31:00.692Z	DEBUG	cockroachdb.status	GET http://172.18.0.2:8080/_status/vars request failed with status code: 404
    logger.go:146: 2026-05-07T15:31:00.692Z	DEBUG	cockroachdb.status	error received from prometheus endpoint: 404 page not found
    logger.go:146: 2026-05-07T15:31:00.692Z	DEBUG	cockroachdb.status	Error getting families: unable to decode response from prometheus endpoint: unexpected status code 404 from server
    logger.go:146: 2026-05-07T15:31:00.692Z	DEBUG	cockroachdb.status	Processed 1 families (0 skipped), generated 1 events
    logger.go:146: 2026-05-07T15:31:00.692Z	DEBUG	cockroachdb.status	Fetch completed: reported 1/1 events
    status_integration_test.go:37: Expected 0 error, had 1. [unable to decode response from prometheus endpoint: unexpected status code 404 from server]
--- FAIL: TestFetch (10.07s)