TestDockerStart - `panic: Log in goroutine after TestDockerStart has completed`

[belimawr]

Flaky Test

• Test Name: TestDockerStart
• Link:
• Branch: 9.3
• Artifact Link: https://buildkite.com/elastic/beats-libbeat/builds/29269/canvas?jid=019e02c9-853c-457f-ada3-ed4ce712d8b7&tab=output#L432
• Notes:

Stack Trace

=== Failed
=== FAIL: libbeat/autodiscover/providers/docker  (0.00s)
PASS
panic: Log in goroutine after TestDockerStart has completed: 2026-05-07T14:34:18.278Z	DEBUG	docker.docker	Got a new docker event: {destroy 03d3ca6bf4730e32ff200d3c8a2a145d2599b7160d4fc8e1f0abc50115c1ac71 busybox:latest container destroy {03d3ca6bf4730e32ff200d3c8a2a145d2599b7160d4fc8e1f0abc50115c1ac71 map[image:busybox:latest label:foo label.child:bar name:festive_bhabha]} local 1778164458 1778164458275471000}

[botelastic]

This issue doesn't have a Team:<team> label.

[github-actions]

tl;dr: The most likely fix is to make Docker autodiscover shutdown synchronous (watcher.Stop() + wait for the provider goroutine), because TestDockerStart now uses a test-bound logger and provider shutdown currently leaves watcher goroutines running long enough to log after the test ends.

Recommendation

Update libbeat/autodiscover/providers/docker/docker.go so Provider.Stop() stops the Docker watcher and blocks until the Start() goroutine has fully exited. This directly targets the post-test goroutine logging panic reported in #50545.

Findings

1. TestDockerStart uses a logger tied to testing.T:

   • libbeat/autodiscover/providers/docker/docker_integration_test.go:40
   • log := logptest.NewTestingLogger(t, "docker")

2. The provider starts async processing and watcher listeners:

   • libbeat/autodiscover/providers/docker/docker.go:114 — if err := watcher.Start(); err != nil {
   • libbeat/autodiscover/providers/docker/docker.go:136-162 — Start() runs go func() { ... }

3. Stop() is currently non-blocking and does not stop the watcher:

   • libbeat/autodiscover/providers/docker/docker.go:400-402
   • func (d *Provider) Stop() { close(d.stop) }

4. Test teardown can still trigger Docker events near test end:

   • libbeat/autodiscover/providers/docker/docker_integration_test.go:64 — defer provider.Stop()
   • libbeat/autodiscover/providers/docker/docker_integration_test.go:76-80 — deferred ContainerRemove(ID)

5. In similar provider implementations, Stop() explicitly stops watchers:

   • libbeat/autodiscover/providers/kubernetes/pod.go:347 — p.watcher.Stop()
   • libbeat/autodiscover/providers/kubernetes/node.go:197 — n.watcher.Stop()
   • libbeat/autodiscover/providers/kubernetes/service.go:203 — s.watcher.Stop()

6. Historical context confirms this test has flaked before:

   • Issue: [Libbeat][test failures] master branch got some test failures for the last few days #20306 (docker.TestDockerStart listed)
   • PR: Skip TestDockerStart #20361 (temporary skip)
   • PR: Fix TestDockerStart flaky test #21681 (flaky test fix)

7. Recent logger refactor increased sensitivity of late logs in this test path:

   • git blame shows docker_integration_test.go:40 from commit 46bc5fdd942 (PR [Chore] Replace global logger instances with local logger #6 #44489)
   • git blame shows docker_integration_test.go:42 from commit 8e233dab5dc (PR [Chore] Replace global logger with local logger #12 #45463)

Verification

I ran the focused test in this workspace:

$ go test -v -tags integration ./libbeat/autodiscover/providers/docker -run TestDockerStart -count=1
=== RUN   TestDockerStart
    logger.go:146: 2026-05-07T15:08:35.095Z	DEBUG	docker.docker	Docker client will negotiate the API version on the first request.
    logger.go:146: 2026-05-07T15:08:35.095Z	DEBUG	docker.docker.docker	Docker client will negotiate the API version on the first request.
    docker_integration_test.go:60: error setting up docker autodiscover provider: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
--- FAIL: TestDockerStart (0.00s)
FAIL

So I could not reproduce the exact panic locally due missing Docker daemon, but the code-path evidence above is consistent with the reported panic string and logger scope.

Detailed Action Plan

1. In libbeat/autodiscover/providers/docker/docker.go, extend Provider with a completion signal for the Start() goroutine (e.g., sync.WaitGroup or done chan struct{}).
2. In Start() (docker.go:136-162), ensure the completion signal is always released when the goroutine exits.
3. In Stop() (docker.go:400-402):
   • keep the existing close(d.stop) semantics,
   • call d.watcher.Stop() to stop Docker event ingestion,
   • wait for the Start() goroutine to finish before returning.
4. Guard Stop() against double-close (if needed by current call sites) to avoid introducing close of closed channel panics.
5. Re-run focused integration test:
   • go test -v -tags integration ./libbeat/autodiscover/providers/docker -run TestDockerStart -count=1
6. If flakiness remains, add a regression assertion in docker_integration_test.go around teardown timing/logging and run the same test repeatedly (script/stresstest.sh) in a Docker-enabled CI/dev environment.

Related Items

Type	Link	Relevance
Issue	#50545	Current flaky panic report
Issue	#20306	Historical failures included docker.TestDockerStart
PR	#20361	Temporary skip of TestDockerStart
PR	#21681	Prior flaky-test fix for TestDockerStart
PR	#44489	Logger refactor affecting test logger usage
PR	#45463	Logger wiring updates including docker test client creation
File	libbeat/autodiscover/providers/docker/docker.go:136-162	Async provider goroutine lifecycle
File	libbeat/autodiscover/providers/docker/docker.go:400-402	Current Stop() implementation
File	libbeat/autodiscover/providers/docker/docker_integration_test.go:40	Test logger bound to testing.T
File	libbeat/autodiscover/providers/docker/docker_integration_test.go:64	Deferred provider stop

I couldn’t apply labels via available tooling in this run; please add the triaged label when you accept this triage.

Note

🔒 Integrity filter blocked 23 items

The following items were blocked because they don't meet the GitHub integrity level.

• #50371 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50521 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50157 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50185 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #47942 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #47820 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #47918 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #20360 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #11627 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #19662 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #20382 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #20381 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #20361 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #48623 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50372 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #47724 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 7 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.