Failing test: Agent Builder — sidebar error handling - embeddable sidebar error handling

[kibanamachine]

A test failed on a tracked branch

Scout Test Details:

Field	Value
Test ID	817ba19dd1cfe1c-593e50afbb3e170
Target	local-stateful-classic
Location	x-pack/platform/plugins/shared/agent_builder/test/scout_agent_builder/ui/tests/sidebar_error_handling.spec.ts
Duration	80.44s
Module	agentBuilder (plugin)
Config path	x-pack/platform/plugins/shared/agent_builder/test/scout_agent_builder/ui/playwright.config.ts
Code Owners	elastic/workchat-eng

Error: expect(locator).toContainText(expected) failed

Locator: locator('[data-test-subj="agentBuilderRoundResponse"]').filter({ hasText: 'First successful sidebar response' })
Expected substring: "First successful sidebar response"
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toContainText" with timeout 10000ms
  - waiting for locator('[data-test-subj="agentBuilderRoundResponse"]').filter({ hasText: 'First successful sidebar response' })

    at x-pack/platform/plugins/shared/agent_builder/test/scout_agent_builder/ui/tests/sidebar_error_handling.spec.ts:202:11
    at x-pack/platform/plugins/shared/agent_builder/test/scout_agent_builder/ui/tests/sidebar_error_handling.spec.ts:160:7

First failure: kibana-package-registry-verify-and-promote - main

Failure screenshots are available in the Buildkite HTML report and artifacts.

[github-actions]

Investigation Summary

This is a brand-new Scout test, introduced 2 days ago in #263041 (commit 898e7d6 by @machadoum), migrating the existing FTR sidebar error-handling suite to Scout. The failing assertion in the last test.step (keeps previous conversation rounds visible when there is an error) is the only check in the entire spec that is not wrapped in expect(async () => { ... }).toPass(...), so it falls back to Playwright's default 10 s toContainText timeout. The error UI replaces the optimistic round in useConversationRounds (synthetic error round + previous persisted round), and that re-render race almost certainly hasn't fully settled within 10 s on a slow CI agent — a toPass retry would mask it cleanly, matching every other assertion in the file.

Flakiness Finding

• classification: test
• confidence: medium

Suspected Root Cause

• The failing line at sidebar_error_handling.spec.ts:198-202 is the only expect(...).toContainText(...) in this file not wrapped in toPass({ timeout: ... }). Every analogous assertion (lines 59-65, 105-111, 151-157, 177-183) uses a 60–120 s toPass retry. The default 10 s timeout reported in the failure (Timeout: 10000ms) is the Playwright default — strong evidence the assertion never had retry coverage.
• The Scout test was just authored and migrated from FTR in PR #263041 (2026-05-07, @machadoum). It collapses what were 4 separate it() blocks in the original FTR file into a single test() with 4 test.step calls, so the failing step inherits ~60 s of accumulated UI state from prior steps before its assertion runs.
• When the error fires, the send mutation calls setError then removeOptimisticRound and rebuilds conversationRounds to [firstRound, syntheticErrorRound] (use_conversation.ts:157-176, use_send_message_mutation.ts:227-253). During that cache mutation + React re-render, the previous round's agentBuilderRoundResponse is briefly outside the DOM tree the locator is filtering against — exactly the situation toPass is designed for.
• The product check passes for the same scenario in the FTR variant (which uses retry.try everywhere or relies on the FTR retry service's wider tolerance), and RoundResponse always renders the previous round's data-test-subj="agentBuilderRoundResponse" regardless of hasError (round_response.tsx:42-78), so the previous round's text really is in the DOM after settling — there is no product behavior that would permanently hide it.
• No relevant product changes landed between the migration commit and the failure. The two recent send-message PRs (#267980, #267324) predate the test and are about lifting/concurrency, not error rendering.

Proposed Fix

Wrap the trailing assertion in toPass to match the established pattern in the same file. This is the smallest credible fix and brings parity with all the other assertions in the spec.

-        await expect(
-          page.locator('[data-test-subj="agentBuilderRoundResponse"]', {
-            hasText: FIRST_RESPONSE,
-          })
-        ).toContainText(FIRST_RESPONSE);
+        await expect(async () => {
+          await expect(
+            page.locator('[data-test-subj="agentBuilderRoundResponse"]', {
+              hasText: FIRST_RESPONSE,
+            })
+          ).toContainText(FIRST_RESPONSE);
+        }).toPass({ timeout: 60_000 });

Apply at sidebar_error_handling.spec.ts:198-202. After the fix, run the test through the Flaky Test Runner (per the migrating PR's checklist) to confirm the assertion settles within 60 s consistently. If it still flakes, the next investigation should attach a Playwright trace to confirm the previous round actually re-renders (and is not transiently unmounted by a parent — e.g., the Conversation component swap path in useConversationError).

Evidence Used

• Issue body — failure stack at sidebar_error_handling.spec.ts:202:11, Timeout: 10000ms (Playwright default)
• Test file: sidebar_error_handling.spec.ts — note the toPass wrappers around all other toContainText assertions
• FTR predecessor: sidebar_error_handling.ts
• Migration PR/commit: #263041 / 898e7d6 (@machadoum)
• Product code: round_response.tsx, use_conversation.ts#L157-L176, use_send_message_mutation.ts#L146-L255

Generated by Failed Test Investigator for issue #268565 · ● 1.8M · ◷