[FTR] add mocha retry support (testing only — do not merge)#268582
Draft
TamerlanG wants to merge 4 commits intoelastic:mainfrom
Draft
[FTR] add mocha retry support (testing only — do not merge)#268582TamerlanG wants to merge 4 commits intoelastic:mainfrom
TamerlanG wants to merge 4 commits intoelastic:mainfrom
Conversation
Adds a retries setting to the FTR mocha config so failed tests are re-run automatically. The schema default is 1 (each test runs at least twice on failure), and an explicit --retries N CLI flag overrides the default per-run without clobbering values set in config files.
Mocha emits a failure for every failed attempt of a retried test. FTR's wrapTestFunction previously triggered lifecycle.testFailure (and the --pauseOnError handler) on every attempt, which would cause screenshot capture, snapshot updates and ES dump-on-fail subscribers to run on transient failures that ultimately pass. Gate the trigger on _currentRetry === _retries so these only fire for the final, real failure.
With retries enabled, mocha emits 'fail' for every failed attempt and 'retry' when it will re-run the test. Update the three FTR reporters to treat retried-then-passed tests as a single passing test: - reporter.js: log a concise 'attempt N/M failed, retrying' line on 'retry'; suppress the multi-line failure block on non-final attempts. - ci_stats_ftr_reporter.ts: skip failCount increments and trackRunnable for non-final attempts so retried tests aren't double-counted. - scout_ftr_reporter.ts: skip TEST_END events for non-final attempts so the report only reflects the final outcome of each test.
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
Contributor
💔 Build Failed
Failed CI Steps
Test Failures
Metrics [docs]
History |
The console reporter increments indent on every 'test' event (each retry attempt) but mocha only emits 'test end' once after the final attempt, so retried tests were leaking +2 indent each and subsequent tests appeared progressively nested in the log output. Unwind the indent in the 'retry' handler so all attempts and all subsequent tests render at the same level.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Testing only — do not merge
This PR is opened purely for testing purposes to exercise the FTR mocha retry behavior in CI. It is not intended to be merged in its current form.
What's in here
Three commits adding a retry mechanism to the Functional Test Runner:
mochaOpts.retriesconfig +--retriesCLI flag — new schema field with default1(each test runs at least twice on failure). Explicit--retries Noverrides per-run; config files can opt out viamochaOpts.retries: 0.wrapTestFunctionindecorate_mocha_ui.jsnow suppresseslifecycle.testFailure(and--pauseOnError) on non-final retry attempts, so screenshots/snapshots/ES dumps only run for the real final failure.reporter.js,ci_stats_ftr_reporter.ts, andscout_ftr_reporter.tsnow handle mocha's'retry'event so retried-then-passed tests aren't double-counted in logs, ci-stats, or Scout reports.Things to validate before this could land for real
@kbn/mocha) hasn't been audited for retry awareness yet.retries: 1may produce false-positive passes in CRUD-style FTR suites that share state acrossitblocks.