Skip to content

fix: explain analyze m-cte should work#19797

Open
dqhl76 wants to merge 1 commit into
databendlabs:mainfrom
dqhl76:fix-query-profiles
Open

fix: explain analyze m-cte should work#19797
dqhl76 wants to merge 1 commit into
databendlabs:mainfrom
dqhl76:fix-query-profiles

Conversation

@dqhl76
Copy link
Copy Markdown
Collaborator

@dqhl76 dqhl76 commented Apr 30, 2026

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

fix: m-cte profiles miss

Before this PR, EXPLAIN ANALYZE could lose or mis-attach profile information for nested query executions.

The main problem was that profile collection used plan ids as if they were globally unique inside one QueryContext. However, nested executors, materialized CTE producer CTAS runs, and recursive CTE step pipelines each emit executor-local plan ids. As a result:

  • profiles from different executions could collide or overwrite each other
  • materialized CTE producer plans/profiles were not rendered with the outer explain result
  • recursive CTE step profiles were not attached back under the recursive UnionAll scope
-[ EXPLAIN ]-----------------------------------
Sort(Single)
├── cpu time: 974.872µs
├── output rows: 5
├── output bytes: 88.00 B
├── output columns: [t1.a (#0), t1.cnt (#1), t1.total (#2)]
├── sort keys: [a ASC NULLS LAST]
├── estimated rows: 5.00
└── TableScan
    ├── cpu time: 453.624µs
    ├── wait time: 570.751µs
    ├── output rows: 5
    ├── output bytes: 88.00 B
    ├── bytes scanned: 88.00 B
    ├── read from remote: 159.00 B
    ├── runtime filter bloom time: 1.125µs
    ├── runtime filter inlist/min-max time: 541ns
    ├── table: default.default.t1
    ├── scan id: 0
    ├── output columns: [a (#0), cnt (#1), total (#2)]
    ├── read rows: 5
    ├── read size: < 1 KiB
    ├── partitions total: 1
    ├── partitions scanned: 1
    ├── pruning stats: [segments: <range pruning: 1 to 1 cost: 1 ms>, blocks: <range pruning: 1 to 1 cost: 1 ms>]
    ├── push downs: [filters: [], limit: NONE]
    └── estimated rows: 5.00
MaterializedCTE: t1
└── DistributedInsertSelect
    ├── cpu time: 1.615789ms
    ├── wait time: 7.808586ms
    └── EvalScalar
        ├── cpu time: 41.292µs
        ├── output rows: 5
        ├── output bytes: 90.00 B
        ├── output columns: [COUNT(*) (#2), SUM(number) (#3), a (#4)]
        ├── expressions: [group_item (#1)]
        ├── estimated rows: 33.33
        └── AggregateFinal
            ├── cpu time: 11.273494ms
            ├── wait time: 15.014122ms
            ├── output rows: 5
            ├── output bytes: 90.00 B
            ├── output columns: [COUNT(*) (#2), SUM(number) (#3), number % 5 (#1)]
            ├── group by: [number % 5]
            ├── aggregate functions: [count(), sum(number)]
            ├── estimated rows: 33.33
            └── AggregatePartial
                ├── cpu time: 143.958µs
                ├── output rows: 5
                ├── output bytes: 85.00 B
                ├── group by: [number % 5]
                ├── aggregate functions: [count(), sum(number)]
                ├── estimated rows: 33.33
                └── EvalScalar
                    ├── cpu time: 49.292µs
                    ├── output rows: 100
                    ├── output bytes: 900.00 B
                    ├── output columns: [numbers.number (#0), number % 5 (#1)]
                    ├── expressions: [numbers.number (#0) % 5]
                    ├── estimated rows: 100.00
                    └── TableScan
                        ├── cpu time: 20.5µs
                        ├── output rows: 100
                        ├── output bytes: 800.00 B
                        ├── bytes scanned: 800.00 B
                        ├── table: default.system.numbers
                        ├── scan id: 0
                        ├── output columns: [number (#0)]
                        ├── read rows: 100
                        ├── read size: < 1 KiB
                        ├── partitions total: 1
                        ├── partitions scanned: 1
                        ├── push downs: [filters: [], limit: NONE]
                        └── estimated rows: 100.00

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Apr 30, 2026
@dqhl76 dqhl76 force-pushed the fix-query-profiles branch 4 times, most recently from 2c6539a to bf969bd Compare May 11, 2026 08:19
@dqhl76 dqhl76 changed the title fix: m-cte profiles miss fix: m-cte explain analyze should work May 11, 2026
@dqhl76 dqhl76 changed the title fix: m-cte explain analyze should work fix: explain analyze m-cte should work May 11, 2026
@dqhl76 dqhl76 force-pushed the fix-query-profiles branch from bf969bd to e50e31f Compare May 11, 2026 08:24
Capture materialized CTE producer plans/profiles for EXPLAIN and EXPLAIN ANALYZE, keep executor-local profile ids isolated, and attach recursive CTE step profiles back under the recursive UnionAll scope.
@dqhl76 dqhl76 force-pushed the fix-query-profiles branch from e50e31f to 8cb7795 Compare May 12, 2026 03:22
@dqhl76 dqhl76 marked this pull request as ready for review May 15, 2026 02:45
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cb7795e96

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +218 to +222
ctx.get_function_context()?,
ctx.get_settings(),
ctx.clone(),
);
pipeline.finalize(&plan)?
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore distributed pipeline build for recursive CTE steps

Building recursive-step pipelines with PipelineBuilder::finalize(&plan) bypasses the distributed-plan path that build_query_pipeline_without_render_result_set previously handled. If the recursive branch contains Exchange nodes (common on cluster/warehouse deployments), this path no longer performs fragment/action setup and can fail when executing distributed recursive CTE queries. Please route this code back through the distributed-aware builder (or equivalent logic) while keeping the profile-scope guard.

Useful? React with 👍 / 👎.

Comment on lines 594 to +596
Ok(self
.ctx
.get_query_profiles()
.into_iter()
.filter(|x| x.id.is_some())
.map(|x| (x.id.unwrap(), x))
.collect::<HashMap<_, _>>())
.get_query_profiles_for_execution(&profile_execution_id))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include captured CTE executions in graphical analyze profiles

execute_and_get_profiles now returns only get_query_profiles_for_execution(profile_execution_id), which excludes other execution namespaces such as materialized-CTE producer CTAS runs. Text explain output re-adds those via format_captured_cte_sections, but graphical output uses only this returned map, so EXPLAIN ANALYZE ... GRAPHICAL can silently miss materialized CTE producer profiles.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant