Fix Parquet LIST/MAP wrapper extraction in native and Avro readers

[xiangfu0]

Problem

The native (ParquetNativeRecordReader) and Avro-backed (ParquetAvroRecordReader) Parquet readers were leaking the LIST/MAP wrapper structs into Pinot rows. A column of array<string> came back as [{"element":"abc"}, {"element":"xyz"}] instead of ["abc", "xyz"], and a map<string,string> came back as {"key_value": [{"key":"k","value":"v"}]} instead of {"k":"v"}. The broken shape propagated into segment generation, multi-value scans, and JSON paths.

Fixes #17420.

Before

{
  "topLevelTags": [{"element": "top-a"}, {"element": "top-b"}],
  "topLevelProperties": {"key_value": [{"key": "k-a", "value": "v-a"}, {"key": "k-b", "value": "v-b"}]},
  "metadata": {
    "element": "real-element-field",
    "tags": [{"element": "abc"}, {"element": "xyz"}]
  }
}

After

{
  "topLevelTags": ["top-a", "top-b"],
  "topLevelProperties": {"k-a": "v-a", "k-b": "v-b"},
  "metadata": {
    "element": "real-element-field",
    "tags": ["abc", "xyz"]
  }
}

Fix

Drive wrapper detection from the file schema, not from the extracted row data.

ParquetNativeRecordExtractor — use Parquet LogicalTypeAnnotation to tell LIST and MAP groups apart from plain structs.

• extractList: hoist the per-row isListElementWrapper check out of the loop and dispatch the whole list down a single branch. Handles the standard 3-level encoding and legacy 2-level encodings (repeated primitive, repeated multi-field group, repeated single-field group not named element).
• extractKeyValueMap: resolve the key/value field indices once from the schema and reuse them for every entry.

ParquetAvroRecordExtractor — check the Avro field schema first, then recurse via transformValue on the inner element field so nested LIST<LIST<...>> wrappers and handleDeprecatedTypes (e.g. INT96 → long) keep applying at every level.

Real struct fields named element are preserved; only fields the external schema actually identifies as LIST/MAP wrappers are normalized.

Note on MAP order

Parquet does not guarantee MAP entries are returned in any particular order on read (neither sorted nor insertion order). Writers, page boundaries, and dictionary encodings can all reorder entries. If you need a stable order, use LIST<STRUCT<key, value>> instead of the native MAP logical type. This is documented inline on extractKeyValueMap.

Tests

ParquetCollectionRecordReaderTest covers:

• Hand-authored Parquet schemas through both readers.
• Avro-schema-written Parquet files through both readers.
• A checked-in golden Parquet fixture (collection-reader-fixture.parquet) with primitive types, DECIMAL/DATE/TIMESTAMP logical types, nested structs, LIST and MAP of scalars, struct lists, empty collections, and a real struct field named element.
• Legacy 2-level list encodings (single-field non-element, multi-field, single-field-named-element wrapping a struct).
• Nullable list elements.
• Nested LIST<LIST<STRING>> through the Avro reader, including null inner element and null inner wrapper.

Validation

• ./mvnw -pl pinot-plugins/pinot-input-format/pinot-parquet test — 19/19 pass.
• ./mvnw spotless:apply checkstyle:check license:check -pl pinot-plugins/pinot-input-format/pinot-parquet — clean.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 63.39%. Comparing base (068907e) to head (0638c31).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
...utformat/parquet/ParquetNativeRecordExtractor.java	96.96%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18325      +/-   ##
============================================
- Coverage     63.61%   63.39%   -0.22%     
- Complexity     1659     1668       +9     
============================================
  Files          3246     3252       +6     
  Lines        197549   198661    +1112     
  Branches      30577    30770     +193     
============================================
+ Hits         125662   125950     +288     
- Misses        61847    62642     +795     
- Partials      10040    10069      +29     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	?
java-21	63.39% <97.14%> (-0.19%)	⬇️
temurin	63.39% <97.14%> (-0.22%)	⬇️
unittests	63.39% <97.14%> (-0.22%)	⬇️
unittests1	55.37% <ø> (-0.23%)	⬇️
unittests2	34.90% <97.14%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Found one high-signal correctness issue; see inline comment.