feat(spark-4): add Apache Spark 4.x connector + batch-ingestion plugin (JDK 21+)

[xiangfu0]

Summary

Adds first-class Apache Spark 4.x support to Pinot, alongside the existing Spark 3 connector and batch-ingestion plugin. The two new modules live next to their Spark 3 siblings:

New module	Replaces (for Spark 4 users)	Scope
pinot-connectors/pinot-spark-4-connector	pinot-spark-3-connector	Spark DataSourceV2 read + write
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark-4	pinot-batch-ingestion-spark-3	Segment generation + push runners

Both modules are activated only on JDK 21+ via the pinot-spark-4-connector / pinot-batch-ingestion-spark-4 profile in their respective parent poms (<jdk>[21,)</jdk>). On JDK 11/17 the reactor still builds Spark 3 and completely skips Spark 4, so existing Spark 3 users are unaffected.

Why a separate module?

Apache Spark 4.0 requires JDK 17+, is Scala 2.13 only (no more 2.12), ships Jetty 12 / Jakarta Servlet 5, and made a few small DataSourceV2 API tweaks. Running the Spark 3 connector under Spark 4 doesn't work cleanly. The modules sit side-by-side so:

• Spark 3.5.x users keep the existing pinot-spark-3-connector unchanged.
• Spark 4.0.x users pick up pinot-spark-4-connector.
• The Maven coordinates, artifact names, and shaded-jar layout mirror the Spark 3 pair, so deployment automation (docker assembly, plugin discovery) doesn't need special cases beyond a JDK choice.

End-to-end tutorial

The End-to-end Docker tutorial walks through:

1. Starting Pinot via docker run apachepinot/pinot:latest QuickStart
2. Building the shaded connector jar
3. Building a apache/spark:4.0.0 + JDK 21 image
4. Reading baseballStats (97,889 rows) from Spark 4 via PySpark
5. Creating a target table and writing a DataFrame through the connector
6. Pushing the produced segment tars to the controller (REST or via the SparkSegment*PushJobRunner runners)
7. Reading the written data back through the connector

Every command was executed against apachepinot/pinot:latest and apache/spark:4.0.0 before being checked in.

Quick usage (read)

from pyspark.sql import SparkSession
spark = (SparkSession.builder.appName("pinot-read").getOrCreate())

df = (spark.read.format("pinot")
      .option("table", "baseballStats")
      .option("tableType", "offline")
      .option("controller", "pinot-controller:9000")
      .option("broker", "pinot-broker:8000")
      .load())

df.filter("yearID >= 2000").groupBy("teamID").count().show()

Invoke with spark-submit (Spark 4 on JDK 21):

spark-submit \
  --jars pinot-spark-4-connector-<VER>-shaded.jar,commons-lang3-3.20.0.jar \
  --conf spark.driver.extraClassPath=commons-lang3-3.20.0.jar \
  --conf spark.executor.extraClassPath=commons-lang3-3.20.0.jar \
  my_read_job.py

The prepended commons-lang3-3.20.0.jar works around Spark 4 shipping an older version that lacks ObjectUtils.getIfNull(...). See the tutorial's Known gotchas for the full list.

Quick usage (write)

(df.write.format("pinot")
    .mode("append")
    .option("table", "myTable")
    .option("tableType", "OFFLINE")
    .option("segmentNameFormat", "{table}_{partitionId:03}")
    .option("invertedIndexColumns", "name,category")
    .option("timeColumnName", "ts")
    .option("timeFormat", "EPOCH|MILLISECONDS")
    .option("timeGranularity", "1:MILLISECONDS")
    .option("controller", "pinot-controller:9000")
    .save("s3a://my-bucket/pinot-segments/myTable"))

This produces segment tar files at the save(...) path. Then push them to the controller — either REST for one-offs:

for seg in /path/to/*.tar.gz; do
  curl -X POST -H 'UPLOAD_TYPE: SEGMENT' -F "file=@${seg}" \
    "http://pinot-controller:9000/v2/segments?tableName=myTable&tableType=OFFLINE"
done

…or in production use the runners from pinot-batch-ingestion-spark-4 via a SegmentGenerationJobSpec YAML (see the module's README):

• SparkSegmentGenerationJobRunner — build segments from Avro/CSV/Parquet input
• SparkSegmentTarPushJobRunner — upload tars to the controller
• SparkSegmentUriPushJobRunner — register tars already staged in deep storage (S3/GCS/HDFS)
• SparkSegmentMetadataPushJobRunner — metadata-only push for large segments (avoid streaming the whole tar through the controller)

⚠️ Overwrite semantics

df.write.mode("overwrite").save(...), df.writeTo(...).overwrite(...), and SQL INSERT OVERWRITE all fail fast with UnsupportedOperationException. Pinot's write path can only append new segments — it cannot atomically drop or replace rows matching a Spark predicate. To replace data, drop the table first (controller REST) or use a push runner with REFRESH / consistent-push enabled. Prior to the fixes in this PR, overwrite calls silently appended without replacement, leaving stale data queryable.

Compatibility matrix

Module	Spark	Scala	JDK (build & runtime)
pinot-spark-3-connector / pinot-batch-ingestion-spark-3	3.5.x	2.12 / 2.13	8 / 11 / 17
pinot-spark-4-connector / pinot-batch-ingestion-spark-4	4.0.x	2.13 only	21 only

The Spark 4 jars are compiled with --release 21 (class file major version 65) so they cannot be loaded on a JDK 17 runtime even though Spark 4 itself supports JDK 17.

Correctness fixes (in addition to the Spark 4 port)

Review of the ported Spark 4 code surfaced several pre-existing issues that also exist in the Spark 3 connector. They are fixed symmetrically in both modules (Spark 3 changes are contained to the classes where the bug lives):

1. FilterPushDown compound filter soundness. Or / And / Not were marked as unconditionally supported even if a child was a filter type the connector does not recognize. Spark would remove the compound from its residual filter list, while compileFilter silently returned None, so Pinot received no filter at all and returned unfiltered rows. Fix: isFilterSupported is now recursive — compounds are accepted only when every child is supported.

2. LIKE pushdown escaping + backslash rejection. StringStartsWith / StringEndsWith / StringContains interpolated raw user-supplied strings into LIKE '...' without escaping %, _, \, or '. Values containing these changed the predicate meaning (or produced invalid SQL). Fix: escapeLikeLiteral escapes LIKE wildcards, backslashes, and single-quotes; each LIKE predicate now carries an ESCAPE '\' clause. Additionally, values containing a literal \ are rejected upstream (isFilterSupported returns false) and fall through to Spark's post-scan filter evaluation because Pinot's runtime RegexpPatternConverterUtils#likeToRegexpLike does not round-trip \\ correctly.

3. SupportsOverwrite[V2] silently dropped the overwrite intent. The writer advertised V1 SupportsOverwrite (Spark 3) and V2 SupportsOverwriteV2 (Spark 4) but ignored the filters/predicates and returned an append-only writer. Spark callers using df.write.mode("overwrite"), df.writeTo(...).overwrite(...), or SQL INSERT OVERWRITE believed matching rows would be replaced while Pinot only appended new segments. Fix: both overwrite(...) and truncate() now throw UnsupportedOperationException with a message pointing at the supported alternatives. The truncate() override is load-bearing: SupportsOverwrite[V2] extends SupportsTruncate, and Spark's V2Writes rule dispatches df.write.mode("overwrite") → truncate() (which defaulted to returning this → silent append) rather than overwrite([AlwaysTrue]). Both connectors now reject both paths.

4. PinotDataWriter leaked the per-partition temp segment build directory on every successful task. Files.createTempDirectory in generateSegment() was never deleted, so long-running executors accumulated uncompressed segment trees under the JVM tmpdir. Fix: track the temp dir on a module-private field; wrap commit() in a try/finally that deletes it; also delete from abort() and close() (idempotent via null-out).

5. PinotWrite.commit/abort routed writer commit messages through println instead of SLF4J. Fix: switch to LoggerFactory.getLogger(...) so messages reach the standard log collection. Also added an inline TODO documenting the separate gap that abort() does not yet delete leftover segment tars from already-succeeded tasks at savePath (tracked as a follow-up; the contract-level guard against silent overwrite is provided by the overwrite/truncate rejection above).

Regression tests for (1), (2), and (3) are added to both pinot-spark-3-connector and pinot-spark-4-connector.

Test plan

• ./mvnw -pl pinot-connectors/pinot-spark-4-connector,pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark-4 -am test on JDK 21 — 3 junit + 31 scalatest cases, all green.
• ./mvnw -pl pinot-connectors/pinot-spark-3-connector -am test on JDK 21 — Spark 3 still passes (31 scalatest).
• ./mvnw -pl pinot-integration-tests -am test-compile — integration tests still compile unchanged.
• spotless:apply checkstyle:check license:check apache-rat:check clean on all modified modules.
• Live end-to-end validation against apachepinot/pinot:latest + apache/spark:4.0.0 (+JDK 21): read 97,889-row baseballStats table via gRPC, write a 10-row DataFrame to a fresh OFFLINE table, push via controller REST, verify roundtrip through both the Pinot SQL broker and the connector's own read path.

Known coverage gaps (tracked for follow-up)

• End-to-end Spark 4 integration test — A SparkSegmentMetadataPushIntegrationTest equivalent for Spark 4 was added then dropped from this PR (commit c66b2171aa) because the Spark 4 runtime (Jakarta Servlet 5) conflicts with pinot-integration-tests' Spark 3 + javax stack in a shared Maven module. Restoring end-to-end integration coverage requires either a dedicated pinot-spark-4-integration-tests module or failsafe with a separate classpath realm. Noted in the pinot-batch-ingestion-spark-4 README.
• Push-runner unit tests — SparkSegmentMetadataPushJobRunner / SparkSegmentTarPushJobRunner / SparkSegmentUriPushJobRunner have no unit tests in either the Spark 3 or the Spark 4 batch-ingestion module today; only SparkSegmentGenerationJobRunnerTest exercises the in-process driver path. Adding coverage for the push runners is orthogonal to this PR and would benefit both modules.
• PinotWrite.abort(...) leftover tars at savePath — on partial task failure the writer leaves already-committed tars at the user-supplied savePath, so a retried job after the push step can produce duplicate segments. The contract-level guard against silent overwrite is provided by the overwrite/truncate rejection; this runtime-level guard is tracked separately. TODO comment added in both connectors' PinotWrite.scala.

Notes for reviewers / future follow-ups

• Commons-lang3 shading — currently users must prepend commons-lang3:3.20+ on spark.{driver,executor}.extraClassPath. A follow-up PR can relocate org.apache.commons.lang3.* in the connector's shade config to eliminate this step.
• Scala 2.12 — Apache Spark 4 is Scala 2.13 only. The modules have requireProperty scala.compat.version=2.13 enforcer rules so -Pscala-2.12 builds fail fast with a clear message.
• -Ppinot-fastdev — activation is JDK-based, so -Ppinot-fastdev on JDK 21 also builds the Spark 4 modules. Intentional; the Spark 4 shade is disabled under -Ppinot-fastdev via each module's own profile. Explicit opt-out: -P!pinot-spark-4-connector / -P!pinot-batch-ingestion-spark-4.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.38%. Comparing base (4499bf5) to head (9a6d7a5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18261      +/-   ##
============================================
- Coverage     63.38%   63.38%   -0.01%     
+ Complexity     1668     1633      -35     
============================================
  Files          3252     3252              
  Lines        198661   198661              
  Branches      30770    30770              
============================================
- Hits         125925   125922       -3     
- Misses        62666    62667       +1     
- Partials      10070    10072       +2     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	63.38% <ø> (-0.01%)	⬇️
temurin	63.38% <ø> (-0.01%)	⬇️
unittests	63.38% <ø> (-0.01%)	⬇️
unittests1	55.34% <ø> (+<0.01%)	⬆️
unittests2	34.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Found a few high-signal correctness issues; see inline comments.

[xiangfu0]

Marking compound filters as supported without recursively checking their children can silently drop predicates. For example, an Or/And/Not containing an unsupported child is removed from Spark residual filtering, then compileFilter returns None, so Pinot can return rows that do not satisfy the original Spark filter. Recursively accept compounds only when every child compiles; otherwise return the original filter as residual.

[xiangfu0]

The LIKE pushdowns interpolate the raw Spark value while equality predicates escape string literals. Values containing quotes, percent, or underscore change the Pinot SQL predicate semantics, which can produce silent wrong results. Escape SQL literals and LIKE wildcards for StringStartsWith/StringEndsWith/StringContains before pushing these filters.

[xiangfu0]

This advertises SupportsOverwriteV2 but drops the overwrite predicates and builds an unconditional writer. Spark overwrite-by-filter callers will believe Pinot replaced the matching data, while this path only writes new segment files, which can leave old matching segments in place and produce duplicate or stale query results. Either implement predicate-aware replacement semantics or reject overwrite predicates explicitly.