feat(spark-4): add Apache Spark 4.x connector + batch-ingestion plugin (JDK 21+)#18261

Open

xiangfu0 wants to merge 30 commits intoapache:masterfrom

xiangfu0:claude/musing-neumann-2a8e62

Commits on Apr 26, 2026

feat: add Apache Spark 4.x support (JDK 21+)

xiangfu0
and
claude
committed
refactor: move Spark 4 modules alongside their Spark 3 siblings

xiangfu0
and
claude
committed
fix(ci): drop SparkSegmentMetadataPushIntegrationTest4 to unblock JDK 21 CI

xiangfu0
and
claude
committed
fixup(review): drop stray blank line + modernize Scala 2.13 imports

xiangfu0
and
claude
committed
docs(spark-4): note Spark-4 integration-test coverage gap in README

xiangfu0
and
claude
committed
docs(spark-4): add end-to-end Docker tutorial for read + write

xiangfu0
and
claude
committed
fix(spark-connector): address review comments + CI rat failure

xiangfu0
and
claude
committed
fix(spark-3-connector): apply symmetric overwrite-rejection fix

xiangfu0
and
claude
committed
docs(spark-connector): polish review minors

xiangfu0
and
claude
committed
fix(spark-connector): reject backslash LIKE pushdowns + slf4j logger in BatchWrite

xiangfu0
and
claude
committed
fix(spark-connector): plug CRITICAL truncate() bypass + clean up temp dirs

xiangfu0
and
claude
committed
fix(spark-connector): abort cleanup + temp-dir lifecycle tests

xiangfu0
and
claude
committed
fix(spark-connector): handle null entries in PinotWrite.abort() messages

xiangfu0
and
claude
committed
test(spark-connector): add PinotWrite.abort() runtime cleanup tests

xiangfu0
and
claude
committed
docs(spark-connector): polish review minors

xiangfu0
and
claude
committed
fix(spark-connector): tighten escapeLikeLiteral + DataWriter close semantics

xiangfu0
and
claude
committed
docs(spark-connector): document LIKE ESCAPE cross-module invariant

xiangfu0
and
claude
committed
fix(spark-4-connector): fail fast when both v3 and v4 jars share classpath

xiangfu0
and
claude
committed
fix(spark-4-connector): close classpath-guard race via lazy val

xiangfu0
and
claude
committed
fix(spark-4-connector): catch LinkageError in v3 classpath probe

xiangfu0
and
claude
committed
fix(spark-connector): close several MAJOR review gaps in filter pushdown + abort

xiangfu0
and
claude
committed
fix(spark-connector): close MAJOR review gaps in null-leaf gating + abort + classpath guard

xiangfu0
and
claude
committed
docs(spark-3-connector): correct PinotWriteBuilder type-hierarchy comment

xiangfu0
and
claude
committed
fix(spark-connector): closeOnce temp-dir cleanup runs even if reader.close() throws

xiangfu0
and
claude
committed
docs(spark-4-connector): note one-shot probe limitation in classpath guard

xiangfu0
and
claude
committed
docs(spark-3): deprecate pinot-spark-3-connector + batch-ingestion-spark-3

xiangfu0
and
claude
committed
fix(spark-connector): override canOverwrite to false + clean up LOGGER imports

xiangfu0
and
claude
committed
fix(spark-connector): reject pushdown for comparison filters with null literal

xiangfu0
and
claude
committed
docs(spark-connector): fix README link path, tutorial placeholder, null-leaf comment

xiangfu0
and
claude
committed
fix(spark-connector): array accessors, segment-name format, translator throw

xiangfu0
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(spark-4): add Apache Spark 4.x connector + batch-ingestion plugin (JDK 21+)#18261

feat(spark-4): add Apache Spark 4.x connector + batch-ingestion plugin (JDK 21+)#18261
xiangfu0 wants to merge 30 commits intoapache:masterfrom
xiangfu0:claude/musing-neumann-2a8e62

Commits on Apr 26, 2026

feat: add Apache Spark 4.x support (JDK 21+)

refactor: move Spark 4 modules alongside their Spark 3 siblings

fix(ci): drop SparkSegmentMetadataPushIntegrationTest4 to unblock JDK 21 CI

fixup(review): drop stray blank line + modernize Scala 2.13 imports

docs(spark-4): note Spark-4 integration-test coverage gap in README

docs(spark-4): add end-to-end Docker tutorial for read + write

fix(spark-connector): address review comments + CI rat failure

fix(spark-3-connector): apply symmetric overwrite-rejection fix

docs(spark-connector): polish review minors

fix(spark-connector): reject backslash LIKE pushdowns + slf4j logger in BatchWrite

fix(spark-connector): plug CRITICAL truncate() bypass + clean up temp dirs

fix(spark-connector): abort cleanup + temp-dir lifecycle tests

fix(spark-connector): handle null entries in PinotWrite.abort() messages

test(spark-connector): add PinotWrite.abort() runtime cleanup tests

docs(spark-connector): polish review minors

fix(spark-connector): tighten escapeLikeLiteral + DataWriter close semantics

docs(spark-connector): document LIKE ESCAPE cross-module invariant

fix(spark-4-connector): fail fast when both v3 and v4 jars share classpath

fix(spark-4-connector): close classpath-guard race via lazy val

fix(spark-4-connector): catch LinkageError in v3 classpath probe

fix(spark-connector): close several MAJOR review gaps in filter pushdown + abort

fix(spark-connector): close MAJOR review gaps in null-leaf gating + abort + classpath guard

docs(spark-3-connector): correct PinotWriteBuilder type-hierarchy comment

fix(spark-connector): closeOnce temp-dir cleanup runs even if reader.close() throws

docs(spark-4-connector): note one-shot probe limitation in classpath guard

docs(spark-3): deprecate pinot-spark-3-connector + batch-ingestion-spark-3

fix(spark-connector): override canOverwrite to false + clean up LOGGER imports

fix(spark-connector): reject pushdown for comparison filters with null literal

docs(spark-connector): fix README link path, tutorial placeholder, null-leaf comment

fix(spark-connector): array accessors, segment-name format, translator throw