[PostgreSQL] Extend the allowed Debezium properties#1247
[PostgreSQL] Extend the allowed Debezium properties#1247gayaldassanayake merged 19 commits intoballerina-platform:mainfrom
Conversation
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughRefactors CDC listener configuration into separate Debezium and listener maps, adds rich PostgreSQL-specific config types (replication, publication, streaming, extended snapshot, options), updates version/dependencies to 1.17.0 and cdc to 1.3.0, removes unused packages, expands tests and docs. Changes
Sequence Diagram(s)sequenceDiagram
participant App as CdcListener
participant Utils as utils.bal
participant Native as NativeBackend
participant Debez as DebeziumEngine
App->>Utils: populateDebeziumProperties(config)
App->>Utils: populateListenerProperties(config)
Utils-->>App: debeziumConfigs, listenerConfigs
App->>Native: externStartWithExtendedConfigs(debeziumConfigs, listenerConfigs)
Native->>Debez: initialize engine with debeziumConfigs
Native->>App: started / status events
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai please review |
|
✅ Actions performedReview triggered.
|
gayaldassanayake
requested review from
aashikam,
niveathika and
shafreenAnfar
as code owners
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
changelog.md (1)
13-15: Clarify the compatibility story for the filter-field move.This entry explains where the fields live now, but not whether the previous placement is still accepted. A one-line migration note would make upgrades much safer.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@changelog.md` around lines 13 - 15, Add a one-line migration note clarifying compatibility after the bullet about PostgresDatabaseConnection: state whether the filter fields (includedTables, excludedTables, includedColumns, excludedColumns) previously accepted on cdc:DatabaseConnection are still supported (e.g., deprecated aliasing/forwarding) or must be moved to PostgresDatabaseConnection, and give the recommended upgrade action (move fields to PostgresDatabaseConnection or leave as-is) so users know if they need to update configs; reference the symbol names PostgresDatabaseConnection and cdc:DatabaseConnection and the four field names in the sentence.ballerina/README.md (1)
520-522: Add one raw Debezium-property example here.The paragraph says extra fields can be passed through, but a tiny snippet would remove ambiguity about where those keys belong in the listener config.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ballerina/README.md` around lines 520 - 522, Add a minimal inline example showing how to pass a raw Debezium property into the listener config under the options parameter: demonstrate using the options parameter (type PostgresSqlOptions) with an extra field like "snapshot.select.statement.overrides" (or another real Debezium key) so readers see that raw keys are placed directly on options in the listener config; keep it one small config snippet and label it as a raw Debezium property example for clarity.ballerina/tests/listener_tests.bal (1)
270-318: Add one precedence test for the deprecated top-level fields.The implementation in
ballerina/utils.balexplicitly prefersreplicationConfigandpublicationConfigoverpluginName,slotName, andpublicationName, but the new suite only exercises nested-only inputs. A mixed legacy+nested case would lock the migration behavior down.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ballerina/tests/listener_tests.bal` around lines 270 - 318, Add a precedence test that mixes deprecated top-level fields with the new nested configs to assert nested configs win; create a test (e.g., testPostgresReplicationPrecedence / testPostgresPublicationPrecedence) that constructs a PostgresDatabaseConnection containing both top-level fields pluginName/slotName/publicationName (and/or slotDropOnStop/publicationAutocreateMode) set to one value and replicationConfig/publicationConfig set to a different value, call populateDatabaseConfigurations(connection, actualProperties), and assert actualProperties["plugin.name"], ["slot.drop.on.stop"], and ["publication.name"] / ["publication.autocreate.mode"] equal the values from replicationConfig/publicationConfig (not the deprecated top-level ones).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@ballerina/tests/listener_tests.bal`:
- Around line 263-291: The test seeds expectedProperties and then calls
populateDatabaseConfigurations(connection, actualProperties) but only asserts
two keys; add assertions for the remaining seeded properties so the mapping is
fully validated: assert actualProperties["slot.name"] ==
expectedProperties["slot.name"], actualProperties["slot.stream.params"] ==
expectedProperties["slot.stream.params"], and
actualProperties["xmin.fetch.interval.ms"] ==
expectedProperties["xmin.fetch.interval.ms"] (and any other seeded keys) using
test:assertEquals with explanatory msg strings; locate these in the same test
after the existing plugin.name and slot.drop.on.stop assertions and mirror their
style.
In `@ballerina/utils.bal`:
- Around line 122-128: The code writes LSN_FLUSH_MODE inside
populateStreamingConfiguration even though the bundled Debezium may not support
lsn.flush.mode; either upgrade the PostgreSQL CDC driver to a version that
bundles Debezium >= 3.4 or change populateStreamingConfiguration to avoid
writing LSN_FLUSH_MODE when the runtime driver lacks support: detect support (or
add a feature flag/validation) and if unsupported either skip setting
LSN_FLUSH_MODE and log/return a clear validation error when config.lsnFlushMode
is set; update the logic around populateStreamingConfiguration, LSN_FLUSH_MODE,
and config.lsnFlushMode accordingly so users don’t get a runtime error.
In `@gradle.properties`:
- Around line 61-63: Update the property stdlibPostgresqlDriverVersion in
gradle.properties to match the pinned postgresql.driver version in
ballerina/Dependencies.toml (change 1.6.1 → 1.6.2), so the value consumed by
build.gradle (the property stdlibPostgresqlDriverVersion) and the package
metadata (postgresql.driver in Dependencies.toml) remain consistent.
---
Nitpick comments:
In `@ballerina/README.md`:
- Around line 520-522: Add a minimal inline example showing how to pass a raw
Debezium property into the listener config under the options parameter:
demonstrate using the options parameter (type PostgresSqlOptions) with an extra
field like "snapshot.select.statement.overrides" (or another real Debezium key)
so readers see that raw keys are placed directly on options in the listener
config; keep it one small config snippet and label it as a raw Debezium property
example for clarity.
In `@ballerina/tests/listener_tests.bal`:
- Around line 270-318: Add a precedence test that mixes deprecated top-level
fields with the new nested configs to assert nested configs win; create a test
(e.g., testPostgresReplicationPrecedence / testPostgresPublicationPrecedence)
that constructs a PostgresDatabaseConnection containing both top-level fields
pluginName/slotName/publicationName (and/or
slotDropOnStop/publicationAutocreateMode) set to one value and
replicationConfig/publicationConfig set to a different value, call
populateDatabaseConfigurations(connection, actualProperties), and assert
actualProperties["plugin.name"], ["slot.drop.on.stop"], and ["publication.name"]
/ ["publication.autocreate.mode"] equal the values from
replicationConfig/publicationConfig (not the deprecated top-level ones).
In `@changelog.md`:
- Around line 13-15: Add a one-line migration note clarifying compatibility
after the bullet about PostgresDatabaseConnection: state whether the filter
fields (includedTables, excludedTables, includedColumns, excludedColumns)
previously accepted on cdc:DatabaseConnection are still supported (e.g.,
deprecated aliasing/forwarding) or must be moved to PostgresDatabaseConnection,
and give the recommended upgrade action (move fields to
PostgresDatabaseConnection or leave as-is) so users know if they need to update
configs; reference the symbol names PostgresDatabaseConnection and
cdc:DatabaseConnection and the four field names in the sentence.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: fd423c2c-69c6-40f9-9117-0caf4dafe545
📒 Files selected for processing (12)
ballerina/Ballerina.tomlballerina/CompilerPlugin.tomlballerina/Dependencies.tomlballerina/README.mdballerina/cdc_listener.balballerina/listener_types.balballerina/tests/listener_tests.balballerina/tests/test_cleanup.balballerina/utils.balbuild.gradlechangelog.mdgradle.properties
💤 Files with no reviewable changes (1)
- build.gradle
gayaldassanayake
force-pushed
the
all-fields
branch
from
f7a79a3 to
d998d6f
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (3)
gradle.properties (1)
61-62:⚠️ Potential issue | 🟠 MajorKeep the Gradle PostgreSQL driver pin aligned with the lockfile.
Line 61 still pins
stdlibPostgresqlDriverVersionto1.6.1, butballerina/Dependencies.tomlnow locksballerinax/postgresql.driverto1.6.2. That can leave Gradle resolving a different driver than the package metadata.🔧 Minimal fix
-stdlibPostgresqlDriverVersion=1.6.1 +stdlibPostgresqlDriverVersion=1.6.2🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@gradle.properties` around lines 61 - 62, The gradle property stdlibPostgresqlDriverVersion is pinned to 1.6.1 while Dependencies.toml locks ballerinax/postgresql.driver to 1.6.2; update the stdlibPostgresqlDriverVersion value to 1.6.2 to match the lockfile so Gradle won't resolve a mismatched driver, and re-run dependency resolution/verification to ensure versions are aligned.ballerina/tests/listener_tests.bal (1)
261-291:⚠️ Potential issue | 🟡 MinorAssert every property you seed in these mapping tests.
These cases populate
slot.name,slot.stream.params, andxmin.fetch.interval.ms, but never assert them. That leaves part of the new mapping surface unverified.🧪 Missing assertions
test:assertEquals(actualProperties["plugin.name"], expectedProperties["plugin.name"], msg = "Plugin name does not match."); + test:assertEquals(actualProperties["slot.name"], + expectedProperties["slot.name"], + msg = "Slot name does not match."); test:assertEquals(actualProperties["slot.drop.on.stop"], expectedProperties["slot.drop.on.stop"], msg = "Slot drop on stop does not match."); + test:assertEquals(actualProperties["slot.stream.params"], + expectedProperties["slot.stream.params"], + msg = "Slot stream params does not match."); @@ test:assertEquals(actualProperties["status.update.interval.ms"], expectedProperties["status.update.interval.ms"], msg = "Status update interval does not match."); + test:assertEquals(actualProperties["xmin.fetch.interval.ms"], + expectedProperties["xmin.fetch.interval.ms"], + msg = "XMIN fetch interval does not match."); test:assertEquals(actualProperties["lsn.flush.mode"], expectedProperties["lsn.flush.mode"], msg = "LSN flush mode does not match.");Also applies to: 321-349
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ballerina/tests/listener_tests.bal` around lines 261 - 291, The test seed seeds additional replication properties but only asserts two of them; update the testPostgresReplicationConfiguration (and the similar test at lines 321-349) to assert all seeded keys by adding assertions that compare actualProperties["slot.name"], actualProperties["slot.stream.params"], and actualProperties["xmin.fetch.interval.ms"] against expectedProperties entries; locate the assertions around populateDatabaseConfigurations and add test:assertEquals calls for those three keys to ensure populateDatabaseConfigurations produces the full mapping.ballerina/utils.bal (1)
122-128:⚠️ Potential issue | 🟠 MajorDon't emit
lsn.flush.modeagainst the current CDC driver.Line 127 writes
lsn.flush.mode, but this PR still depends onballerinax/postgresql.cdc.driver:1.0.0, which Ballerina Central documents as bundling Debezium Postgres Driver3.0.8.Final. Debezium introducedlsn.flush.modein 3.4, sostreamingConfig.lsnFlushModeis ahead of the bundled runtime here. Please gate or reject this field until the driver is upgraded in the same PR. (central.ballerina.io)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ballerina/utils.bal` around lines 122 - 128, The code is emitting LSN_FLUSH_MODE even though the bundled Debezium driver (ballerinax/postgresql.cdc.driver:1.0.0) is older and doesn't support lsn.flush.mode; update populateStreamingConfiguration to not set debeziumConfigs[LSN_FLUSH_MODE] from StreamingConfiguration.lsnFlushMode unless the runtime/driver version supports it (or reject the field); specifically, gate the assignment that checks lsnFlushMode (the lsnFlushMode variable and LSN_FLUSH_MODE constant) behind a runtime/compile check or remove/ignore lsnFlushMode until the driver dependency is bumped, so populateStreamingConfiguration does not emit the unsupported config key.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@ballerina/tests/listener_tests.bal`:
- Around line 261-291: The test seed seeds additional replication properties but
only asserts two of them; update the testPostgresReplicationConfiguration (and
the similar test at lines 321-349) to assert all seeded keys by adding
assertions that compare actualProperties["slot.name"],
actualProperties["slot.stream.params"], and
actualProperties["xmin.fetch.interval.ms"] against expectedProperties entries;
locate the assertions around populateDatabaseConfigurations and add
test:assertEquals calls for those three keys to ensure
populateDatabaseConfigurations produces the full mapping.
In `@ballerina/utils.bal`:
- Around line 122-128: The code is emitting LSN_FLUSH_MODE even though the
bundled Debezium driver (ballerinax/postgresql.cdc.driver:1.0.0) is older and
doesn't support lsn.flush.mode; update populateStreamingConfiguration to not set
debeziumConfigs[LSN_FLUSH_MODE] from StreamingConfiguration.lsnFlushMode unless
the runtime/driver version supports it (or reject the field); specifically, gate
the assignment that checks lsnFlushMode (the lsnFlushMode variable and
LSN_FLUSH_MODE constant) behind a runtime/compile check or remove/ignore
lsnFlushMode until the driver dependency is bumped, so
populateStreamingConfiguration does not emit the unsupported config key.
In `@gradle.properties`:
- Around line 61-62: The gradle property stdlibPostgresqlDriverVersion is pinned
to 1.6.1 while Dependencies.toml locks ballerinax/postgresql.driver to 1.6.2;
update the stdlibPostgresqlDriverVersion value to 1.6.2 to match the lockfile so
Gradle won't resolve a mismatched driver, and re-run dependency
resolution/verification to ensure versions are aligned.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bd83eb45-5c46-4184-b71e-b6fc5f2cfccd
📒 Files selected for processing (12)
ballerina/Ballerina.tomlballerina/CompilerPlugin.tomlballerina/Dependencies.tomlballerina/README.mdballerina/cdc_listener.balballerina/listener_types.balballerina/tests/listener_tests.balballerina/tests/test_cleanup.balballerina/utils.balbuild.gradlechangelog.mdgradle.properties
💤 Files with no reviewable changes (1)
- build.gradle
🚧 Files skipped from review as they are similar to previous changes (4)
- changelog.md
- ballerina/Ballerina.toml
- ballerina/README.md
- ballerina/CompilerPlugin.toml
gayaldassanayake
force-pushed
the
all-fields
branch
from
d998d6f to
feb09e8
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@ballerina/listener_types.bal`:
- Around line 92-94: The PostgresDatabaseConnection public record's doc block is
missing documentation for the new streamingConfig field; update the
PostgresDatabaseConnection documentation to add a description for
streamingConfig (what it configures, expected type/shape, and any
relation/priority with replicationConfig/publicationConfig), ensuring the doc
mentions the streamingConfig field name and its purpose so the generated API
docs include this new entry point.
- Around line 48-62: The code accepts both legacy fields and new nested records
(ReplicationConfiguration and PublicationConfiguration) but currently overrides
legacy values when nested records are present; update the logic around
replicationConfig and publicationConfig (in the code that currently switches
wholesale) to either reject mixed usage or merge field-by-field: detect if both
legacy fields and the nested record are provided and return an error, or iterate
each field in ReplicationConfiguration and PublicationConfiguration and copy
only non-nil/non-default values into the final config so legacy values are
preserved when nested records omit fields (reference ReplicationConfiguration,
PublicationConfiguration, replicationConfig, publicationConfig, and the
functions that perform the conversion).
- Around line 35-40: The public enum LsnFlushMode exposes a value
CONNECTOR_AND_DRIVER that requires Debezium 3.4+ but the bundled CDC driver
(ballerinax/postgresql.cdc.driver:1.0.0) contains Debezium 3.0.8 and will fail
at runtime; fix by either removing the CONNECTOR_AND_DRIVER enum member from
LsnFlushMode (and any other references to it) so the public API only exposes
supported values (MANUAL, CONNECTOR) and update any docs/tests that reference
it, or instead update the CDC driver dependency to a version that bundles
Debezium 3.4+ and adjust build/dependency metadata accordingly; target symbols:
enum LsnFlushMode and the CONNECTOR_AND_DRIVER variant (also remove/update the
duplicate occurrence referenced at the later LsnFlushMode definition).
- Around line 28-33: Add the missing enum member to PublicationAutocreateMode so
callers can use the Debezium-supported mode "no_tables": add a new enum constant
named NO_TABLES with the string value "no_tables" to the public enum
PublicationAutocreateMode (alongside ALL_TABLES, DISABLED, FILTERED) so the
public API exposes that valid mode.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b9e8ed6a-2125-499f-b91a-19672d353022
📒 Files selected for processing (12)
ballerina/Ballerina.tomlballerina/CompilerPlugin.tomlballerina/Dependencies.tomlballerina/README.mdballerina/cdc_listener.balballerina/listener_types.balballerina/tests/listener_tests.balballerina/tests/test_cleanup.balballerina/utils.balbuild.gradlechangelog.mdgradle.properties
💤 Files with no reviewable changes (1)
- build.gradle
✅ Files skipped from review due to trivial changes (4)
- changelog.md
- ballerina/Ballerina.toml
- gradle.properties
- ballerina/CompilerPlugin.toml
🚧 Files skipped from review as they are similar to previous changes (6)
- ballerina/tests/test_cleanup.bal
- ballerina/cdc_listener.bal
- ballerina/utils.bal
- ballerina/README.md
- ballerina/tests/listener_tests.bal
- ballerina/Dependencies.toml
| # Represents publication autocreate modes. | ||
| public enum PublicationAutocreateMode { | ||
| ALL_TABLES = "all_tables", | ||
| DISABLED = "disabled", | ||
| FILTERED = "filtered" | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
curl -fsSL https://debezium.io/documentation/reference/stable/connectors/postgresql.html |
sed -n '/publication.autocreate.mode/,/replica.identity.autoset.values/p' |
rg -n '`(all_tables|disabled|filtered|no_tables)`'Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Check the actual file to see current enum members
cat -n ballerina/listener_types.bal | head -40Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 1719
🏁 Script executed:
# Search for dependency declarations (look for build files, gradle, maven, etc.)
fd -e 'gradle|xml|toml|yaml|yml|lock' -type f | head -20Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 261
🏁 Script executed:
# Try alternative approach to check Debezium docs - simpler curl without piping to external tools
curl -fsSL 'https://debezium.io/documentation/reference/stable/connectors/postgresql.html' 2>&1 | grep -i 'publication.autocreate.mode' | head -5Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 1680
🏁 Script executed:
# Search for dependency/version files in the repository
fd -type f \( -name "*.gradle" -o -name "pom.xml" -o -name "Ballerina.toml" -o -name "*.lock" \)Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 261
🌐 Web query:
Debezium PostgreSQL 3.0.8 publication.autocreate.mode no_tables
💡 Result:
In Debezium PostgreSQL connector version 3.0.8, the publication.autocreate.mode configuration property controls how the connector handles creation of the PostgreSQL publication used for logical replication with the pgoutput plugin. The no_tables mode creates (or uses an existing) publication with no tables assigned to it initially. The user must then manually add tables to the publication using SQL commands like ALTER PUBLICATION mypub ADD TABLE myschema.mytable;. This is useful when the connector lacks privileges to auto-add tables or when fine-grained control over publication tables is needed. This option was added via DBZ-8156 and documented in a merged pull request in August 2024, present in Debezium 3.0.8 (released March 2025). Other common modes include: - disabled: No auto-creation; publication must pre-exist. - all_tables: Creates publication for all tables (default in older versions). - filtered: Creates publication based on table.include.list/table.exclude.list filters. Configure as: "publication.autocreate.mode": "no_tables"
Citations:
- 1: https://docs.redhat.com/en/documentation/red_hat_build_of_debezium/3.0.8/pdf/debezium_user_guide/Red_Hat_build_of_Debezium-3.0.8-Debezium_User_Guide-en-US.pdf
- 2: DBZ-8156 Add docs for
no_tablesconfig option for `publication.auto… debezium/debezium#5780 - 3: https://issues.redhat.com/browse/DBZ-1494
- 4: https://debezium.io/releases/3.0/release-notes
🏁 Script executed:
# Find dependency files using simpler approach
find . -maxdepth 3 -type f \( -name "Ballerina.toml" -o -name "gradle.build" -o -name "pom.xml" -o -name "*.gradle" \) 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 688
🏁 Script executed:
# Search for "debezium" references in the repository
rg -i "debezium|publication.autocreate" --type-list | head -5 && rg -i "debezium|publication" -A 2 -B 2 | head -40Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 2881
🏁 Script executed:
# Check main build.gradle for dependencies
cat build.gradle | grep -i debezium -A 2 -B 2Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Check native build.gradle
cat native/build.gradle | grep -i debezium -A 2 -B 2Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Read the native/build.gradle file to see dependencies
cat native/build.gradle | head -80Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 2803
🏁 Script executed:
# Broader search for debezium anywhere in the repo
rg 'debezium' -i --max-count 20Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 3615
🏁 Script executed:
# Check gradle.properties for version variables
cat gradle.properties | grep -i 'cdc\|debezium' -A 1 -B 1Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 186
🏁 Script executed:
# Check ballerina/Ballerina.toml for dependencies
cat ballerina/Ballerina.toml | head -40Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 830
🏁 Script executed:
# Check what version of cdc driver is being used and search for its Debezium version
# Look at settings.gradle to understand dependency resolution
cat settings.gradle | grep -i cdc -A 3 -B 3Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🌐 Web query:
ballerinax postgresql.cdc.driver 1.0.0 Debezium version
💡 Result:
The ballerinax/postgresql.cdc.driver version 1.0.0 bundles Debezium Postgres Driver version 3.0.8.Final.
Citations:
- 1: https://github.com/ballerina-platform/module-ballerinax-postgresql.cdc.driver
- 2: https://central.ballerina.io/ballerinax/postgresql.cdc.driver/latest
🏁 Script executed:
# Check if there are any other references to the actual Debezium version in this repo
rg 'debezium.*3\.0' -iRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
Add the missing NO_TABLES enum member to PublicationAutocreateMode.
The currently published ballerinax/postgresql.cdc.driver 1.0.0 bundles Debezium Postgres Driver 3.0.8.Final, which supports publication.autocreate.mode=no_tables. Omitting it from the enum blocks users from configuring this valid mode through the public API.
Fix
public enum PublicationAutocreateMode {
ALL_TABLES = "all_tables",
DISABLED = "disabled",
- FILTERED = "filtered"
+ FILTERED = "filtered",
+ NO_TABLES = "no_tables"
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Represents publication autocreate modes. | |
| public enum PublicationAutocreateMode { | |
| ALL_TABLES = "all_tables", | |
| DISABLED = "disabled", | |
| FILTERED = "filtered" | |
| } | |
| # Represents publication autocreate modes. | |
| public enum PublicationAutocreateMode { | |
| ALL_TABLES = "all_tables", | |
| DISABLED = "disabled", | |
| FILTERED = "filtered", | |
| NO_TABLES = "no_tables" | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ballerina/listener_types.bal` around lines 28 - 33, Add the missing enum
member to PublicationAutocreateMode so callers can use the Debezium-supported
mode "no_tables": add a new enum constant named NO_TABLES with the string value
"no_tables" to the public enum PublicationAutocreateMode (alongside ALL_TABLES,
DISABLED, FILTERED) so the public API exposes that valid mode.
| # Represents LSN flush modes. | ||
| public enum LsnFlushMode { | ||
| MANUAL = "manual", | ||
| CONNECTOR = "connector", | ||
| CONNECTOR_AND_DRIVER = "connector_and_driver" | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Candidate manifest files =="
fd '^(Ballerina\.toml|Dependencies\.toml|Package\.md|README\.md)$' -x sh -c '
printf "\n--- %s ---\n" "$1"
sed -n "1,220p" "$1"
' sh {}
echo
echo "== Debezium / CDC driver references =="
rg -n 'postgresql\.cdc\.driver|debezium-connector-postgres|Debezium Postgres Driver|lsn\.flush\.mode|connector_and_driver' .Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 29179
🏁 Script executed:
# Check the full postgresql.cdc.driver version in Dependencies.toml
cat ballerina/Dependencies.toml | grep -A 10 'name = "postgresql.cdc.driver"'Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 583
🏁 Script executed:
# Check build.gradle for CDC driver version
grep -n 'stdlibPostgresCdcDriverVersion\|postgresql.*cdc.*driver' build.gradle | head -20Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 204
🏁 Script executed:
# Check if there's a gradle.properties or similar with version definitions
fd '(gradle\.properties|version\.txt|versions\.gradle)$' -x cat {}Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 1257
🏁 Script executed:
# Search for any Debezium version references in the native build or pom files
fd '(pom\.xml|build\.gradle)' -x grep -l 'debezium\|cdc.*driver' {} 2>/dev/null | head -5Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 105
🏁 Script executed:
# Find and examine native build files for Debezium/CDC driver dependencies
find ./native -name 'pom.xml' -o -name 'build.gradle' 2>/dev/null | head -5Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 112
🏁 Script executed:
# Check for any native module files that might reference Debezium version
fd 'pom\.xml' ./native -x grep -l 'debezium' {} 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Search for any version or release documentation
fd '(CHANGELOG|RELEASE|NEWS|VERSION)' -i | grep -i 'cdc\|debezium' | head -10Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Check native build gradle files for CDC driver or Debezium references
find ./native -name '*.gradle' -x grep -n 'debezium\|cdc.*driver\|postgres.*connector' {} 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
# Examine native/build.gradle for CDC driver dependencies
cat ./native/build.gradle | grep -A 5 -B 5 'cdc\|debezium\|postgres' -iRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 934
🏁 Script executed:
# Check if there's a separate CDC driver repository or if it's defined in build
rg 'postgresql\.cdc\.driver|Debezium' ballerina/Dependencies.toml -A 5 -B 2Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 705
🏁 Script executed:
# Search for any gradle property or variable that tracks CDC driver bundled Debezium version
rg 'debezium.*version|cdc.*version' -i gradle.properties build.gradleRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 400
🏁 Script executed:
# Check if this PR changes any CDC driver version by looking for gradle version assignments
grep -r 'stdlibPostgresCdcDriverVersion\|postgresql.cdc.driver' gradle.properties build.gradle native/build.gradle 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 268
🏁 Script executed:
# Search for any Debezium or version configuration in tests
rg 'debezium|3\.0\.|3\.4\.' -i ballerina/tests listener_tests.bal | head -20Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 149
🏁 Script executed:
# Check README or documentation for Debezium version requirements
cat ballerina/README.md | grep -A 5 -B 5 'debezium\|cdc.*driver\|3\.0\|3\.4' -iRepository: ballerina-platform/module-ballerinax-postgresql
Length of output: 999
🏁 Script executed:
# Search for any version constraints or compatibility notes
rg 'version.*3\.' ballerina/ -A 2 -B 2Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 336
🌐 Web query:
ballerinax/postgresql.cdc.driver version 1.0.0 Debezium
💡 Result:
ballerinax/postgresql.cdc.driver version 1.0.0 is a Ballerina library that provides the Debezium PostgreSQL connector dependencies for Change Data Capture (CDC) functionality in Ballerina applications. It enables seamless listening to changes (inserts, updates, deletes) in PostgreSQL databases. Key details: - Released on 2025-05-29 (latest and only release as of 2026-03-19). - Compatible with Ballerina Language version 2201.12.0. - Bundles Debezium Postgres Driver version 3.0.8.Final. - Usage: Import alongside ballerinax/cdc and ballerinax/postgresql modules: import ballerinax/cdc; import ballerinax/postgresql; import ballerinax/postgresql.cdc.driver as _; Then configure a postgresql:CdcListener to capture events with remote functions like onCreate, onUpdate, onDelete. It is a support module for Ballerina's CDC framework, which is built on Debezium. No additional driver configuration needed as it's pre-bundled. Available on Ballerina Central: https://central.ballerina.io/ballerinax/postgresql.cdc.driver/latest. Source: https://github.com/ballerina-platform/module-ballerinax-postgresql.cdc.driver.
Citations:
- 1: https://central.ballerina.io/ballerinax/postgresql.cdc.driver/latest
- 2: https://github.com/ballerina-platform/module-ballerinax-postgresql.cdc.driver
- 3: https://github.com/ballerina-platform/module-ballerinax-cdc
- 4: https://central.ballerina.io/ballerinax/cdc/latest
Remove CONNECTOR_AND_DRIVER from the public API or upgrade the bundled CDC driver to support Debezium 3.4+.
The enum value CONNECTOR_AND_DRIVER requires Debezium Postgres Driver 3.4 or later to support the lsn.flush.mode property. The bundled ballerinax/postgresql.cdc.driver:1.0.0 currently provides only Debezium 3.0.8.Final, which does not recognize this configuration value. Code using this enum will compile but fail at runtime when Debezium attempts to parse it. This PR does not upgrade the CDC driver version, so either the enum should be removed from the public API for now, or the CDC driver dependency must be upgraded to a version supporting Debezium 3.4+.
Also applies to: lines 69-72
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ballerina/listener_types.bal` around lines 35 - 40, The public enum
LsnFlushMode exposes a value CONNECTOR_AND_DRIVER that requires Debezium 3.4+
but the bundled CDC driver (ballerinax/postgresql.cdc.driver:1.0.0) contains
Debezium 3.0.8 and will fail at runtime; fix by either removing the
CONNECTOR_AND_DRIVER enum member from LsnFlushMode (and any other references to
it) so the public API only exposes supported values (MANUAL, CONNECTOR) and
update any docs/tests that reference it, or instead update the CDC driver
dependency to a version that bundles Debezium 3.4+ and adjust build/dependency
metadata accordingly; target symbols: enum LsnFlushMode and the
CONNECTOR_AND_DRIVER variant (also remove/update the duplicate occurrence
referenced at the later LsnFlushMode definition).
| public type ReplicationConfiguration record {| | ||
| PostgreSQLLogicalDecodingPlugin pluginName = PGOUTPUT; | ||
| string slotName = "debezium"; | ||
| boolean slotDropOnStop = false; | ||
| string slotStreamParams?; | ||
| |}; | ||
|
|
||
| # PostgreSQL publication configuration (pgoutput plugin). | ||
| # | ||
| # + publicationName - Name of PostgreSQL publication | ||
| # + publicationAutocreateMode - Mode for auto-creating publications | ||
| public type PublicationConfiguration record {| | ||
| string publicationName = "dbz_publication"; | ||
| PublicationAutocreateMode publicationAutocreateMode = ALL_TABLES; | ||
| |}; |
There was a problem hiding this comment.
Reject mixed deprecated and nested configs, or merge them field-by-field.
ballerina/utils.bal:74-100 switches wholesale to replicationConfig / publicationConfig when those records are present. Because these new records prefill pluginName, slotName, and publicationName, a partial migration such as {pluginName: DECODERBUFS, replicationConfig: {slotDropOnStop: true}} silently resolves to pgoutput / debezium instead of preserving the legacy values. That is a behavior change, not just a deprecation path.
Also applies to: 108-119
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ballerina/listener_types.bal` around lines 48 - 62, The code accepts both
legacy fields and new nested records (ReplicationConfiguration and
PublicationConfiguration) but currently overrides legacy values when nested
records are present; update the logic around replicationConfig and
publicationConfig (in the code that currently switches wholesale) to either
reject mixed usage or merge field-by-field: detect if both legacy fields and the
nested record are provided and return an error, or iterate each field in
ReplicationConfiguration and PublicationConfiguration and copy only
non-nil/non-default values into the final config so legacy values are preserved
when nested records omit fields (reference ReplicationConfiguration,
PublicationConfiguration, replicationConfig, publicationConfig, and the
functions that perform the conversion).
| # + replicationConfig - Replication configuration (logical decoding plugin, slot name and parameters). Takes priority over deprecated top-level fields | ||
| # + publicationConfig - Publication configuration (publication name and autocreate mode). Takes priority over deprecated top-level fields | ||
| public type PostgresDatabaseConnection record {| |
There was a problem hiding this comment.
Document streamingConfig in the public record docs.
Line 119 adds a new public field, but the PostgresDatabaseConnection doc block never lists or describes streamingConfig. That leaves the generated API docs incomplete for one of the new entry points.
Also applies to: 117-119
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@ballerina/listener_types.bal` around lines 92 - 94, The
PostgresDatabaseConnection public record's doc block is missing documentation
for the new streamingConfig field; update the PostgresDatabaseConnection
documentation to add a description for streamingConfig (what it configures,
expected type/shape, and any relation/priority with
replicationConfig/publicationConfig), ensuring the doc mentions the
streamingConfig field name and its purpose so the generated API docs include
this new entry point.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1247 +/- ##
============================================
+ Coverage 81.63% 81.77% +0.13%
+ Complexity 864 863 -1
============================================
Files 28 28
Lines 3523 3566 +43
Branches 484 494 +10
============================================
+ Hits 2876 2916 +40
- Misses 431 433 +2
- Partials 216 217 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 participants
Purpose
Issue : ballerina-platform/ballerina-library#8572
Examples
Checklist
PostgreSQL Debezium Properties Extension
Overview
This PR extends the PostgreSQL CDC module to support a broader set of Debezium configuration properties, enabling finer-grained control over replication, publication, streaming, and snapshot behavior. It also refactors configuration handling to separate Debezium engine properties from listener-specific options and updates module dependencies.
Key Changes
Dependency and Version Updates
Configuration Extensions
Architecture and Behavioral Changes
Utilities and Population Logic
Documentation and Tests
Impact