[PLUGIN-1950] Log warning when a table has no records to read#73
Open
psainics wants to merge 2 commits intodata-integrations:developfrom
Open
[PLUGIN-1950] Log warning when a table has no records to read#73psainics wants to merge 2 commits intodata-integrations:developfrom
psainics wants to merge 2 commits intodata-integrations:developfrom
Conversation
When the multi-table source reads from tables that contain zero rows, the lack of output can be confusing to debug. This adds a WARN-level log line with the table name in both DBTableRecordReader and SQLStatementRecordReader so operators can quickly identify empty tables.
Contributor
Author
|
sahusanket
reviewed
Apr 6, 2026
| } | ||
| if (!results.next()) { | ||
| if (pos == 0) { | ||
| LOG.warn("Table '{}' had no records to read.", tableName.getTable()); |
There was a problem hiding this comment.
This is 100% safe only when there is 1 split.
Imagine a table with 2,000 records, where the primary key ID has a massive gap. The records exist from ID = 1 to 1000, and ID = 3000 to 4000.
If your job calculates splits in chunks of 1000 IDs, you might get the following splits:
Split 1 (ID 1 - 1000): Has 1000 records.
Split 2 (ID 1001 - 2000): Has 0 records (Empty split).
Split 3 (ID 2001 - 3000): Has 0 records (Empty split).
Split 4 (ID 3001 - 4000): Has 1000 records.
When the RecordReader for Split 2 runs, its query returns an empty ResultSet. Since it is the first read attempt for that split (pos == 0), it will log: Table 'MyTable' had no records to read. even though the table actually contains 2,000 records.
So there is this corner case.
I am not sure if we can solve at a split level unless we make sure the no of splits is 1. or run a SELECT 1 FROM somewhere at a higher level.
Contributor
Author
There was a problem hiding this comment.
Sql query is single split, for table i am logging for each split.
- DBTableRecordReader: log table name and split query so users can identify which table and range returned no records - SQLStatementRecordReader: log the SQL statement instead of the resolved table name for clarity
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Log warning when a table has no records to read
Jira : Plugin-1950
Description
When the multi-table source reads from tables that contain zero rows, the lack of output can be confusing to debug. This adds a WARN-level log line with the table name in both DBTableRecordReader and SQLStatementRecordReader so operators can quickly identify empty tables.
Test setup
table_r0table_r1table_r3, suffix represents the number of records in the tableCase 1) Custom SQL Statements
Case 2) Table Allow List / Table Block List
Case 3) Table Block List (skip empty table) (no empty table log expected)