Honor deletedSegmentsRetentionPeriod for REFRESH tables in segment lineage cleanup

[swaminathanmanish]

Problem

REFRESH (non-APPEND) offline tables have their superseded segments deleted via manageSegmentLineageCleanupForTable. This method called deleteSegments(tableNameWithType, segmentsToDelete) — the no-arg overload that ignores deletedSegmentsRetentionPeriod from the table config and always falls back to the cluster-level default (7 days).

manageRetentionForTable (which does read per-table retention config) exits early for REFRESH tables:

if (tableConfig.getTableType() == TableType.OFFLINE && !"APPEND".equalsIgnoreCase(segmentPushType)) {
    return; // REFRESH tables skip this entirely
}

So even if an operator sets deletedSegmentsRetentionPeriod: "0d" on a REFRESH table, the staging retention is silently ignored — segments always sit in Deleted_Segments/ for the full cluster default before permanent deletion.

Fix

Pass the table's deletedSegmentsRetentionPeriod to deleteSegments in manageSegmentLineageCleanupForTable:

// before
_pinotHelixResourceManager.deleteSegments(tableNameWithType, segmentsToDelete);

// after
String retentionPeriod = tableConfig.getValidationConfig().getDeletedSegmentsRetentionPeriod();
_pinotHelixResourceManager.deleteSegments(tableNameWithType, segmentsToDelete, retentionPeriod);

When deletedSegmentsRetentionPeriod is null (not configured), the existing deleteSegments(String, List, String) overload falls through to re-read the table config and then the cluster default — identical behavior to before.

Usage

To enable immediate deletion of superseded segments for a REFRESH table:

"segmentsConfig": {
  "deletedSegmentsRetentionPeriod": "0d"
}

Backward Compatibility

No behavior change for tables that do not set deletedSegmentsRetentionPeriod. APPEND and realtime tables are unaffected.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.47%. Comparing base (3eff094) to head (06cdb8e).

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18257      +/-   ##
============================================
- Coverage     63.48%   63.47%   -0.01%     
  Complexity     1627     1627              
============================================
  Files          3244     3244              
  Lines        197365   197366       +1     
  Branches      30540    30540              
============================================
- Hits         125306   125287      -19     
- Misses        62019    62037      +18     
- Partials      10040    10042       +2     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-11	63.46% <100.00%> (-0.01%)	⬇️
java-21	63.45% <100.00%> (-0.02%)	⬇️
temurin	63.47% <100.00%> (-0.01%)	⬇️
unittests	63.47% <100.00%> (-0.01%)	⬇️
unittests1	55.45% <ø> (-0.01%)	⬇️
unittests2	34.96% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.