Skip to content

Commit 7b85a5d

Browse files
authored
[ISSUE #10240] Add BatchSplittingMetricExporter to prevent OTLP gRPC export failures (#10239)
* Add BatchSplittingMetricExporter to prevent OTLP gRPC export failures When high-cardinality metrics (consumer_group x topic) produce OTLP export payloads exceeding the gRPC 32MB limit or SLS per-RPC processing limit, all metrics fail to export. This adds a MetricExporter decorator that: - Splits large batches of MetricData objects into smaller sub-batches - Splits single oversized MetricData objects by their internal data points into multiple smaller MetricData objects (supports all 7 MetricDataType) - Configurable via BrokerConfig.metricsExportBatchMaxDataPoints (default 1000) - Fast path with zero overhead when data points are within threshold - Logs failed batch details for debugging * fix(metrics): snapshot MetricData points before export to prevent AIOOBE The OTel SDK's NumberDataPointMarshaler.createRepeated allocates an array based on points.size() then iterates. If callback threads concurrently add data points between size() and iteration, an ArrayIndexOutOfBoundsException occurs. This adds a defensive snapshot of all data point collections at the start of export(), ensuring the delegate exporter always receives immutable point collections. * test(metrics): add unit tests for snapshot defensive copy - testSnapshotCreatesNewMetricData: verify delegate receives snapshotted MetricData, not the original reference - testSnapshotFallsBackToOriginal: verify catch block falls back to original when snapshot fails (e.g., mock without type) - testSnapshotPointsAreIndependentCopy: verify the snapshotted points collection is a separate instance from the original
1 parent 2c2cc92 commit 7b85a5d

File tree

4 files changed

+1266
-2
lines changed

4 files changed

+1266
-2
lines changed

0 commit comments

Comments
 (0)