feat(observability): add baggage attribute mapping for span customization

[vishu-bh]

Problem

Issue #4700 requires selected tenant/user context carried through OpenTelemetry baggage to appear in exported spans without the baggage. prefix. Clients can send values such as tenant.id and user.id as OTEL baggage, but some observability backends expect those dimensions as span attributes named tenant.id / user.id, not baggage.tenant.id / baggage.user.id.

The gateway still needs to stay aligned with OTEL semantics: baggage remains the propagation mechanism, while span attributes are the reporting surface.

Solution

This PR replaces the earlier span-processor remapping approach with a baggage emission policy that is applied where baggage is copied onto spans.

The SpanAttributeCustomizer plugin now supports:

config:
  allowed_baggage_span_attributes:
    - "tenant.id"
    - "user.id"
  emit_baggage_prefixed_attributes: false

With that configuration:

• OTEL baggage continues to carry tenant.id and user.id
• only allowlisted baggage keys are promoted to span attributes
• exported span attributes are emitted as tenant.id / user.id
• baggage.tenant.id / baggage.user.id are not emitted for those allowlisted keys

If the new config is not provided, the gateway preserves legacy behavior and emits baggage as baggage.<key>.

Changes

• mcpgateway/observability.py

  • Added BaggageSpanAttributePolicy
  • Added startup extraction for the SpanAttributeCustomizer baggage policy
  • Applies the policy directly in both request-root span baggage injection and create_span()
  • Removed the old BaggageAttributeSpanProcessor remapping path

• mcpgateway/main.py

  • Loads the baggage span attribute policy after plugin config is available and before telemetry is initialized

• plugins/span_attribute_customizer/config_schema.py

  • Added allowed_baggage_span_attributes
  • Added emit_baggage_prefixed_attributes
  • Added validation/normalization for the baggage allowlist

• plugins/config.yaml

  • Added ready-to-use baggage policy config for tenant.id and user.id

• Docs

  • Updated OpenTelemetry architecture docs
  • Updated span attribute reference docs
  • Updated SpanAttributeCustomizer docs and README

• Tests

  • Added coverage for policy extraction, malformed config handling, allowlist behavior, default legacy behavior, and both baggage-to-span injection paths

Testing

Validated locally:

make pre-commit

Also verified focused observability coverage and diff-cover behavior locally after adding tests for the changed baggage paths.

Fixes #4700

[ja8zyjits]

Minor Issues

1. Missing Inline Comment ⚠️

Location: mcpgateway/main.py:1478-1480

Issue: Telemetry init reordering not explained.

Recommendation: Add comment:

# Extract span attribute mapping from plugin config before telemetry init
span_attribute_mapping = extract_span_attribute_mapping(get_plugin_manager_factory())
# Initialize telemetry with mapping (must run after plugin manager init)
init_telemetry(span_attribute_mapping=span_attribute_mapping)

Priority: Low (code is correct, just needs documentation)

2. Missing User Documentation ⚠️

Issue: No user-facing documentation for attribute_mapping configuration.

Missing:

1. Plugin configuration guide
2. Example in plugin README
3. Mention in observability documentation

Recommendation: Create follow-up issue for documentation.

Priority: Medium (feature works but users need guidance)

Optional Enhancements

3. Integration Test (Optional)

Suggestion: Add integration test with actual baggage propagation end-to-end.

Current Coverage: Unit tests cover processor logic thoroughly.

Value: Would verify full pipeline (baggage injection → processor → export).

Priority: Low (nice-to-have, not blocking)

Final Verdict

✅ APPROVE with Minor Recommendations

Strengths:

1. ✅ Correct architectural approach (SpanProcessor is right layer)
2. ✅ Defensive implementation (fail-closed, validates input)
3. ✅ Comprehensive test coverage (13 test cases, all edge cases)
4. ✅ Backward compatible (additive feature)
5. ✅ Addresses root cause (timing issue with baggage injection)
6. ✅ Safe and performant (minimal overhead)

Minor Issues:

1. ⚠️ Add inline comment explaining telemetry init ordering
2. ⚠️ Missing user-facing documentation (create follow-up issue)

[ja8zyjits]

Lgtm

[ja8zyjits]

testing needed.