Skip to content

Commit 93d3951

Browse files
committed
Add service.name and service.instance.id to PostgreSQL metrics
Add a dedicated pipeline for PostgreSQL metrics with a resource processor that sets service.name and service.instance.id. This ensures Prometheus generates target_info for PostgreSQL metrics, enabling the info() function to work correctly. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
1 parent c9fa3a9 commit 93d3951

6 files changed

Lines changed: 214 additions & 86 deletions

File tree

CLAUDE.md

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
This is the **OpenTelemetry Astronomy Shop Demo** - a polyglot microservices e-commerce application showcasing OpenTelemetry instrumentation across multiple programming languages. It serves as a realistic example for demonstrating distributed tracing, metrics, and logging.
8+
9+
## Common Commands
10+
11+
### Running the Demo
12+
```bash
13+
make start # Start all services (http://localhost:8080)
14+
make start-minimal # Start minimal set of services
15+
make stop # Stop all services
16+
```
17+
18+
### Building
19+
```bash
20+
make build # Build all Docker images
21+
make redeploy service=<name> # Rebuild and restart a single service
22+
```
23+
24+
### Testing
25+
```bash
26+
make run-tests # Run all tests (frontend + trace-based)
27+
make run-tracetesting # Run trace-based tests only
28+
make run-tracetesting SERVICES_TO_TEST="ad payment" # Test specific services
29+
```
30+
31+
### Linting & Validation
32+
```bash
33+
make check # Run all checks (misspell, markdownlint, license, links)
34+
make misspell # Check spelling in markdown files
35+
make markdownlint # Lint markdown files
36+
make checklicense # Check license headers
37+
```
38+
39+
### Protobuf Generation
40+
```bash
41+
make generate-protobuf # Generate protobuf code (requires local tools)
42+
make docker-generate-protobuf # Generate protobuf code via Docker
43+
make clean # Remove generated protobuf files
44+
```
45+
46+
## Architecture
47+
48+
### Service Communication
49+
- **gRPC**: Primary protocol for inter-service communication (defined in `pb/demo.proto`)
50+
- **HTTP/REST**: Used by frontend, email service, and external-facing endpoints
51+
- **Kafka**: Async messaging for checkout -> accounting/fraud-detection flow
52+
- **Envoy**: Frontend proxy handling routing to all services
53+
54+
### Microservices by Language
55+
56+
| Language | Services |
57+
|----------|----------|
58+
| **Go** | checkout, product-catalog, shipping |
59+
| **Java** | ad, fraud-detection (with OTel Java agent) |
60+
| **.NET/C#** | accounting, cart |
61+
| **Python** | recommendation, product-reviews, load-generator |
62+
| **TypeScript/Node.js** | frontend (Next.js), payment |
63+
| **Ruby** | email |
64+
| **PHP** | quote |
65+
| **C++** | currency |
66+
| **Rust** | shipping |
67+
| **Elixir** | flagd-ui |
68+
69+
### Key Infrastructure Components
70+
- **OpenTelemetry Collector**: Central telemetry pipeline (`src/otel-collector/`)
71+
- **Jaeger**: Distributed tracing backend (http://localhost:8080/jaeger/ui)
72+
- **Grafana**: Dashboards and visualization (http://localhost:8080/grafana)
73+
- **Prometheus**: Metrics storage
74+
- **Flagd**: Feature flags service (`src/flagd/demo.flagd.json`)
75+
- **Kafka**: Event streaming for order processing
76+
- **Valkey**: Cart session storage (Redis-compatible)
77+
- **PostgreSQL**: Persistent storage for accounting
78+
79+
### Directory Structure
80+
```
81+
src/
82+
├── <service>/ # Each microservice has its own directory
83+
│ ├── Dockerfile # Build definition
84+
│ └── README.md # Service-specific documentation
85+
pb/
86+
└── demo.proto # Shared protobuf definitions for gRPC services
87+
test/
88+
└── tracetesting/ # Trace-based test definitions
89+
```
90+
91+
## Configuration
92+
93+
- **Environment variables**: Defined in `.env` (base) and `.env.override` (local customizations)
94+
- **Docker Compose**: Main orchestration in `docker-compose.yml`
95+
- **Feature flags**: Configured in `src/flagd/demo.flagd.json`
96+
97+
## Development Workflow
98+
99+
1. Make code changes to a service in `src/<service>/`
100+
2. Rebuild and restart only that service: `make redeploy service=<name>`
101+
3. View traces in Jaeger and logs via `docker logs <container_name>`
102+
4. For protobuf changes, update `pb/demo.proto` then run `make docker-generate-protobuf`
103+
104+
## PromQL Conventions
105+
106+
### Prefer `info()` over Resource Attribute Promotion
107+
108+
When writing PromQL queries that need to filter or group by OpenTelemetry resource attributes (e.g., `service_name`, `deployment_environment_name`, `k8s_cluster_name`), prefer using the experimental `info()` function over resource attribute promotion in the collector.
109+
110+
**Pattern:**
111+
```promql
112+
# Preferred: Use info() with data-label-selector
113+
sum by (service_name) (
114+
info(rate(http_server_request_duration_seconds_count[$__rate_interval]),
115+
{deployment_environment_name=~"$env", service_name="$service"})
116+
)
117+
118+
# Avoid: Resource attributes promoted directly onto metrics
119+
sum by (service_name) (
120+
rate(http_server_request_duration_seconds_count{
121+
deployment_environment_name=~"$env",
122+
service_name="$service"
123+
}[$__rate_interval])
124+
)
125+
```
126+
127+
**Why:**
128+
- Reduces metric cardinality in Prometheus
129+
- Resource attributes are stored once in `target_info` rather than on every metric
130+
- The `info()` function joins metrics with `target_info` at query time
131+
132+
**Note:** Requires Prometheus with `--enable-feature=promql-experimental-functions`.
133+
134+
## Forking Notes
135+
136+
This is a Grafana Labs fork of the upstream OpenTelemetry demo. The main branch is `grafana`.

docker-compose.yml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ services:
147147
- GOMEMLIMIT=16MiB
148148
- OTEL_EXPORTER_OTLP_ENDPOINT
149149
- OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
150-
- OTEL_RESOURCE_ATTRIBUTES
150+
- OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=checkout
151151
- OTEL_SERVICE_NAME=checkout
152152
depends_on:
153153
cart:
@@ -500,7 +500,7 @@ services:
500500
- GOMEMLIMIT=16MiB
501501
- OTEL_EXPORTER_OTLP_ENDPOINT
502502
- OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
503-
- OTEL_RESOURCE_ATTRIBUTES
503+
- OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=product-catalog
504504
- OTEL_SERVICE_NAME=product-catalog
505505
- OTEL_SEMCONV_STABILITY_OPT_IN=database
506506
- DB_CONNECTION_STRING=postgres://otelu:otelp@${POSTGRES_HOST}/${POSTGRES_DB}?sslmode=disable
@@ -669,7 +669,7 @@ services:
669669
- FLAGD_OTEL_COLLECTOR_URI=${OTEL_COLLECTOR_HOST}:${OTEL_COLLECTOR_PORT_GRPC}
670670
- FLAGD_METRICS_EXPORTER=otel
671671
- GOMEMLIMIT=60MiB
672-
- OTEL_RESOURCE_ATTRIBUTES
672+
- OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=flagd
673673
- OTEL_SERVICE_NAME=flagd
674674
command: [
675675
"start",
@@ -907,6 +907,7 @@ services:
907907
- --web.route-prefix=/
908908
- --web.enable-otlp-receiver
909909
- --enable-feature=exemplar-storage
910+
- --enable-feature=promql-experimental-functions
910911
volumes:
911912
- ./src/prometheus/prometheus-config.yaml:/etc/prometheus/prometheus-config.yaml
912913
deploy:

0 commit comments

Comments
 (0)