Skip to content

FIX(REDIS): Raise maxclients from 10000 to 30000 to support 10 replicas#4724

Open
claudia-gray wants to merge 1 commit into
mainfrom
fix/4630-redis-maxclients-capacity
Open

FIX(REDIS): Raise maxclients from 10000 to 30000 to support 10 replicas#4724
claudia-gray wants to merge 1 commit into
mainfrom
fix/4630-redis-maxclients-capacity

Conversation

@claudia-gray
Copy link
Copy Markdown
Collaborator

🔗 Related Issue

Closes #4630


📝 Summary

Fixes redis "Too many connections" error when scaling gateway to 10 replicas.

Problem: The Redis server-side maxclients limit (10,000) is exceeded when scaling to 10 replicas with default worker and connection-pool settings, causing redis.exceptions.ConnectionError: Too many connections in the rate-limiter and session-registry.

Root Cause: Connection formula replicas × workers × REDIS_MAX_CONNECTIONS exceeds server limit:

  • 10 replicas × 24 workers × 100 pool = 24,000 connections > 10,000 maxclients

Solution: Raise Redis maxclients from 10,000 to 30,000 to support 10 replicas with 20% headroom:
10 replicas x 24 workers x 100 pool = 24,000 connections < 30000 maxclients.

This addresses the root cause as server-side capacity, rather than degrading client-side performance by reducing connection pools. Even if pool was scaled down to 50 from 100, 10replicas x 24 workers x 50 pool = 12,000 > 10,000 (still greater than 10,000)


🏷️ Type of Change

  • Bug fix
  • Feature / Enhancement
  • Documentation
  • Refactor
  • Chore (deps, CI, tooling)
  • Other (describe below)

🧪 Verification

Check Command Status
Redis maxclients setting docker compose exec redis redis-cli CONFIG GET maxclients ✅ Returns 30000
Baseline connections docker compose exec redis redis-cli INFO clients ✅ ~900-1500 clients
No connection errors docker compose logs gateway | grep "too many connections" ✅ No errors

✅ Checklist

  • Code formatted (make black isort pre-commit)
  • Tests added/updated for changes
  • Documentation updated
  • No secrets or credentials committed

📓 Notes (optional)

Files Changed (11 total):

Docker Compose (4 files):

  • docker-compose.yml: maxclients 10000 → 30000
  • docker-compose-performance.yml: maxclients 20000 → 30000
  • docker-compose-debug.yml: maxclients 10000 → 30000
  • docker-compose-verbose-logging.yml: maxclients 10000 → 30000

Helm Chart (3 files):

  • charts/mcp-stack/templates/configmap-redis.yaml: Added maxclients directive
  • charts/mcp-stack/values.yaml: Added redis.maxclients: 30000
  • charts/mcp-stack/values.schema.json: Added schema validation

Documentation (4 files):

  • docs/docs/architecture/performance-architecture.md
  • docs/docs/manage/scale.md (2 locations)
  • docs/docs/manage/tuning.md
  • docs/docs/manage/configuration.md (added warning box)

Capacity Table:

Configuration Replicas Workers Pool/worker Total Within 30K limit?
docker-compose default 3 24 100 7,200
docker-compose @ 10 replicas 10 24 100 24,000 ✅ (was not with 10,000)
Helm chart default 10 2 50 1,000
Helm chart + 24 workers 10 24 50 12,000

Why 30,000?

  • Required capacity: 10 replicas × 24 workers × 100 pool = 24,000
  • With 20% headroom: 24,000 × 1.2 = 28,800

…4630)

- Update maxclients to 30,000 in all docker-compose files
- Add redis.maxclients configuration to Helm chart (values.yaml, schema, configmap)
- Update capacity formula comments: replicas × workers × pool < maxclients
- Add documentation about server-side vs client-side connection limits
- Fixes #4630: Redis maxclients exceeded at 10 replicas

Formula: 10 replicas × 24 workers × 100 pool = 24,000 connections
With 20% headroom: 28,800 → 30,000 maxclients

This addresses the root cause (server-side capacity) rather than
degrading client-side performance by reducing connection pools.

Signed-off-by: Claudia Gray <[email protected]>
@claudia-gray claudia-gray requested a review from Lang-Akshay May 12, 2026 12:53
@claudia-gray claudia-gray added the bug Something isn't working label May 12, 2026
@claudia-gray claudia-gray self-assigned this May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Redis maxclients limit exceeded when scaling to 10 replicas — connection formula unguarded

1 participant