A specialized diagnostic data collection tool for Red Hat Developer Hub (RHDH) deployments on Kubernetes and OpenShift clusters.
This must-gather helps users collect essential RHDH data from their deployments. This helps support teams and engineers troubleshoot issues effectively. This tool allows for focused data gathering across any installation method and platform supported by RHDH.
- Multi-platform: OpenShift and standard Kubernetes
- Multi-deployment: Helm-based and Operator-managed RHDH instances
- RHDH-focused collection: Only RHDH-specific logs, configurations, and resources
You can use the OpenShift client CLI:
# Use the published image with the default options
oc adm must-gather --image=quay.io/rhdh-community/rhdh-must-gather
# Or to pass specific options to the gather script
oc adm must-gather --image=quay.io/rhdh-community/rhdh-must-gather -- /usr/bin/gather [options...]You can use the RHDH Must-Gather Helm chart:
# 1. Install must-gather with default options
helm install my-rhdh-must-gather redhat-developer-hub-must-gather \
--repo https://redhat-developer.github.io/rhdh-chart
# 2. Wait for the gather to complete (pod becomes ready when init container finishes)
kubectl wait --for=condition=ready pod \
-l app.kubernetes.io/instance=my-rhdh-must-gather,app.kubernetes.io/component=gather \
--timeout=3600s
# 3. Download the collected data from the data-holder container
kubectl exec deploy/my-rhdh-must-gather -c data-holder -- \
tar czf - -C /must-gather . > rhdh-must-gather-output.tar.gz
# 4. Clean up
helm uninstall my-rhdh-must-gatherWith custom options:
# Enable heap dump collection
helm install my-rhdh-must-gather redhat-developer-hub-must-gather \
--repo https://redhat-developer.github.io/rhdh-chart \
--set gather.heapDump.enabled=true
# Collect from specific namespaces with secrets
helm install my-rhdh-must-gather redhat-developer-hub-must-gather \
--repo https://redhat-developer.github.io/rhdh-chart \
--set "gather.namespaces={rhdh-prod,rhdh-staging}" \
--set gather.withSecrets=true
# Use a specific image version
helm install my-rhdh-must-gather redhat-developer-hub-must-gather \
--repo https://redhat-developer.github.io/rhdh-chart \
--set image.tag=v1.0.0See the chart documentation for all available options.
See data-collected.md for more details.
See omc.md for more details.
See heap-dumps-collection.md for more details.
See secret-collection-and-sanitization.md for more details.
For running in airgapped environments (partially or fully disconnected), see disconnected-environments.md for instructions on mirroring the container image and/or Helm chart to your internal registry.
| Variable | Default | Description |
|---|---|---|
BASE_COLLECTION_PATH |
/must-gather |
Output directory for collected data |
LOG_LEVEL |
info |
Logging level (info, debug, trace) |
CMD_TIMEOUT |
30 |
Timeout for individual kubectl/helm commands (seconds) |
MUST_GATHER_SINCE |
- | Relative time for log collection (e.g., "2h", "30m") |
MUST_GATHER_SINCE_TIME |
- | Absolute timestamp for log collection (RFC3339) |
HEAP_DUMP_TIMEOUT |
600 |
Timeout for heap dump collection in seconds |
HEAP_DUMP_BUFFER_SIZE |
16777216 |
WebSocket buffer size in bytes (16MB) for inspector method |
HEAP_DUMP_REMOTE_DIR |
/tmp |
Directory in container for heap dumps (SIGUSR2 method) |
Usage: ./must_gather [params...]
A client tool for gathering RHDH information from Helm and Operator installations in an OpenShift or Kubernetes cluster
Available options:
> To see this help menu and exit, use:
--help
> By default, the tool collects RHDH-specific information including:
> - platform
> - helm
> - operator
> - orchestrator
> - route
> - ingress
> - namespace-inspect
> You can exclude specific data collection types:
--without-operator Skip operator-based RHDH deployment data collection
--without-orchestrator Skip Orchestrator-flavored deployment data collection
(OpenShift Serverless, Serverless Logic, SonataFlowPlatform)
--without-helm Skip Helm-based RHDH deployment data collection
--without-platform Skip platform detection and information
--without-route Skip OpenShift route collection
--without-ingress Skip Kubernetes ingress collection
--without-namespace-inspect Skip deep Namespace inspect
> You can also choose to enable optional collectors:
--cluster-info Collect cluster-wide diagnostic information
> You can limit collection to specific namespaces:
--namespaces ns1,ns2 Collect data only from specified comma-separated namespaces
> Security and Privacy Options:
--with-secrets Include Kubernetes Secrets in collection (opt-in, disabled by default)
When disabled, secret resources are excluded from all collectors
When enabled, secrets are collected but automatically sanitized
> Diagnostic and Troubleshooting Options:
--with-heap-dumps Collect heap dumps from running backstage-backend processes (opt-in, disabled by default)
Heap dumps are collected immediately after pod logs for each deployment/CR
Useful for troubleshooting memory leaks and performance issues
IMPORTANT: Requires NODE_OPTIONS environment variable to be set on the backstage-backend RHDH container:
NODE_OPTIONS=--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp
Why these flags?
β’ --heapsnapshot-signal=SIGUSR2: Built into Node.js v12.0.0+, enables heap dumps
β’ --diagnostic-dir=/tmp: REQUIRED for read-only root filesystems
No image rebuild or source code changes needed!
Collection method: SIGUSR2 signal sent directly via kubectl exec
Works with any Kubernetes version, no special RBAC permissions needed
Warning: May take several minutes and generate large files (100MB-1GB+ per pod)
Examples:
# Default collection (includes Namespace inspect for OMC compatibility)
./must_gather
# Collect only Helm data (skip operator data)
./must_gather --without-operator
# Collect only operator data (skip Helm data)
./must_gather --without-helm
# Skip Namespace inspect (not recommended - removes OMC compatibility)
./must_gather --without-namespace-inspect
# Minimal collection (only platform info, no Namespace inspect)
./must_gather --without-operator --without-helm --without-route --without-ingress --without-namespace-inspect
# Collect from specific namespaces only
./must_gather --namespaces rhdh-prod,rhdh-staging
# Combine namespace filtering with exclusions
./must_gather --namespaces my-rhdh-ns --without-operator
# Include secrets in collection (for detailed troubleshooting - secrets will be sanitized)
./must_gather --with-secrets
# Collect heap dumps for memory troubleshooting (requires app configured with NODE_OPTIONS)
# Prerequisites: Add NODE_OPTIONS=--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp to the backstage-backend container
./must_gather --with-heap-dumps
# Full diagnostic collection (secrets + heap dumps - generates large output)
./must_gather --with-secrets --with-heap-dumps| Flag | Description | Use Case |
|---|---|---|
--without-operator |
Skip operator-based RHDH deployment data | When you know RHDH is deployed via Helm only |
--without-orchestrator |
Skip Orchestrator-related data (Serverless, Serverless Logic, SonataFlowPlatform) | When you know you don't have any Orchestrator flavor instances of RHDH |
--without-helm |
Skip Helm-based RHDH deployment data | When you know RHDH is deployed via Operator only |
--without-platform |
Skip platform detection and information | For minimal collections when platform info is not needed |
--without-route |
Skip OpenShift route collection | For non-OpenShift clusters or when routes are not relevant |
--without-ingress |
Skip Kubernetes ingress collection | When ingresses are not used for RHDH access |
--without-namespace-inspect |
Skip deep Namespace inspect | Not recommended as it removes OMC compatibility. Use only for minimal/quick collections |
| Flag | Description | Use Case |
|---|---|---|
--namespaces ns1,ns2 |
Limit collection to specified comma-separated namespaces | When RHDH is deployed in specific known namespaces |
--namespaces=ns1,ns2 |
Alternative syntax for namespace filtering | Same as above with equals syntax |
Examples:
--namespaces rhdh-prod,rhdh-staging- Collect only from production and staging namespaces--namespaces=my-rhdh-ns- Collect only from a single namespace- Combine with exclusions:
--namespaces prod-ns --without-helm- Only operator data from prod-ns
| Flag | Description | Use Case |
|---|---|---|
--cluster-info |
Collect cluster-wide diagnostic information | For comprehensive cluster analysis |
--with-secrets |
Include Kubernetes Secrets (sanitized) | For detailed troubleshooting requiring secret metadata |
--with-heap-dumps |
Collect heap dumps from backstage-backend containers | For memory leak investigation and performance analysis |
Examples:
--with-heap-dumps- Collect heap dumps for all backstage-backend pods--with-secrets --with-heap-dumps- Full diagnostic collection--namespaces prod-ns --with-heap-dumps- Heap dumps from specific namespace only
Click to expand
/must-gather/
βββ version # Tool version information (e.g., "rhdh-must-gather x.y.z-sha")
βββ sanitization-report.txt # Data sanitization summary and details
βββ all-routes.txt # All OpenShift routes cluster-wide
βββ all-ingresses.txt # All Kubernetes ingresses cluster-wide
βββ must-gather.log # Must-gather container logs (if running in pod)
βββ cluster-info/ # Cluster-wide information (if --cluster-info used)
β βββ [cluster-info dump output]
βββ namespace-inspect/ # Deep Namespace inspect (collected by default)
β βββ inspect.log # Inspection command logs
β βββ inspection-summary.txt # Summary of inspected namespaces and data collected
β βββ namespaces/ # All inspected namespaces (OMC-compatible structure)
β β βββ [namespace-1]/ # First namespace (e.g., "rhdh-prod")
β β β βββ [namespace].yaml # Namespace definition
β β β βββ apps/ # Application resources (Deployments, StatefulSets, etc.)
β β β βββ core/ # Core resources (ConfigMaps, Secrets, Services, etc.)
β β β βββ networking.k8s.io/ # Network policies and configurations
β β β βββ batch/ # Jobs and CronJobs
β β β βββ autoscaling/ # HPA and scaling configurations
β β β βββ pods/ # Detailed pod information with logs
β β β βββ [pod-name]/
β β β βββ [pod-name].yaml
β β β βββ [container-name]/
β β β βββ logs/
β β β βββ current.log
β β β βββ previous.log
β β β βββ previous.insecure.log
β β βββ [namespace-2]/ # Second namespace (e.g., "rhdh-staging")
β β β βββ [same structure as above]
β β βββ [namespace-N]/ # Additional namespaces...
β βββ aggregated-discovery-api.yaml
β βββ aggregated-discovery-apis.yaml
β βββ event-filter.html # Events visualization
βββ platform/ # Platform and infrastructure information
β βββ platform.json # Structured platform data (platform, underlying, versions)
β βββ platform.txt # Human-readable platform summary
βββ helm/ # Helm deployment data (native releases + standalone)
β βββ all-rhdh-releases.txt # List of all detected RHDH deployments (native + standalone)
β βββ all-rhdh-releases.json # Same data in JSON format
β βββ releases/ # Native Helm releases (tracked by Helm)
β β βββ ns=[namespace]/ # Per-namespace organization
β β βββ _configmaps/ # Namespace-wide ConfigMaps with both formats
β β β βββ [configmap-name].yaml # Full ConfigMap YAML
β β β βββ [configmap-name].describe.txt # kubectl describe output
β β βββ _secrets/ # Namespace-wide Secrets (sanitized)
β β β βββ [secret-name].yaml # Full Secret YAML (sanitized)
β β β βββ [secret-name].describe.txt # kubectl describe output (data redacted)
β β βββ [release-name]/ # Per-release directory
β β βββ values.yaml # User-provided values
β β βββ all-values.yaml # All computed values (25KB+ files)
β β βββ manifest.yaml # Deployed manifest (18KB+ files)
β β βββ hooks.yaml # Helm hooks
β β βββ history.txt # Release history
β β βββ history.yaml # Release history (YAML)
β β βββ status.txt # Release status (text)
β β βββ status.yaml # Release status (YAML, 21KB+ files)
β β βββ notes.txt # Release notes
β β βββ deployment/ # Application deployment info
β β β βββ deployment.yaml
β β β βββ deployment.describe.txt
β β β βββ logs/ # Per-pod, per-container logs (all replicas)
β β β β βββ pod=[pod-name]/
β β β β βββ logs-app.current.txt # Aggregated current logs (all containers)
β β β β βββ logs-app.previous.txt # Aggregated previous logs (all containers)
β β β β βββ container=[container-name]/
β β β β βββ current.txt # Current container logs
β β β β βββ previous.txt # Previous container instance logs
β β β βββ data/ # Per-pod application data (all running replicas)
β β β β βββ pod=[pod-name]/
β β β β βββ container=backstage-backend/
β β β β βββ env-vars.txt # RHDH/Backstage environment variables
β β β β βββ app-container-userid.txt # "uid=1001 gid=0(root) groups=0(root)"
β β β β βββ backstage.json # {"version": "1.39.1"}
β β β β βββ build-metadata.json # RHDH version, source repos
β β β β βββ node-version.txt # "v22.16.0"
β β β β βββ dynamic-plugins-root.fs.txt # Directory listing with plugin packages
β β β β βββ app-config.dynamic-plugins.yaml # Generated app config
β β β βββ heap-dumps/ # Memory heap dumps (if --with-heap-dumps used)
β β β β βββ pod=[pod-name]/
β β β β βββ pod-spec.yaml # Pod specification
β β β β βββ container=[container-name]/
β β β β βββ heapdump-[timestamp].heapsnapshot # Heap dump (100MB-1GB+)
β β β β βββ process-info.txt # Process and memory info
β β β β βββ heap-dump.log # Collection logs
β β β βββ processes/ # Process list from running pods (all replicas)
β β β β βββ pod=[pod-name]/
β β β β βββ container=[container-name].txt # Process list per container
β β β βββ rollout-history/ # Rollout revision history
β β β β βββ history.txt # kubectl rollout history output
β β β β βββ replicasets/ # ReplicaSets backing deployment revisions
β β β β βββ replicasets.yaml
β β β β βββ replicasets.describe.txt
β β β βββ pods/ # Pod listing and details
β β β βββ pods.txt
β β β βββ pods.yaml
β β β βββ pods.describe.txt
β β βββ db-statefulset/ # Database StatefulSet info (if database enabled)
β β βββ db-statefulset.yaml
β β βββ db-statefulset.describe.txt
β β βββ logs-db.txt # Database logs
β β βββ logs-db-previous.txt
β β βββ rollout-history/ # Rollout revision history
β β β βββ history.txt # kubectl rollout history output
β β β βββ controllerrevisions/ # ControllerRevisions backing statefulset revisions
β β β βββ controllerrevisions.yaml
β β β βββ controllerrevisions.describe.txt
β β βββ pods/ # Database pod details
β β βββ pods.txt
β β βββ pods.yaml
β β βββ pods.describe.txt
β βββ standalone/ # Standalone Helm deployments (helm template + kubectl apply)
β βββ ns=[namespace]/ # Per-namespace organization
β βββ [workload-name]/ # Per-workload directory (Deployment or StatefulSet name)
β βββ standalone-note.txt # Explanation of standalone detection
β βββ helm-metadata.txt # Extracted Helm labels (chart, instance, version)
β βββ deployment.yaml # Deployment YAML (or statefulset.yaml)
β βββ deployment.describe.txt
β βββ deployment/ # Application deployment info (same as native releases)
β β βββ logs/
β β βββ data/
β β βββ pods/
β β βββ processes/
β β βββ rollout-history/
β βββ dependencies/ # Dependent services (e.g., PostgreSQL from subchart)
β βββ [dep-name]/ # Per-dependency directory
β βββ statefulset.yaml # Dependency workload YAML
β βββ statefulset.describe.txt
β βββ logs-[pod].txt # Dependency logs
βββ orchestrator/ # Orchestrator-flavored deployment data (if detected)
β βββ summary.txt # Summary of all detected Orchestrator components and versions
β βββ serverless-operators/ # OpenShift Serverless operators information
β β βββ ns=openshift-serverless/ # OpenShift Serverless operator namespace
β β β βββ csv-list.txt # ClusterServiceVersion list (operator version)
β β β βββ csv-all.yaml # CSV details
β β β βββ deployments.txt # Operator deployments
β β β βββ pods.txt # Operator pods
β β β βββ subscriptions.txt # OLM subscriptions
β β β βββ logs-*.txt # Operator logs
β β βββ ns=openshift-serverless-logic/ # OpenShift Serverless Logic operator namespace
β β βββ csv-list.txt # ClusterServiceVersion list (operator version)
β β βββ csv-all.yaml # CSV details
β β βββ deployments.txt # Operator deployments
β β βββ pods.txt # Operator pods
β β βββ subscriptions.txt # OLM subscriptions
β β βββ logs-logic-operator.txt # Logic operator logs
β βββ crds/ # Orchestrator-related Custom Resource Definitions
β β βββ found-crds.txt # List of detected Orchestrator CRDs
β β βββ sonataflowplatforms.sonataflow.org.yaml # SonataFlowPlatform CRD
β β βββ sonataflows.sonataflow.org.yaml # SonataFlow CRD
β β βββ knativeservings.operator.knative.dev.yaml # KnativeServing CRD
β β βββ knativeeventings.operator.knative.dev.yaml # KnativeEventing CRD
β βββ sonataflow-platforms/ # SonataFlowPlatform Custom Resources
β β βββ all-sonataflow-platforms.txt # List of all SonataFlowPlatform CRs
β β βββ ns=[namespace]/ # Per-namespace SonataFlowPlatform data
β β βββ [platform-name]/ # Per-platform directory
β β βββ [platform-name].yaml # SonataFlowPlatform CR definition
β β βββ describe.txt # CR description
β β βββ related-deployments.txt # Platform-related deployments
β β βββ related-services.txt # Platform-related services
β β βββ logs.txt # Platform-related logs
β βββ sonataflow-workflows/ # SonataFlow Workflow Custom Resources
β β βββ all-sonataflow-workflows.txt # List of all SonataFlow workflows
β β βββ ns=[namespace]/ # Per-namespace workflow data
β β βββ all-workflows.txt # List of workflows in this namespace
β β βββ [workflow-name]/ # Per-workflow directory
β β βββ workflow.yaml # SonataFlow workflow definition
β β βββ describe.txt # Workflow description
β β βββ pods.txt # Workflow pods
β β βββ logs.txt # Workflow logs
β βββ knative/ # Knative resources
β βββ knative-serving-list.txt # KnativeServing CRs list
β βββ knative-serving.yaml # KnativeServing CR details
β βββ knative-eventing-list.txt # KnativeEventing CRs list
β βββ knative-eventing.yaml # KnativeEventing CR details
β βββ knative-kafka-list.txt # KnativeKafka CRs list
β βββ knative-kafka.yaml # KnativeKafka CR details
β βββ knative-serving/ # knative-serving namespace resources
β β βββ deployments.txt
β β βββ pods.txt
β β βββ services.txt
β βββ knative-eventing/ # knative-eventing namespace resources
β βββ deployments.txt
β βββ pods.txt
β βββ services.txt
βββ operator/ # Operator deployment data (if RHDH operators found)
βββ all-deployments.txt # List of all RHDH operator deployments
βββ olm/ # OLM information
β βββ rhdh-csv-all.txt # ClusterServiceVersions
β βββ rhdh-subscriptions-all.txt # Subscriptions
β βββ installplans-all.txt # InstallPlans
β βββ operatorgroups-all.txt # OperatorGroups
β βββ catalogsources-all.txt # CatalogSources
βββ crds/ # Custom Resource Definitions
β βββ all-crds.txt # All CRDs in cluster
β βββ backstages.rhdh.redhat.com.yaml # RHDH CRD definition
β βββ backstages.rhdh.redhat.com.describe.txt # CRD description
βββ ns=[operator-namespace]/ # Per-operator-namespace data (e.g., ns=rhdh-operator)
β βββ all-resources.txt # All resources in namespace
β βββ configs/ # ConfigMaps with both formats
β β βββ all-configmaps.txt
β β βββ [configmap-name].yaml # Full ConfigMap YAML
β β βββ [configmap-name].describe.txt # kubectl describe output
β βββ deployments/ # Operator deployments
β β βββ all-deployments.txt
β β βββ [deployment-selector].yaml
β β βββ [deployment-selector].describe.txt
β βββ logs.txt # Operator logs
β βββ logs-previous.txt # Operator logs (previous instances)
βββ backstage-crs/ # Backstage Custom Resources
βββ all-backstage-crs.txt # List of all Backstage CRs
βββ ns=[cr-namespace]/ # Per-CR-namespace data (where Backstage CRs are deployed)
βββ _configmaps/ # Namespace-wide ConfigMaps with both formats
β βββ [configmap-name].yaml # Full ConfigMap YAML
β βββ [configmap-name].describe.txt # kubectl describe output
βββ _secrets/ # Namespace-wide Secrets (sanitized)
β βββ [secret-name].yaml # Full Secret YAML (sanitized)
β βββ [secret-name].describe.txt # kubectl describe output (data redacted)
βββ [cr-name]/ # Per-CR directory
βββ [cr-name].yaml # CR definition
βββ describe.txt # CR description
βββ warning-dual-workload.txt # Warning if both Deployment and StatefulSet exist
βββ deployment/ # RHDH workload (Deployment or StatefulSet)
β βββ deployment.yaml # or statefulset.yaml if kind is StatefulSet
β βββ deployment.describe.txt # or statefulset.describe.txt
β βββ logs/ # Per-pod, per-container logs (all replicas)
β β βββ pod=[pod-name]/
β β βββ logs-app.current.txt # Aggregated current logs (all containers)
β β βββ logs-app.previous.txt # Aggregated previous logs (all containers)
β β βββ container=[container-name]/
β β βββ current.txt # Current container logs
β β βββ previous.txt # Previous container instance logs
β βββ data/ # Per-pod application data (all running replicas)
β β βββ pod=[pod-name]/
β β βββ container=backstage-backend/
β β βββ env-vars.txt # RHDH/Backstage environment variables
β β βββ app-container-userid.txt # "uid=1001 gid=0(root) groups=0(root)"
β β βββ backstage.json # {"version": "1.39.1"}
β β βββ build-metadata.json # RHDH version, source repos
β β βββ node-version.txt # "v22.16.0"
β β βββ dynamic-plugins-root.fs.txt # Directory listing with plugin packages
β β βββ app-config.dynamic-plugins.yaml # Generated app config
β βββ heap-dumps/ # Memory heap dumps (if --with-heap-dumps used)
β β βββ pod=[pod-name]/
β β βββ pod-spec.yaml # Pod specification
β β βββ container=[container-name]/
β β βββ heapdump-[timestamp].heapsnapshot # Heap dump (100MB-1GB+)
β β βββ process-info.txt # Process and memory info
β β βββ heap-dump.log # Collection logs
β βββ processes/ # Process list from running pods (all replicas)
β β βββ pod=[pod-name]/
β β βββ container=[container-name].txt # Process list per container
β βββ rollout-history/ # Rollout revision history
β β βββ history.txt # kubectl rollout history output
β β βββ replicasets/ # or controllerrevisions/ for StatefulSets
β β βββ replicasets.yaml
β β βββ replicasets.describe.txt
β βββ pods/ # Pod listing and details
β βββ pods.txt
β βββ pods.yaml
β βββ pods.describe.txt
βββ rhdh-statefulset/ # Only if dual workload detected (see warning-dual-workload.txt)
β βββ [same structure as deployment/ above, with statefulset.yaml]
βββ db-statefulset/ # Database StatefulSet (if database enabled)
βββ db-statefulset.yaml
βββ db-statefulset.describe.txt
βββ logs-db.txt # Database logs
βββ logs-db-previous.txt
βββ rollout-history/ # Rollout revision history
β βββ history.txt # kubectl rollout history output
β βββ controllerrevisions/ # ControllerRevisions backing statefulset revisions
β βββ controllerrevisions.yaml
β βββ controllerrevisions.describe.txt
βββ pods/ # Database pods
βββ pods.txt
βββ pods.yaml
βββ pods.describe.txt
Note: The tool automatically detects and collects data for both Helm and Operator-based RHDH deployments. For cluster-wide information, use the
--cluster-infoflag or combine with standardoc adm must-gather.
See CONTRIBUTING.md.
Apache License 2.0 - see LICENSE for details.