ReportPortal Migrations Complex Service

A Helm chart for migrating ReportPortal data between storage systems and upgrading database schemas. This service handles three types of migrations that can run independently or together in a single execution.

Quick Start
What This Service Does
Migration Types
Prerequisites
Configuration Guide
Migration Scenarios
Installation Steps
Troubleshooting

Quick Start

Add the Helm repository:

helm repo add reportportal-migrations https://reportportal.io/migrations-complex/
helm repo update

Download default values:

helm show values reportportal-migrations/migrations-complex > my-migration-values.yaml

Edit the values file to enable your desired migration(s) and configure credentials

Deploy the migration:

helm install my-migration reportportal-migrations/migrations-complex -f my-migration-values.yaml

Monitor the migration:

kubectl get pods -l app.kubernetes.io/name=migrations-complex
kubectl logs -l app.kubernetes.io/name=migrations-complex --follow

Clean up after completion:
```
helm uninstall my-migration
```

What This Service Does

This migration service helps you:

Upgrade to ReportPortal 23.3+: Migrate OAuth access tokens to API keys (required for versions 23.3 and higher)
Consolidate Storage: Merge multiple storage buckets into a single bucket (supports MinIO and S3)
Switch Storage Providers: Migrate data from MinIO to Amazon S3

Key Benefits:

✅ All migrations can run in a single job execution
✅ Migrations execute sequentially and automatically
✅ No service restart required between migrations
✅ Preserves data integrity throughout the process

Downtime Summary:

Migration Type	Downtime Required	Notes
API Keys Migration	⚠️ Yes	ReportPortal must be stopped
Multi-Bucket to Single-Bucket	⚠️ Yes	ReportPortal must be stopped (attachments table blocked)
MinIO to S3	✅ No	ReportPortal can run normally - switch to S3 and continue working

Migration Types

1. Database API Keys Migration

When to use: Upgrading from ReportPortal versions older than 23.3 to version 23.3 or higher

What it does: Converts OAuth access tokens in the database to API keys format

Downtime Required: ⚠️ Yes - ReportPortal must be stopped during this migration

Requirements: Database connection credentials

2. Multi-Bucket to Single-Bucket Migration

When to use: Consolidating data from multiple buckets (e.g., prj-1, prj-2, rp-bucket) into one bucket

What it does:

Finds all buckets matching your project prefix (e.g., prj-*)
Consolidates all data into a single destination bucket
Updates database records to point to the new bucket structure

Downtime Required: ⚠️ Yes - ReportPortal must be stopped during this migration (attachments table will be blocked)

Requirements:

Database connection credentials
Storage credentials (MinIO or S3, depending on destination)
Source buckets must be accessible

3. MinIO to S3 Migration

When to use: Moving from MinIO storage to Amazon S3

What it does: Transfers all data from a single MinIO bucket to a single S3 bucket

Downtime Required: ✅ No - ReportPortal can continue running! You can switch ReportPortal to use S3 storage and work normally while the migration runs in the background. The migration will copy data from MinIO to S3 without blocking ReportPortal operations.

Requirements:

MinIO credentials (source)
S3 credentials (destination)
Both buckets must exist and be accessible

Prerequisites

Before starting a migration, ensure you have:

✅ Kubernetes cluster with Helm 3.x installed
✅ Access to your ReportPortal database
✅ Storage credentials (MinIO and/or S3) with read/write permissions
✅ Sufficient cluster resources (see Resource Configuration)
✅ Backup of your database (recommended)

Important Notes:

⚠️ API Keys Migration requires downtime - ReportPortal must be stopped during this migration
⚠️ Multi-Bucket to Single-Bucket Migration requires downtime - ReportPortal must be stopped during this migration
⚠️ MinIO to S3 Migration - ReportPortal can continue running! You can switch ReportPortal to use S3 storage and work normally while the migration runs in the background
⚠️ API Keys migration is irreversible - OAuth tokens will be permanently deleted
⚠️ Ensure you have sufficient storage space in the destination bucket

Configuration Guide

Quick Reference: All Parameters

Migration Enablement

migrations:
  database:
    apiKeys:
      enabled: false  # Set to true for API keys migration
  storage:
    multiBucketToSingleBucket:
      enabled: false  # Set to true to consolidate buckets
      destinationType: minio  # or "s3"
      removeSourceBuckets: false  # Set to true to delete source buckets after migration
      buckets:
        projectPrefix: "prj-"  # Prefix for project buckets (e.g., "prj-" matches prj-1, prj-2)
        pluginsBucket: "rp-bucket"  # Name of plugins bucket
        singleBucketName: "rp-storage"  # Destination bucket name
    minioToS3:
      enabled: false  # Set to true to migrate from MinIO to S3
      buckets:
        sourceBucket: "rp-storage"  # MinIO source bucket
        destinationBucket: "rp-s3-storage"  # S3 destination bucket

Database Configuration

database:
  secretName: ""  # Optional: K8s secret name containing password
  passwordKeyName: "postgresql-password"  # Key name in secret
  endpoint: <postgresql-release-name>-postgresql.default.svc.cluster.local
  port: 5432
  user: rpuser
  dbName: reportportal
  password: ""  # Required if secretName is not set

MinIO Configuration

storage:
  minio:
    secretName: ""  # Optional: K8s secret name
    accessKey: <minio-accesskey>  # Required if secretName not set
    secretKey: <minio-secretkey>  # Required if secretName not set
    accessKeyName: "access-key"  # Key name in secret
    secretKeyName: "secret-key"  # Key name in secret
    endpoint: <minio-release-name>-minio.default.svc.cluster.local  # Hostname only
    ssl: false  # true for https, false for http
    port: 9000

S3 Configuration

storage:
  s3:
    region: "eu-central-1"  # AWS region
    secretName: ""  # Optional: K8s secret name
    accessKey: <s3-accesskey>  # Required if secretName not set
    secretKey: <s3-secretkey>  # Required if secretName not set
    accessKeyName: "access-key"  # Key name in secret
    secretKeyName: "secret-key"  # Key name in secret
    ssl: true  # Use https (true) or http (false) for endpoint
    endpoint: s3.eu-central-1.amazonaws.com  # S3 host (protocol from ssl)

Resource Configuration

resources:
  limits:
    cpu: 500m  # Increase for faster migrations (e.g., 2000m for large datasets)
    memory: 512Mi  # Increase for faster migrations (e.g., 2Gi for large datasets)
  requests:
    cpu: 250m
    memory: 248Mi

Migration Scenarios

Scenario 1: API Keys Migration Only

Use case: Upgrading from ReportPortal < 23.3 to 23.3+

⚠️ Downtime Required: Yes - ReportPortal must be stopped during this migration

Configuration:

migrations:
  database:
    apiKeys:
      enabled: true

database:
  endpoint: my-postgresql.default.svc.cluster.local
  port: 5432
  user: rpuser
  dbName: reportportal
  password: "your-password"
  # OR use a secret:
  # secretName: "postgresql-secret"
  # passwordKeyName: "postgresql-password"

Steps:

⚠️ Stop ReportPortal before starting the migration
Configure database credentials
Deploy: helm install api-keys-migration reportportal-migrations/migrations-complex -f values.yaml
Wait for completion: kubectl wait --for=condition=complete job/api-keys-migration-migrations-complex --timeout=3600s
Clean up: helm uninstall api-keys-migration
Restart ReportPortal after migration completes

Scenario 2: Multi-Bucket to Single-Bucket

Use case: Consolidating multiple buckets into one bucket

⚠️ Downtime Required: Yes - ReportPortal must be stopped during this migration (attachments table will be blocked)

Configuration:

migrations:
  storage:
    multiBucketToSingleBucket:
      enabled: true
      destinationType: minio  # or "s3"
      removeSourceBuckets: false  # Set to true to delete source buckets after migration
      buckets:
        projectPrefix: "prj-"
        pluginsBucket: "rp-bucket"
        singleBucketName: "rp-storage"

storage:
  minio:  # Use this if destinationType is "minio"
    endpoint: my-minio.default.svc.cluster.local
    ssl: false
    port: 9000
    accessKey: "minioadmin"
    secretKey: "minioadmin"
  # OR
  s3:  # Use this if destinationType is "s3"
    region: "eu-central-1"
    ssl: true
    endpoint: s3.eu-central-1.amazonaws.com
    accessKey: "your-s3-access-key"
    secretKey: "your-s3-secret-key"

database:
  endpoint: my-postgresql.default.svc.cluster.local
  port: 5432
  user: rpuser
  dbName: reportportal
  password: "your-password"

Steps:

⚠️ Stop ReportPortal to avoid data conflicts
Configure storage and database credentials
Ensure destination bucket exists (or will be created automatically)
Deploy: helm install multi-to-single reportportal-migrations/migrations-complex -f values.yaml
Monitor: kubectl logs -f job/multi-to-single-migrations-complex
After completion, update ReportPortal configuration to use single bucket
Clean up: helm uninstall multi-to-single

Scenario 3: MinIO to S3 Migration

Use case: Moving from MinIO to Amazon S3

✅ No Downtime Required: ReportPortal can continue running! You can switch ReportPortal to use S3 storage and work normally while the migration runs in the background.

Configuration:

migrations:
  storage:
    minioToS3:
      enabled: true
      buckets:
        sourceBucket: "rp-storage"  # MinIO bucket name
        destinationBucket: "rp-s3-storage"  # S3 bucket name (must exist)

storage:
  minio:
    endpoint: my-minio.default.svc.cluster.local
    ssl: false
    port: 9000
    accessKey: "minioadmin"
    secretKey: "minioadmin"
  s3:
    region: "eu-central-1"
    ssl: true
    endpoint: s3.eu-central-1.amazonaws.com
    accessKey: "your-s3-access-key"
    secretKey: "your-s3-secret-key"

Steps:

Create the destination S3 bucket in AWS
Configure MinIO and S3 credentials
Switch ReportPortal to use S3 storage (update ReportPortal configuration to point to S3)
Deploy: helm install minio-to-s3 reportportal-migrations/migrations-complex -f values.yaml
ReportPortal can continue working normally - users can access ReportPortal while migration runs
Monitor progress: kubectl logs -f job/minio-to-s3-migrations-complex
Migration will copy all data from MinIO to S3 in the background
Once migration completes, clean up: helm uninstall minio-to-s3

Scenario 4: Complete Migration (All Steps)

Use case: Full migration: API keys + consolidate buckets + move to S3

Configuration:

migrations:
  database:
    apiKeys:
      enabled: true
  storage:
    multiBucketToSingleBucket:
      enabled: true
      destinationType: minio
      buckets:
        projectPrefix: "prj-"
        pluginsBucket: "rp-bucket"
        singleBucketName: "rp-storage"
    minioToS3:
      enabled: true
      buckets:
        sourceBucket: "rp-storage"  # Must match singleBucketName above
        destinationBucket: "rp-s3-storage"

storage:
  minio:
    endpoint: my-minio.default.svc.cluster.local
    ssl: false
    port: 9000
    accessKey: "minioadmin"
    secretKey: "minioadmin"
  s3:
    region: "eu-central-1"
    ssl: true
    endpoint: s3.eu-central-1.amazonaws.com
    accessKey: "your-s3-access-key"
    secretKey: "your-s3-secret-key"

database:
  endpoint: my-postgresql.default.svc.cluster.local
  port: 5432
  user: rpuser
  dbName: reportportal
  password: "your-password"

Execution Order:

API Keys Migration runs first (requires downtime)
Multi-Bucket to Single-Bucket runs second (requires downtime)
MinIO to S3 runs third (no downtime - ReportPortal can run)

Steps:

⚠️ Stop ReportPortal before starting migrations
Create destination S3 bucket
Configure all credentials
Deploy: helm install complete-migration reportportal-migrations/migrations-complex -f values.yaml
Monitor: kubectl logs -f job/complete-migration-migrations-complex
After API Keys and Multi-to-Single migrations complete, switch ReportPortal to use S3 storage and restart ReportPortal
ReportPortal can now run normally while MinIO-to-S3 migration continues in the background
After all migrations complete, clean up: helm uninstall complete-migration

Installation Steps

Step-by-Step Guide

Add Helm Repository

helm repo add reportportal-migrations https://reportportal.io/migrations-complex/
helm repo update

Download Default Values

helm show values reportportal-migrations/migrations-complex > migration-values.yaml

Edit Configuration
- Open migration-values.yaml in your editor
- Enable the migrations you need
- Fill in all required credentials
- Adjust resource limits if needed for large datasets

Verify Configuration

# Validate your values file
helm template test-migration reportportal-migrations/migrations-complex -f migration-values.yaml --debug

Deploy Migration

helm install my-migration reportportal-migrations/migrations-complex -f migration-values.yaml

Monitor Progress

# Watch pod status
kubectl get pods -l app.kubernetes.io/name=migrations-complex -w

# View logs
kubectl logs -f -l app.kubernetes.io/name=migrations-complex

# Check job status
kubectl get job -l app.kubernetes.io/name=migrations-complex

Verify Completion

# Job should show "1/1 Completed"
kubectl get job -l app.kubernetes.io/name=migrations-complex

# Check pod logs for success messages
kubectl logs -l app.kubernetes.io/name=migrations-complex | tail -20

Clean Up
```
helm uninstall my-migration
```

Troubleshooting

Common Issues

Migration Job Fails to Start

Symptoms: Pod stays in Pending state

Solutions:

Check resource availability: kubectl describe pod <pod-name>
Verify image pull secrets if using private registry
Check node selectors and tolerations match your cluster

Database Connection Errors

Symptoms: Logs show database connection failures

Solutions:

Verify database endpoint is correct and accessible from the pod
Check database credentials (password or secret)
Ensure database allows connections from the pod's network
Test connection: kubectl run -it --rm debug --image=postgres:13 -- psql -h <db-endpoint> -U <user> -d <dbname>

Storage Access Errors

Symptoms: Logs show "access denied" or "bucket not found"

Solutions:

Verify storage credentials are correct
Check bucket exists and is accessible
For S3: Verify IAM permissions allow read/write operations
For MinIO: Verify endpoint, SSL, and port settings
Test access manually using AWS CLI or MinIO client

Migration Takes Too Long

Symptoms: Migration runs for hours without progress

Solutions:

Increase resource limits (CPU and memory)
Check network bandwidth between cluster and storage
Verify storage system performance
Monitor pod resource usage: kubectl top pod <pod-name>

"Bucket Already Exists" Error

Symptoms: Error when creating destination bucket

Solutions:

Bucket may already exist - this is usually fine
Ensure bucket is empty if reusing
Check bucket naming conflicts

Migration Partially Completes

Symptoms: Some data migrated, but not all

Solutions:

Check logs for specific errors
Verify source buckets are accessible
Ensure sufficient storage space in destination
Check for network interruptions during migration

Getting Help

Check Logs:

kubectl logs -l app.kubernetes.io/name=migrations-complex --tail=100

Describe Job/Pod:

kubectl describe job <job-name>
kubectl describe pod <pod-name>

Verify Configuration:
```
helm get values my-migration
```

Test Connectivity:

# Test database
kubectl run -it --rm test-db --image=postgres:13 -- psql -h <db-endpoint> -U <user>

# Test MinIO
kubectl run -it --rm test-minio --image=minio/mc -- mc alias set myminio http://<endpoint>:<port> <accesskey> <secretkey>

# Test S3
kubectl run -it --rm test-s3 --image=amazon/aws-cli -- aws s3 ls --endpoint-url <s3-endpoint>

Best Practices

✅ Always backup your database before running migrations
✅ Test migrations in a staging environment first
✅ Stop ReportPortal during storage migrations to avoid conflicts
✅ Monitor resource usage and adjust limits for large datasets
✅ Verify bucket names match between sequential migrations
✅ Keep migration job logs for troubleshooting
✅ Use Kubernetes secrets for sensitive credentials instead of plain text

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ReportPortal Migrations Complex Service

Table of Contents

Quick Start

What This Service Does

Migration Types

1. Database API Keys Migration

2. Multi-Bucket to Single-Bucket Migration

3. MinIO to S3 Migration

Prerequisites

Configuration Guide

Quick Reference: All Parameters

Migration Enablement

Database Configuration

MinIO Configuration

S3 Configuration

Resource Configuration

Migration Scenarios

Scenario 1: API Keys Migration Only

Scenario 2: Multi-Bucket to Single-Bucket

Scenario 3: MinIO to S3 Migration

Scenario 4: Complete Migration (All Steps)

Installation Steps

Step-by-Step Guide

Troubleshooting

Common Issues

Migration Job Fails to Start

Database Connection Errors

Storage Access Errors

Migration Takes Too Long

"Bucket Already Exists" Error

Migration Partially Completes

Getting Help

Best Practices

Additional Resources

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

ReportPortal Migrations Complex Service

Table of Contents

Quick Start

What This Service Does

Migration Types

1. Database API Keys Migration

2. Multi-Bucket to Single-Bucket Migration

3. MinIO to S3 Migration

Prerequisites

Configuration Guide

Quick Reference: All Parameters

Migration Enablement

Database Configuration

MinIO Configuration

S3 Configuration

Resource Configuration

Migration Scenarios

Scenario 1: API Keys Migration Only

Scenario 2: Multi-Bucket to Single-Bucket

Scenario 3: MinIO to S3 Migration

Scenario 4: Complete Migration (All Steps)

Installation Steps

Step-by-Step Guide

Troubleshooting

Common Issues

Migration Job Fails to Start

Database Connection Errors

Storage Access Errors

Migration Takes Too Long

"Bucket Already Exists" Error

Migration Partially Completes

Getting Help

Best Practices

Additional Resources