Skip to content

Support for rustfs as S3 storage for backups #71550

@balporig

Description

@balporig

Feature request

Is your feature request related to a problem? Please describe.

As you may know the community version of minio has been abandoned and should not be used anymore. We want to be able to use rustfs instead.

Describe the solution you'd like

First class support for rustfs as minio replacement.

Additional context

Did some testing with 300GB database and only making backup of a single table works. Backups of a database/multiple tables crashes rustfs. In particular if affects only partitioned tables where there are multiple small partitions eg. ~40 K. The problem might be with rustfs.

CREATE REPOSITORY backup_repo
WITH BROKER
ON LOCATION "s3://starrocks/snapshots"
PROPERTIES(
    "aws.s3.access_key" = "rustfsadmin",
    "aws.s3.secret_key" = "rustfsadmin",
    "aws.s3.enable_path_style_access" = "true",
    "aws.s3.region" = "some-reg",
    "aws.s3.enable_ssl" = "false",
    "aws.s3.endpoint" = "http://172.18.0.13:9000",
    "aws.s3.use_instance_profile" = "false",
    "aws.s3.use_aws_sdk_default_behavior" = "false"
);

works

BACKUP DATABASE testdb SNAPSHOT backup
TO backup_repo
ON (TABLE account_emailaddress);

does not

BACKUP DATABASE testdb SNAPSHOT backup
TO backup_repo
PROPERTIES ("type" = "full", "timeout" = "86400");

BACKUP DATABASE testdb SNAPSHOT sr_all_backup
TO backup_repo
ON (ALL TABLES);

BACKUP DATABASE testdb SNAPSHOT sr_core_backup_small
TO backup_repo
ON (TABLE table_one, TABLE table_two, TABLE table_three);

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions