Support s3-compatible buckets that are not AWS

[rsignell]

It would be great to be more flexible to accept buckets that are s3-compatible, but need to pass endpoint_url (and don't use region). Like Cloudflare-R2, Wasabi, OSN, EGI, MinIO, Rustfs, etc.

Here's an example that hits my Cloudflare public bucket:

  "storage:schemes": {
    "r2-osc-pub": {
      "type": "aws-s3",
      "platform": "https://9cbdcb4884f86a6779032ae561e474a5.r2.cloudflarestorage.com",
      "bucket": "osc-pub",
      "region": "not-used",
      "endpoint_url": "https://9cbdcb4884f86a6779032ae561e474a5.r2.cloudflarestorage.com"
    },

which is contained in this full STAC Item example (should be reproducible):
https://github.com/OpenScienceComputing/NCICS-2026/blob/main/ndvi_cdr_stac_item.json

[jsignell]

Is "aws-s3" the established type for cloudflare? Have you seen others using it this way. I'm just wondering if there should be a different "type" that is "generic-s3" or something.

[rsignell]

Yeah, I just put that in there hoping that would make it work. 😄
Usually people just refer to these as "s3-compatible" How about that?

[rsignell]

An additional wrinkle. While some s3-compatible providers support anonymous S3 access to public buckets, Cloudflare R2 does not.

So for public access to Cloudflare Icechunk repos from Python, we need to use HTTPS access, not S3, and for Icechunk therefore we specify r2_storage, for example:

import icechunk

storage = icechunk.r2_storage(
    endpoint_url="https://r2-pub.openscicomp.io",
    bucket="ndvi-cdr-pyramid",
    prefix=None,
    anonymous=True
)
repo = icechunk.Repository.open(storage)

What does this mean for STAC (or xstac)?

[rsignell]

This works, but it would be nice to use xpystac:

import requests
import xarray as xr
import icechunk

item = requests.get("https://r2-pub.openscicomp.io/stac/ndvi/ndvi_cdr_stac_item.json").json()
asset = next(v for v in item["assets"].values()
             if v.get("type") == "application/vnd.zarr+icechunk")
scheme = item["storage:schemes"][asset["storage:refs"][0]]
storage = icechunk.r2_storage(
    endpoint_url=scheme["endpoint_url"],
    bucket=scheme["bucket"],
    anonymous=True,
)
repo = icechunk.Repository.open(storage)
session = repo.readonly_session(snapshot_id=asset["icechunk:snapshot_id"])
ds = xr.open_zarr(session.store, consolidated=False)