support for non-amazonaws endpoint urls (e.g. s3.<company>.com) #436
Description
We are utilizing EMC on-premise s3-compatible storage.
Htslib does not accommodate non (.s3.amazonaws.com) endpoint urls.
We can successfully modify, re-compile and use htslib/samtools if we swap in our own endpoint url as below:
line 846 of hfile_libcurl.c
// Use virtual hosted-style access if possible, otherwise path-style.
if (is_dns_compliant(bucket, path)) {
kputsn(bucket, path - bucket, &url);
kputs("<custom_endpoint_url>", &url);
}
else {
kputs("<custom_endpoint_url>/", &url);
kputsn(bucket, path - bucket, &url);
}
kputs(path, &url);
Can htslib be modified to accept custom endpoint urls from all potential config locations?
e.g.
- embeded within link -
s3://id:secret:endpoint@bucket/(or similar) - env variable -
AWS_ENDPOINT_URL= - within ~/.aws/config -
aws_endpoint_url=(or some other standard, extracted from aws profile?)
Or modify to utilize a 's3_domain' parameter?
Looks like you've already touched on a few of these related to gcs:
#390
It would also be nice to modify the s3 logic to also take in a --profile parameter as used in aws-cli.