mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Added a comment

Browse source
This commit is contained in:
jpds 2023-01-31 22:28:16 +00:00 • committed by admin
parent bc6e606014
commit 048e77a04c
1 changed files with 45 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

45
doc/bugs/S3_remote_errors_with_garage_endpoint/comment_6_513bdbb6595612432fd4f33bd8e4e737._comment Normal file
View file

@ -0,0 +1,45 @@
[[!comment format=mdwn
username="jpds"
avatar="http://cdn.libravatar.org/avatar/24d746ec6a7726b162c12ecceb3ee267"
subject="comment 6"
date="2023-01-31T22:28:16Z"
content="""
> Firstly, this code is working, as far as I know, when accessing AWS.
It's working, because presumably AWS accept 's3.amazon.com' as a region setting for the scope in their cloud.
> I've verified that the s3Endpoint is supposed to be a hostname. It's used as such in s3SignQuery:
>
> (host, path) = case s3RequestStyle of
> PathStyle -> ([Just s3Endpoint], [Just \"/\", fmap (`B8.snoc` '/') s3QBucket, urlEncodedS3QObject])
>
> If s3Endpoint were just \"garage\" that would break.
My S3 endpoint is indeed the FQDN of my self-hosted S3 endpoint. My \"AWS region\" is \"garage\".
To give exact examples, this is what I configure in my environment when interacting with garage with any other S3 utility:
export AWS_ACCESS_KEY_ID=foo
export AWS_SECRET_ACCESS_KEY=bar
export AWS_DEFAULT_REGION='garage'
export AWS_ENDPOINT='https://my-s3-endpoint.example.com/'
The region bit is what's added to the credentialV4 in Core.hs (and what AWS appear to also accept as s3.amazon.com(? - I do not have an AWS account to test this). To make git-annex work, I need to do the equivalent of this (after changing garage's configuration):
export AWS_DEFAULT_REGION='my-s3-endpoint.example.com'
Once I do this, awscli and every other tool breaks:
$ aws s3 ls s3://
Provided region_name 'my-s3-endpoint.example.com' doesn't match a supported format.
> s3SignQuery also has region = s3ExtractRegion s3Endpoint. s3ExtractRegion parses a hostname like \"s3-foo.amazonaws.com\" to \"foo\", and that is used as the region (or \"scope\" as you've called it).
>
> git-annex makes sure to set s3Endpoint to a hostname. When the default AWS hostname is used, it converts the datacenter=foo value to a hostname like s3-foo.amazonaws.com and sets s3Endpoint to that. When some other hostname= is provided, it set s3Endpoints to that hostname, ignoring the datacenter= value.
It might be best to deprecate datacenter= and add region= directly to match AWS parlance
> This seems like the only thing git-annex can do. Your proposed patch would make a configuration of \"datacenter=us-west-1\" set s3Endpoint to \"us-west-1\" and s3SignQuery would use that as the host.
What I'd really like to test is just being able to set datacenter=garage but with my change it's not accepted as a [Char].
"""]]