mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

initial idea on s3

Browse source
This commit is contained in:
yarikoptic 2018-09-19 13:17:41 +00:00 committed by admin
parent 6af5a5d043
commit 4528bbab43
1 changed files with 25 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
doc/todo/Natively_support_s3__58____47____47___urls___40__for_addurl__44___get__44___etc__41__.mdwn Normal file
View file

@ -0,0 +1,25 @@
Given that git-annex has interactions with AWS S3 built-in, similar to my whining about ssh:// urls, I wondered if may be s3:// urls could be supported directly by git-annex.
Unfortunately not the case, and messages are a tiny bit misleading (see below) that initially annex just says that configuration disallows access to S3 but when tried to allow, seems to offload that to libcurl which doesn't support it.
The reason I am asking, is that lots of data is on S3 and for now we either relied on our datalad special remote to provide access to S3:// so we could authenticate, but for public buckets it would be overkill to demand datalad. Although we could replace them with http urls, I thought it might be better if annex could just download s3:// directly.
[[!format sh """
$> git annex addurl s3://images.cocodataset.org/annotations/image_info_test2014.zip
addurl s3://images.cocodataset.org/annotations/image_info_test2014.zip Configuration does not allow accessing s3://images.cocodataset.org/annotations/image_info_test2014.zip
Configuration does not allow accessing s3://images.cocodataset.org/annotations/image_info_test2014.zip
failed
git-annex: addurl: 1 failed
$> git -c annex.security.allowed-url-schemes="http https s3" -c annex.security.allowed-http-addresses=all annex addurl s3://images.cocodataset.org/annotations/image_info_test2014.zip
addurl s3://images.cocodataset.org/annotations/image_info_test2014.zip
curl: (1) Protocol "s3" not supported or disabled in libcurl
failed
git-annex: addurl: 1 failed
$> git annex version
git-annex version: 6.20180913+git33-g2cd5a723f-1~ndall+1
"""]]
[[!meta author=yoh]]