mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

web special remote is no longer a singleton

Browse source 
Allow initremote of additional special remotes with type=web, in addition
to the default web special remote.

When --sameas=web is used, these provide additional names for the web
special remote, and may also have their own additional configuration
(once there is any for the web special remote) and cost.

Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
Joey Hess 2023-01-09 15:49:20 -04:00
parent 60f7cfc96e
commit 8d06930c88
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 102 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

5
doc/git-annex.mdwn
View file

@ -1655,6 +1655,11 @@ Remotes are configured using these settings in `.git/config`.
Used to identify Amazon Glacier special remotes.
Normally this is automatically set up by `git annex initremote`.
* `remote.<name>.annex-web`
Used to identify web special remotes.
Normally this is automatically set up by `git annex initremote`.
* `remote.<name>.annex-webdav`
Used to identify webdav special remotes.

5
doc/special_remotes/bittorrent.mdwn
View file

@ -13,6 +13,9 @@ If git-annex is not built using the haskell torrent library to parse
torrents, it also needs the `btshowmetainfo` program, from either
bittornado or the original BitTorrent client.
The bittorrent special remote is always enabled, without any manual setup being
needed. Its name is "bittorrent".
## notes
Currently git-annex only supports downloading content from a torrent;
@ -27,5 +30,3 @@ git-annex takes a cautious approach and when dropping a file, won't
treat this special remote as one of the required [[copies]]. It's probably
a good idea to configure git-annex to fully distrust this remote, by
running `git annex untrust bittorrent`
This feature is available only from version `5.20141219`.

9
doc/special_remotes/web.mdwn
View file

@ -1,11 +1,12 @@
git-annex can use the WWW as a special remote, associating an url with an
git-annex can use the web as a special remote, associating an url with an
annexed file, and downloading the file content from the web.
See [[tips/using_the_web_as_a_special_remote]] for usage examples.
## notes
The web special remote is always enabled, without any manual setup being
needed. Its name is "web".
Currently git-annex only supports downloading content from the web;
it cannot upload to it or remove content.
This special remote can only be used for downloading content,
not uploading content, or removing content from the web.
This special remote uses urls on the web as the source for content.
There are several other ways http can be used to download annexed objects,

10
doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_2_1ab2634c8a7058c5aa09f1ba72be3471._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2023-01-09T18:25:56Z"
content="""
It does happen to try the urls in the order listed in the log file.
With that said, the order of lines in files in the git-annex branch is
not guaranteed to be preserved when eg, merging..
"""]]

50
doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_3_6970e60b7d826beb025fda452cc1bc13._comment Normal file
View file

@ -0,0 +1,50 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2023-01-09T18:27:36Z"
content="""
See previous discussion in the since-closed todo
[[assign_costs_per_URL_or_better_repo-wide_(regexes)]].
In that I suggested using --sameas, and yarikoptic thought the datalad
special remote could use that to be used to handle urls and do its own
prioritization. I suppose that probably didn't get done in datalad.
But I do like the idea of using --sameas. It avoids several problems
with providing "url-priority-N=" configs to the web special remote:
* Two clones could have url-priority-1 set to different values, and merging
the remote.log would lose one of them.
* It may be that one group of urls is slow from repo A, but fast from repo
B. So there needs to be a local override, and we already have that
for remote costs.
* Using --sameas means that cost configs can be used, rather than
adding a separate config that's essentially for the same thing.
So, what if it were possible to initremote versions of the web special
remote that were limited to particular urls, and skipped over any other
urls:
git-annex initremote --sameas web s3public type=web urllimit=s3.amazonaws.com/ autoenable=true
git config remote.s3public.annex-cost 150
git-annex initremote --sameas web dandiapi type=web urllimit=/api.dandiarchive.org/ autoenable=true
git config remote.dandiapi.annex-cost 250
As well as adding the urllimit= config, that would need the web special
remote to allow initremote of other instances of it. Currently, that will
fail:
git-annex initremote --sameas web web2 type=web autoenable=true
git-annex: not supported
CallStack (from HasCallStack):
error, called at ./Remote/Web.hs:32:19 in main:Remote.Web
failed
initremote: 1 failed
Which is not ideal when it comes to using autoenable=true because using
a current git-annex after this gets implemented would try to autoenable
the remote, and display all that. Compare with how autoenable handles
remote types it does not know -- it silently skips them. This could be avoided
by using something other than type=web for these.
"""]]