mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

fix addurl concurrency issue

Browse source 
addurl: Support adding the same url to multiple files at the same time when
using -J with --batch --with-files.

Implementation was easier than expected, was able to reuse OnlyActionOn.

While it will download the url's content multiple times, that seems like
the best thing to do; see my comment for why.

Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
Joey Hess 2021-10-27 16:15:41 -04:00
parent a3cdff3fd5
commit eb95ed4863
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
5 changed files with 42 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
doc/bugs/addurl_failure_has_empty_error-messages.mdwn
View file

@ -2,3 +2,5 @@ While running `git-annex addurl --batch --with-files --jobs 10 --json --json-err
[[!meta author=jwodder]]
[[!tag projects/dandi]]
> [[fixed|done]] --[[Joey]]

2
doc/bugs/addurl_failure_has_empty_error-messages/comment_4_74e37e96d01bfb8b9521000d5faa7e53._comment
View file

@ -8,7 +8,7 @@ downloading, and the key transfer machinery prevents redundant downloads
of the same Key at the same time.
Arguably, the problem is not where the message gets put, but that
it fails when adding an url to two different paths.
it fails when adding an url to two different paths at the same time.
I have, though, moved that message so it will appear in error-messages.
"""]]

26
doc/bugs/addurl_failure_has_empty_error-messages/comment_5_1b87390e69204b42878930cc1614437e._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2021-10-27T18:56:23Z"
content="""
The best solution I can find is for it to notice when another thread is
downloading the same url, and wait until it finishes. Then proceed
with downloading the url for a second time.
It's not very satisfying to re-download. But once the url Key is downloaded,
it does not keep that url Key populated, but hashes the content and moves
the content to the final Key. It would be a real complication to
communicate, across threads, what Key the content ended up at, and have the
waiting thread use that. And addurl is already complicated well beyond a
point I am comfortable with.
Also, the content of an url can of course change over time. If I feed
"$url foo" into git-annex addurl --batch -J10 and then some time
later, I feed "$url bar", I might expect that file bar gets whatever
content the url has now, not the content that the url had back when I added
the same url to file foo. And if I cared about avoiding re-downloading,
I could add the url to the first file, and then copy the annex link to the
second file myself.
Implemented this approach.
"""]]