mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

fix transfer lock file for Download to not include uuid

Browse source 
While redundant concurrent transfers were already prevented in most
cases, it failed to prevent the case where two different repositories were
sending the same content to the same repository. By removing the uuid
from the transfer lock file for Download transfers, one repository
sending content will block the other one from also sending the same
content.

In order to interoperate with old git-annex, the old lock file is still
locked, as well as locking the new one. That added a lot of extra code
and work, and the plan is to eventually stop locking the old lock file,
at some point in time when an old git-annex process is unlikely to be
running at the same time.

Note that in the case of 2 repositories both doing eg
`git-annex copy foo --to origin`
the output is not that great:

copy b (to origin...)
  transfer already in progress, or unable to take transfer lock
git-annex: transfer already in progress, or unable to take transfer lock
97%   966.81 MiB      534 GiB/s 0sp2pstdio: 1 failed

  Lost connection (fd:14: hPutBuf: resource vanished (Broken pipe))

  Transfer failed

Perhaps that output could be cleaned up? Anyway, it's a lot better than letting
the redundant transfer happen and then failing with an obscure error about
a temp file, which is what it did before. And it seems users don't often
try to do this, since nobody ever reported this bug to me before.
(The "97%" there is actually how far along the *other* transfer is.)

Sponsored-by: Joshua Antonishen on Patreon
This commit is contained in:
Joey Hess 2024-03-25 14:47:38 -04:00
parent 62129f0b24
commit f04d9574d6
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 147 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

7
doc/bugs/redundant_transfer_not_prevented.mdwn
View file

@ -21,6 +21,9 @@ I'm easily able to reproduce this in a bench test with 2 clones of a repo.
(cd b; git-annex copy --to origin) &
(cd c; git-annex copy --to origin) &
Also same happens when running `git-annex get --from` two different remotes
concurrently in the same repo.
Aha... Looking at the code, this seems like a fundamental oversight.
The `transferFile` depends on the uuid of the remote being transfered
to/from, so there are two different ones in this case. And the transfer
@ -47,5 +50,7 @@ noticing multiple downloads from the same uuid. Which is not a great
behavior to break either, even if it would usually only break transiently.
Of course that could be avoided by keeping the current lock file, and
adding a second level lock file.
adding a second lock file. [[done]] this, with plans in
[[todo/v11_changes]] to transition to use only the new lock file in
the future.
--[[Joey]]