mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/Command
History
Joey Hess 68a49adcda
Improve behavior when -J transfers multiple files that point to the same key 
After a false start, I found a fairly non-intrusive way to deal with it.
Although it only handles transfers -- there may be issues with eg
concurrent dropping of the same key, or other operations.

There is no added overhead when -J is not used, other than an added
inAnnex check. When -J is used, it has to maintain and check a small
Set, which should be negligible overhead.

It could output some message saying that the transfer is being done by
another thread. Or it could even display the same progress info for both
files that are being downloaded since they have the same content. But I
opted to keep it simple, since this is rather an edge case, so it just
doesn't say anything about the transfer of the file until the other
thread finishes.

Since the deferred transfer action still runs, actions that do more than
transfer content will still get a chance to do their other work. (An
example of something that needs to do such other work is P2P.Annex,
where the download always needs to receive the content from the peer.)
And, if the first thread fails to complete a transfer, the second thread
can resume it.

But, this unfortunately means that there's a risk of redundant work
being done to transfer a key that just got transferred.
That's not ideal, but should never cause breakage; the same
thing can occur when running two separate git-annex processes.

The get/move/copy/mirror --from commands had extra inAnnex checks added,
inside the download actions. Without those checks, the first thread
downloaded the content, and then the second thread woke up and
downloaded the same content redundantly.

move/copy/mirror --to is left doing redundant uploads for now. It
would need a second checkPresent of the remote inside the upload
to avoid them, which would be expensive. A better way to avoid
redundant work needs to be found..

This commit was supported by the NSF-funded DataLad project.
2017-10-17 17:10:50 -04:00
..
Add.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
AddUnused.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
AddUrl.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
Adjust.hs adjust: Add --fix adjustment, which is useful when the git directory is in a nonstandard place. 2016-05-16 17:18:33 -04:00
Assistant.hs assistant: Make --autostart --foreground wait for the children it starts. 2017-02-07 13:31:45 -04:00
Benchmark.hs change keys database to use IKey type with more efficient serialization 2016-01-12 14:01:50 -04:00
CalcKey.hs calckey: New plumbing command, calculates the key that would be used to refer to a file 2016-04-20 13:50:26 -04:00
CheckPresentKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Commit.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Config.hs inheritable annex.securehashesonly 2017-02-27 16:08:23 -04:00
ConfigList.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
ContentLocation.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Copy.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Dead.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Describe.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
DiffDriver.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Direct.hs correct spelling mistakes 2017-02-12 17:30:23 -04:00
Drop.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
DropKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
DropUnused.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
EnableRemote.hs implement exporttree=yes configuration 2017-09-04 13:09:38 -04:00
EnableTor.hs refactor 2016-12-30 12:31:17 -04:00
ExamineKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Expire.hs Added GIT_ANNEX_VECTOR_CLOCK environment variable 2017-08-14 14:19:58 -04:00
Export.hs update transfer info and notify when exporting 2017-09-20 12:58:23 -04:00
Find.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
FindRef.hs improve type 2017-08-31 12:47:08 -04:00
Fix.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Forget.hs Added GIT_ANNEX_VECTOR_CLOCK environment variable 2017-08-14 14:19:58 -04:00
FromKey.hs Make all --batch input, as well as fromkey and registerurl stdin be processed without requiring it to be in the current encoding. 2016-12-13 15:35:04 -04:00
Fsck.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
FuzzTest.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
GCryptSetup.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Get.hs Improve behavior when -J transfers multiple files that point to the same key 2017-10-17 17:10:50 -04:00
Group.hs make query commands not output extraneous messages 2017-02-16 13:24:35 -04:00
GroupWanted.hs make query commands not output extraneous messages 2017-02-16 13:24:35 -04:00
Help.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Import.hs import --reinject-duplicates 2017-02-09 15:41:00 -04:00
ImportFeed.hs toFeed was unused so remove 2017-08-28 12:51:25 -04:00
InAnnex.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Indirect.hs annex.securehashesonly 2017-02-27 13:33:59 -04:00
Info.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
Init.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
InitRemote.hs add annex-ignore-command and annex-sync-command configs 2017-08-17 13:54:14 -04:00
List.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Lock.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
LockContent.hs git-annex-shell, remotedaemon, git remote: Fix some memory DOS attacks. 2016-12-09 13:34:32 -04:00
Log.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
LookupKey.hs Fix reversion in lookupkey, contentlocation, and examinekey which caused them to sometimes output side messages. 2016-01-29 13:20:24 -04:00
Map.hs Support GIT_SSH and GIT_SSH_COMMAND 2017-03-17 16:20:37 -04:00
MatchExpression.hs matchexpression: Added --largefiles option to parse an annex.largefiles expression. 2016-02-03 16:58:36 -04:00
Merge.hs configuration to disable automatic merge conflict resolution 2017-06-01 12:51:01 -04:00
MetaData.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Migrate.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Mirror.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Move.hs Improve behavior when -J transfers multiple files that point to the same key 2017-10-17 17:10:50 -04:00
Multicast.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
NotifyChanges.hs make tor hidden service work when directory watching is not available 2016-12-09 16:40:47 -04:00
NumCopies.hs make query commands not output extraneous messages 2017-02-16 13:24:35 -04:00
P2P.hs rename to avoid name conflict 2017-05-11 18:31:14 -04:00
PostReceive.hs configuration to disable automatic merge conflict resolution 2017-06-01 12:51:01 -04:00
PreCommit.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Proxy.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
ReadPresentKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
RecvKey.hs get, move, copy, mirror: Added --failed switch which retries failed copies/moves 2016-08-03 12:37:12 -04:00
RegisterUrl.hs Make all --batch input, as well as fromkey and registerurl stdin be processed without requiring it to be in the current encoding. 2016-12-13 15:35:04 -04:00
Reinit.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Reinject.hs annex.securehashesonly 2017-02-27 13:33:59 -04:00
ReKey.hs rekey --force: Incorrectly marked the new key's content as being present in the local repo even when it was not. 2016-12-19 18:18:57 -04:00
RemoteDaemon.hs remotedaemon: serve tor hidden service 2016-11-20 15:48:12 -04:00
Repair.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Required.hs started converting to use optparse-applicative 2015-07-08 13:36:25 -04:00
ResolveMerge.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
RmUrl.hs rekey: Added --batch mode. 2016-12-05 12:55:50 -04:00
Schedule.hs make query commands not output extraneous messages 2017-02-16 13:24:35 -04:00
Semitrust.hs convert all commands to work with optparse-applicative 2015-07-08 15:08:02 -04:00
SendKey.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
SetKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
SetPresentKey.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Smudge.hs add KeyVariety type 2017-02-24 15:16:56 -04:00
Status.hs status: Propigate nonzero exit code from git status. 2017-03-02 14:09:42 -04:00
Sync.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Test.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
TestRemote.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
TransferInfo.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
TransferKey.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
TransferKeys.hs AssociatedFile newtype 2017-03-10 13:35:31 -04:00
Trust.hs trust level overridden message adjusted for forced untrusted export remotes 2017-09-13 12:08:46 -04:00
Unannex.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Undo.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Ungroup.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Uninit.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Unlock.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00
Untrust.hs convert all commands to work with optparse-applicative 2015-07-08 15:08:02 -04:00
Unused.hs unused: When large files are checked right into git, avoid buffering their contents in memory. 2017-01-31 19:09:37 -04:00
Upgrade.hs When auto-upgrading a v3 remote, avoid upgrading to version 6, instead keep it at version 5. 2016-10-05 16:23:09 -04:00
VAdd.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
VCycle.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Version.hs version: Added "dependency versions" line. 2017-04-07 18:16:11 -04:00
VFilter.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Vicfg.hs make git annex config settings editable in vicfg 2017-01-30 17:08:05 -04:00
View.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
VPop.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Wanted.hs make query commands not output extraneous messages 2017-02-16 13:24:35 -04:00
Watch.hs remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
WebApp.hs Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors. 2016-11-15 21:29:54 -04:00
Whereis.hs Avoid repeated checking that files passed on the command line exist. 2017-10-16 14:10:20 -04:00