mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/Command
History
Joey Hess 579d9b60c1
 improve concurrency of move/copy --from --to 
Use separate stages for download and upload. In the common case where
it downloads the file from one remote and then uploads to the other,
those are by far the most expensive operations, and there's a decent
chance the two remotes bottleneck on different resources.

Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads
will be started both downloading from the src remote. They will probably
finish at the same time. Then two threads will be started uploading to
the dst remote. They will probably take the same time as well. Before
this change, it would alternate back and forth, bottlenecking on src and dst.
With this change, as soon as the two threads start uploading to dst, two
more threads are able to start, downloading from src. So bandwidth to
both remotes is saturated more often.

Other commands that use transferStages only send in one direction at a
time. So the worker threads for the other direction will sit idle, and
there will be no change in their behavior.

Sponsored-by: Dartmouth College's DANDI project
2023-01-24 13:59:39 -04:00
..
Add.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
AddUnused.hs get rid of racy addLink 2022-06-14 14:47:15 -04:00
AddUrl.hs convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
Adjust.hs rename to --unlock-present and better reverse adjusting 2020-11-13 14:56:43 -04:00
Assistant.hs avoid redundant prompt for http password in git-annex get that does autoinit 2022-09-09 14:43:43 -04:00
Benchmark.hs improve benchmark --databases 2019-11-21 17:25:20 -04:00
CalcKey.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
CheckPresentKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
Config.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
ConfigList.hs don't frontload reconcileStaged in git-annex init 2022-11-18 13:58:47 -04:00
ContentLocation.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Copy.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Dead.hs improve usage 2022-10-03 13:49:42 -04:00
Describe.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
DiffDriver.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Direct.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
Drop.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
DropKey.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
DropUnused.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
EnableRemote.hs enableremote: Better handling of the unusual case where multiple special remotes have been initialized with the same name 2022-01-05 15:12:11 -04:00
EnableTor.hs Make git-annex enable-tor work when using the linux standalone build 2022-10-26 15:45:08 -04:00
ExamineKey.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Expire.hs Future proof activity log parsing 2021-06-14 14:18:19 -04:00
Export.hs export: fix multi-file delete bug 2022-11-09 16:24:37 -04:00
FilterBranch.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
FilterProcess.hs smudge: Warn when encountering a pointer file that has other content appended to it 2022-02-23 15:17:08 -04:00
Find.hs findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
FindKeys.hs findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
FindRef.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Fix.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Forget.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
FromKey.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Fsck.hs fix reversion on skipping dead keys in --all/bare 2022-09-13 14:38:13 -04:00
FuzzTest.hs avoid unix-compat's rename 2022-07-12 14:55:02 -04:00
GCryptSetup.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Get.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Group.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
GroupWanted.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Help.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Import.hs add --dry-run: New option 2022-08-03 11:16:04 -04:00
ImportFeed.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
InAnnex.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Indirect.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
Info.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
Init.hs don't frontload reconcileStaged in git-annex init 2022-11-18 13:58:47 -04:00
InitRemote.hs renameremote: Better handling of case where there are multiple special remotes with a name 2022-01-05 15:24:02 -04:00
Inprogress.hs more RawFilePath conversion 2020-11-02 16:31:28 -04:00
List.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Lock.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Log.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
LookupKey.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Map.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
MatchExpression.hs split ProvidedInfo and UserProvidedInfo 2020-09-28 12:12:38 -04:00
Merge.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
MetaData.hs use lookupKeyStaged in --batch code paths 2022-10-26 14:43:06 -04:00
Migrate.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
MinCopies.hs prevent numcopies or mincopies being configured to 0 2022-03-28 15:20:34 -04:00
Mirror.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Move.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Multicast.hs avoid unix-compat's rename 2022-07-12 14:55:02 -04:00
NotifyChanges.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
NumCopies.hs prevent numcopies or mincopies being configured to 0 2022-03-28 15:20:34 -04:00
P2P.hs complete the magic wormhole pairin appid transition 2021-12-30 12:16:22 -04:00
P2PStdIO.hs finish CommandStart transition 2019-06-12 13:24:01 -04:00
PostReceive.hs remove direct mode remnant of merging unrelated histories 2021-07-19 11:41:26 -04:00
PreCommit.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Proxy.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
ReadPresentKey.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
RecvKey.hs avoid flushing keys db queue after each Annex action 2022-10-12 14:12:23 -04:00
RegisterUrl.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Reinit.hs don't frontload reconcileStaged in git-annex init 2022-11-18 13:58:47 -04:00
Reinject.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
ReKey.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
RemoteDaemon.hs assistant: Fix a crash on startup by avoiding using forkProcess 2021-05-12 15:08:03 -04:00
RenameRemote.hs renameremote: Better handling of case where there are multiple special remotes with a name 2022-01-05 15:24:02 -04:00
Repair.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
Required.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
ResolveMerge.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Restage.hs restage: New git-annex command, handles restaging unlocked files 2022-09-23 16:29:59 -04:00
RmUrl.hs use lookupKeyStaged in --batch code paths 2022-10-26 14:43:06 -04:00
Schedule.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Semitrust.hs improve usage 2022-10-03 13:49:42 -04:00
SendKey.hs remove git-annex-shell compat code 2021-10-11 15:36:51 -04:00
SetKey.hs use RawFilePath version of rename 2022-06-22 16:47:34 -04:00
SetPresentKey.hs reject batch options combined with non-batch options 2022-01-26 13:00:19 -04:00
Smudge.hs smudge: Warn when encountering a pointer file that has other content appended to it 2022-02-23 15:17:08 -04:00
Status.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Sync.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Test.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
TestRemote.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
TransferKey.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
TransferKeys.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
Transferrer.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
Trust.hs improve usage 2022-10-03 13:49:42 -04:00
Unannex.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Undo.hs finished this stage of the RawFilePath conversion 2020-11-04 14:20:37 -04:00
Ungroup.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Uninit.hs remove whenAnnexed and ifAnnexed 2022-10-26 14:06:32 -04:00
Unlock.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
UnregisterUrl.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
Untrust.hs improve usage 2022-10-03 13:49:42 -04:00
Unused.hs move several readonly values to AnnexRead 2022-06-28 15:40:19 -04:00
Upgrade.hs don't frontload reconcileStaged in git-annex init 2022-11-18 13:58:47 -04:00
VAdd.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
VCycle.hs restage: New git-annex command, handles restaging unlocked files 2022-09-23 16:29:59 -04:00
Version.hs fix spelling of upgradeable 2022-01-19 12:14:50 -04:00
VFilter.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Vicfg.hs vicfg: Include mincopies configuration 2022-09-15 15:11:59 -04:00
View.hs more RawFilePath conversion 2020-10-30 15:55:59 -04:00
VPop.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Wanted.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Watch.hs remotedaemon: Don't list --stop in help since it's not supported. 2019-09-30 14:40:46 -04:00
WebApp.hs Make git-annex enable-tor work when using the linux standalone build 2022-10-26 15:45:08 -04:00
Whereis.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
WhereUsed.hs remove --backend from global options 2022-06-29 13:33:25 -04:00