mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/Types
History
Joey Hess 579d9b60c1
improve concurrency of move/copy --from --to 
Use separate stages for download and upload. In the common case where
it downloads the file from one remote and then uploads to the other,
those are by far the most expensive operations, and there's a decent
chance the two remotes bottleneck on different resources.

Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads
will be started both downloading from the src remote. They will probably
finish at the same time. Then two threads will be started uploading to
the dst remote. They will probably take the same time as well. Before
this change, it would alternate back and forth, bottlenecking on src and dst.
With this change, as soon as the two threads start uploading to dst, two
more threads are able to start, downloading from src. So bandwidth to
both remotes is saturated more often.

Other commands that use transferStages only send in one direction at a
time. So the worker threads for the other direction will sit idle, and
there will be no change in their behavior.

Sponsored-by: Dartmouth College's DANDI project
2023-01-24 13:59:39 -04:00
..
ActionItem.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
AdjustedBranch.hs make sync update --unlock-present branch 2020-11-13 15:04:34 -04:00
Availability.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Backend.hs refactor 2021-08-18 13:19:02 -04:00
Benchmark.hs benchmark: Add --databases to benchmark sqlite databases 2019-10-29 16:59:27 -04:00
BranchState.hs handle transitions with read-only unmerged git-annex branches 2021-12-28 13:23:32 -04:00
CatFileHandles.hs separate handles for cat-file and cat-file --batch-check 2021-09-24 13:16:13 -04:00
CleanupActions.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
Command.hs add --dry-run: New option 2022-08-03 11:16:04 -04:00
Concurrency.hs have setConcurrency stop any running git coprocesses 2021-11-19 12:00:39 -04:00
Creds.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Crypto.hs remove Show Cipher 2020-09-01 18:11:22 -04:00
DeferredParse.hs remove --backend from global options 2022-06-29 13:33:25 -04:00
DesktopNotify.hs remove many old version ifdefs 2019-07-05 15:09:37 -04:00
Difference.hs annex.addunlocked expressions 2019-12-20 15:56:25 -04:00
Direction.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Distribution.hs annex.addunlocked expressions 2019-12-20 15:56:25 -04:00
Export.hs Fix a build failure with ghc 9.2.2 2022-05-02 14:21:48 -04:00
FileMatcher.hs fromkey unlocked files support 2021-05-03 11:26:18 -04:00
GitConfig.hs Fix annex.adviceNoSshCaching having no effect 2022-09-30 14:03:06 -04:00
Group.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Import.hs ImportableContentsChunkable 2021-10-08 13:15:22 -04:00
IndexFiles.hs cache annex index filename for 1.5% speedup to queries 2020-04-10 13:37:04 -04:00
Key.hs Typo fix unncessary -> unnecessary. 2022-08-20 09:40:19 -04:00
KeySource.hs convert KeySource to RawFilePath 2020-02-21 10:04:44 -04:00
Link.hs prep for fixing find --branch --unlocked 2021-03-02 13:39:31 -04:00
LockCache.hs more RawFilePath conversion 2020-10-29 10:50:29 -04:00
Messages.hs display scanning message whenever reconcileStaged has enough files to chew on 2021-06-08 12:48:30 -04:00
MetaData.hs factor out Utility.Aeson.textKey 2022-03-02 18:24:06 -04:00
Mime.hs Added mimeencoding= term to annex.largefiles expressions. 2019-04-30 12:17:22 -04:00
NumCopies.hs prevent numcopies or mincopies being configured to 0 2022-03-28 15:20:34 -04:00
ProposedAccepted.hs separate RemoteConfig parsing basically working 2020-01-14 12:35:08 -04:00
RefSpec.hs Windows: include= and exclude= containing '/' will also match filenames that are written using '\' 2020-12-15 12:39:34 -04:00
Remote.hs all keys are still present on versioned remote after import of a tree 2022-10-11 13:05:40 -04:00
RemoteConfig.hs GETCONFIG name fix 2020-03-09 12:38:04 -04:00
RemoteState.hs improve comment 2020-01-06 12:53:09 -04:00
RepoVersion.hs v7 2018-10-25 18:24:23 -04:00
ScheduledActivity.hs avoid using MonadFail in ParseDuration 2020-08-15 15:53:35 -04:00
StallDetection.hs bwlimit 2021-09-21 16:58:10 -04:00
StandardGroups.hs a few forgotten remote config fields 2020-01-15 11:22:36 -04:00
StoreRetrieve.hs refactor 2021-08-18 13:19:02 -04:00
Test.hs test: Add --test-debug option 2022-11-28 15:12:53 -04:00
Transfer.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
Transferrer.hs simplify transferr protocol 2020-12-11 12:52:22 -04:00
TransferrerPool.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
Transitions.hs handle transitions with read-only unmerged git-annex branches 2021-12-28 13:23:32 -04:00
TrustLevel.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Upgrade.hs split upgrade into v9 and v10 2022-01-19 13:09:33 -04:00
UrlContents.hs remove SafeFilePath 2020-05-11 14:04:56 -04:00
UUID.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
VectorClock.hs deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
View.hs test: Fix some test cases that assumed git's default branch name 2020-06-23 16:40:51 -04:00
WorkerPool.hs improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00