mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc
History
Joey Hess 579d9b60c1
improve concurrency of move/copy --from --to 
Use separate stages for download and upload. In the common case where
it downloads the file from one remote and then uploads to the other,
those are by far the most expensive operations, and there's a decent
chance the two remotes bottleneck on different resources.

Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads
will be started both downloading from the src remote. They will probably
finish at the same time. Then two threads will be started uploading to
the dst remote. They will probably take the same time as well. Before
this change, it would alternate back and forth, bottlenecking on src and dst.
With this change, as soon as the two threads start uploading to dst, two
more threads are able to start, downloading from src. So bandwidth to
both remotes is saturated more often.

Other commands that use transferStages only send in one direction at a
time. So the worker threads for the other direction will sit idle, and
there will be no change in their behavior.

Sponsored-by: Dartmouth College's DANDI project
2023-01-24 13:59:39 -04:00
..
Android
android
assistant
automatic_conflict_resolution
backends
bare_repositories
bugs Added a comment 2023-01-19 16:28:19 +00:00
chunking
coding_style
contribute
copies
design Added a comment: xxHash as the backend 2022-12-12 08:21:35 +00:00
devblog comment 2022-09-13 14:17:01 -04:00
direct_mode
download
encryption
footer
forum comment 2023-01-16 15:15:21 -04:00
future_proofing
git-annex-add
git-annex-addurl response 2023-01-16 15:13:00 -04:00
git-annex-adjust
git-annex-benchmark comment 2022-12-20 14:30:18 -04:00
git-annex-checkpresentkey
git-annex-config
git-annex-copy
git-annex-direct
git-annex-drop
git-annex-enableremote
git-annex-find findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-annex-import
git-annex-importfeed
git-annex-lookupkey
git-annex-metadata
git-annex-move
git-annex-p2p
git-annex-preferred-content Support "inbackend" in preferred content expressions 2022-09-26 16:06:49 -04:00
git-annex-reinject
git-annex-repair Added a comment: When a weird 'no space left on device' error appears 2022-12-21 16:46:44 +00:00
git-annex-rmurl
git-annex-setpresentkey
git-annex-sync
git-annex-test
git-annex-unannex
git-annex-undo
git-annex-uninit
git-annex-unlock
git-annex-unused
git-annex-whereis
how_it_works
install Added a comment: ipfs 2022-10-31 13:36:51 +00:00
internals convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license
links
metadata
news add news item for git-annex 10.20221212 2022-12-12 12:52:20 -04:00
not
polls
preferred_content
profiling
projects move old closed bugs to datalad's bugs 2023-01-05 15:06:56 -04:00
required_content
scalability
security
special_remotes respect urlinclude/urlexclude of other web special remotes 2023-01-10 14:58:53 -04:00
stickers
submodules
sync
templates
testimonials
thanks update 2023-01-07 13:56:23 -04:00
tips added an optional cost= configuration to all special remotes 2023-01-12 13:42:28 -04:00
todo update 2023-01-23 18:08:55 -04:00
trust
tuning
upgrades
use_case
users introduce myself 2022-12-07 14:43:00 +00:00
videos
walkthrough
workflow
Android.mdwn
assistant.mdwn
automatic_conflict_resolution.mdwn
backends.mdwn fix typo 2022-12-26 12:36:56 -04:00
bare_repositories.mdwn
bugs.mdwn
builds.mdwn
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn remove calendar 2023-01-05 14:50:26 -04:00
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn
encryption.mdwn
favicon.ico
forum.mdwn
future_proofing.mdwn
git-annex-add.mdwn
git-annex-addunused.mdwn
git-annex-addurl.mdwn
git-annex-adjust.mdwn
git-annex-assistant.mdwn
git-annex-backends.mdwn improve documentation about backends 2022-09-26 15:59:10 -04:00
git-annex-benchmark.mdwn
git-annex-calckey.mdwn
git-annex-checkpresentkey.mdwn
git-annex-common-options.mdwn Improved handling of --time-limit when combined with -J 2022-09-22 12:54:52 -04:00
git-annex-config.mdwn
git-annex-contentlocation.mdwn
git-annex-copy.mdwn improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
git-annex-dead.mdwn fix reversion on skipping dead keys in --all/bare 2022-09-13 14:38:13 -04:00
git-annex-describe.mdwn
git-annex-diffdriver.mdwn
git-annex-direct.mdwn
git-annex-drop.mdwn Fix typo in heading that seems to result in git-annex-drop not showing up in man -k output 2023-01-22 15:25:35 +00:00
git-annex-dropkey.mdwn
git-annex-dropunused.mdwn
git-annex-edit.mdwn
git-annex-enable-tor.mdwn
git-annex-enableremote.mdwn
git-annex-examinekey.mdwn
git-annex-expire.mdwn
git-annex-export.mdwn
git-annex-filter-branch.mdwn
git-annex-filter-process.mdwn
git-annex-find.mdwn findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-annex-findkeys.mdwn findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-annex-findref.mdwn
git-annex-fix.mdwn
git-annex-forget.mdwn
git-annex-fromkey.mdwn
git-annex-fsck.mdwn
git-annex-fuzztest.mdwn
git-annex-get.mdwn
git-annex-group.mdwn
git-annex-groupwanted.mdwn
git-annex-import.mdwn
git-annex-importfeed.mdwn
git-annex-indirect.mdwn
git-annex-info.mdwn
git-annex-init.mdwn
git-annex-initremote.mdwn added an optional cost= configuration to all special remotes 2023-01-12 13:42:28 -04:00
git-annex-inprogress.mdwn
git-annex-list.mdwn
git-annex-lock.mdwn
git-annex-log.mdwn improve wording 2022-09-13 15:28:57 -04:00
git-annex-lookupkey.mdwn
git-annex-map.mdwn
git-annex-matchexpression.mdwn
git-annex-matching-expression.mdwn
git-annex-matching-options.mdwn --anything and --nothing 2022-12-20 15:44:09 -04:00
git-annex-merge.mdwn
git-annex-metadata.mdwn
git-annex-migrate.mdwn improve documentation about backends 2022-09-26 15:59:10 -04:00
git-annex-mincopies.mdwn
git-annex-mirror.mdwn
git-annex-move.mdwn improve concurrency of move/copy --from --to 2023-01-24 13:59:39 -04:00
git-annex-multicast.mdwn
git-annex-numcopies.mdwn
git-annex-p2p.mdwn
git-annex-post-receive.mdwn
git-annex-pre-commit.mdwn
git-annex-preferred-content.mdwn improve documentation about backends 2022-09-26 15:59:10 -04:00
git-annex-proxy.mdwn
git-annex-readpresentkey.mdwn
git-annex-registerurl.mdwn
git-annex-reinit.mdwn
git-annex-reinject.mdwn
git-annex-rekey.mdwn
git-annex-remotedaemon.mdwn
git-annex-renameremote.mdwn
git-annex-repair.mdwn
git-annex-required.mdwn
git-annex-resolvemerge.mdwn
git-annex-restage.mdwn restage: New git-annex command, handles restaging unlocked files 2022-09-23 16:29:59 -04:00
git-annex-rmurl.mdwn
git-annex-schedule.mdwn
git-annex-semitrust.mdwn
git-annex-setkey.mdwn
git-annex-setpresentkey.mdwn
git-annex-shell.mdwn
git-annex-smudge.mdwn
git-annex-status.mdwn
git-annex-sync.mdwn document that sync --cleanup only deletes branches 2022-11-14 13:00:22 -04:00
git-annex-test.mdwn test: Add --test-debug option 2022-11-28 15:12:53 -04:00
git-annex-testremote.mdwn
git-annex-transferkey.mdwn
git-annex-transferkeys.mdwn
git-annex-transferrer.mdwn
git-annex-trust.mdwn
git-annex-unannex.mdwn
git-annex-undo.mdwn
git-annex-ungroup.mdwn
git-annex-uninit.mdwn
git-annex-unlock.mdwn
git-annex-unregisterurl.mdwn
git-annex-untrust.mdwn
git-annex-unused.mdwn
git-annex-upgrade.mdwn
git-annex-vadd.mdwn
git-annex-vcycle.mdwn
git-annex-version.mdwn
git-annex-vfilter.mdwn
git-annex-vicfg.mdwn
git-annex-view.mdwn
git-annex-vpop.mdwn
git-annex-wanted.mdwn
git-annex-watch.mdwn
git-annex-webapp.mdwn
git-annex-whereis.mdwn mention that whereis --all skips dead keys 2022-09-13 15:29:06 -04:00
git-annex-whereused.mdwn
git-annex.mdwn findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-remote-tor-annex.mdwn
git-union-merge.mdwn
how_it_works.mdwn
index.mdwn
install.mdwn
internals.mdwn convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license.mdwn
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn
polls.mdwn
preferred_content.mdwn
privacy.mdwn
profiling.mdwn
projects.mdwn
publicrepos.mdwn
related_software.mdwn
repomap.png
required_content.mdwn
scalability.mdwn
security.mdwn
shortcuts.mdwn
sidebar.mdwn
sitemap.mdwn
special_remotes.mdwn
stickers.mdwn
submodules.mdwn
summary.mdwn
sync.mdwn
testimonials.mdwn
thanks.mdwn update 2023-01-07 13:56:06 -04:00
tips.mdwn
todo.mdwn
transferring_data.mdwn
trust.mdwn
tuning.mdwn
upgrades.mdwn
users.mdwn
videos.mdwn
walkthrough.mdwn
workflow.mdwn