Joey Hess 68a49adcda Improve behavior when -J transfers multiple files that point to the same key ... After a false start, I found a fairly non-intrusive way to deal with it. Although it only handles transfers -- there may be issues with eg concurrent dropping of the same key, or other operations. There is no added overhead when -J is not used, other than an added inAnnex check. When -J is used, it has to maintain and check a small Set, which should be negligible overhead. It could output some message saying that the transfer is being done by another thread. Or it could even display the same progress info for both files that are being downloaded since they have the same content. But I opted to keep it simple, since this is rather an edge case, so it just doesn't say anything about the transfer of the file until the other thread finishes. Since the deferred transfer action still runs, actions that do more than transfer content will still get a chance to do their other work. (An example of something that needs to do such other work is P2P.Annex, where the download always needs to receive the content from the peer.) And, if the first thread fails to complete a transfer, the second thread can resume it. But, this unfortunately means that there's a risk of redundant work being done to transfer a key that just got transferred. That's not ideal, but should never cause breakage; the same thing can occur when running two separate git-annex processes. The get/move/copy/mirror --from commands had extra inAnnex checks added, inside the download actions. Without those checks, the first thread downloaded the content, and then the second thread woke up and downloaded the same content redundantly. move/copy/mirror --to is left doing redundant uploads for now. It would need a second checkPresent of the remote inside the upload to avoid them, which would be expensive. A better way to avoid redundant work needs to be found.. This commit was supported by the NSF-funded DataLad project.		2017-10-17 17:10:50 -04:00
..
Android
android
assistant	wording	2016-12-28 12:38:34 -04:00
automatic_conflict_resolution
backends
bare_repositories
bugs	Improve behavior when -J transfers multiple files that point to the same key	2017-10-17 17:10:50 -04:00
coding_style
contact
contribute
copies
design	poll vote (/sdcard/annex)	2017-10-16 21:50:31 +00:00
devblog	devblog	2017-09-20 15:36:39 -04:00
direct_mode
download
ekg
encryption	initremote, enableremote: Support gpg subkeys suffixed with an exclamation mark, which forces gpg to use a specific subkey.	2017-05-24 14:08:02 -04:00
footer
forum		2017-10-17 10:17:32 +00:00
future_proofing
git-annex-copy
git-annex-direct
git-annex-drop	Added a comment: RE: choosing remotes and annex-cost-command	2017-02-28 01:10:05 +00:00
git-annex-enableremote	Added a comment	2017-02-14 20:21:41 +00:00
git-annex-importfeed
git-annex-move	Added a comment: Move from A to B	2017-07-26 19:08:07 +00:00
git-annex-preferred-content
git-annex-reinject
git-annex-unannex
git-annex-undo
git-annex-unused
how_it_works
install	Added a comment: For stretch on armhf, install from unstable	2017-08-28 09:25:22 +00:00
internals	correct spelling mistakes	2017-02-12 17:30:23 -04:00
license
links	add some links to new workflow page	2016-12-20 15:13:44 -04:00
metadata
news	add news item for git-annex 6.20171003	2017-10-03 13:19:25 -04:00
not	Added a comment: Can git-annex be used to quasi-realtime replication?	2017-01-16 17:41:07 +00:00
polls
preferred_content
required_content
scalability
special_remotes	Added a comment: IPFS Deduplication	2017-10-07 19:46:12 +00:00
stickers
sync	response; update man page	2017-05-09 14:02:48 -04:00
templates
testimonials
thanks	update	2017-10-02 14:07:10 -04:00
tips	comment	2017-09-29 13:46:33 -04:00
todo	you requested his old closed bugs not be deleted yet	2017-10-02 11:58:31 -04:00
trust
tuning	response	2017-08-28 13:41:19 -04:00
upgrades	comment	2017-03-21 13:44:42 -04:00
use_case
users	fixing ben	2017-03-17 13:34:36 +00:00
videos	add back share_with_a_friend_walkthrough, adapted for tor pairing	2016-12-24 15:46:02 -04:00
walkthrough	remove recent addition of receive.denyNonFastForwards to walkthrough	2017-02-13 16:02:53 -04:00
Android.mdwn
assistant.mdwn	fix markup	2016-12-28 12:34:42 -04:00
automatic_conflict_resolution.mdwn	expand docs	2017-06-01 11:56:56 -04:00
backends.mdwn	annex.backend is the new name for what was annex.backends	2017-05-09 15:04:07 -04:00
bare_repositories.mdwn
bugs.mdwn	limit rss/atom feeds to 10 pages to avoid enormous files	2017-09-29 12:42:09 -04:00
builds.mdwn	windows autobuilder no longer firewalled	2017-10-16 15:07:51 -04:00
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn	github mirror has been removed due to their horrible new anti-free-software TOS	2017-03-01 13:28:02 -04:00
ekg.mdwn
encryption.mdwn
favicon.ico
forum.mdwn	limit rss/atom feeds to 10 pages to avoid enormous files	2017-09-29 12:42:09 -04:00
future_proofing.mdwn
git-annex-add.mdwn	git annex add -u now supported, analagous to git add -u	2017-04-07 15:55:45 -04:00
git-annex-addunused.mdwn
git-annex-addurl.mdwn
git-annex-adjust.mdwn	more consistent backticks with other man pages	2017-05-24 14:09:14 -04:00
git-annex-assistant.mdwn
git-annex-calckey.mdwn	annex.backend is the new name for what was annex.backends	2017-05-09 15:04:07 -04:00
git-annex-checkpresentkey.mdwn
git-annex-config.mdwn	configuration to disable automatic merge conflict resolution	2017-06-01 12:51:01 -04:00
git-annex-contentlocation.mdwn
git-annex-copy.mdwn	move, copy: Support --batch.	2017-08-15 12:39:10 -04:00
git-annex-dead.mdwn	Keys marked as dead are now skipped by --all.	2017-05-09 12:55:21 -04:00
git-annex-describe.mdwn
git-annex-diffdriver.mdwn
git-annex-direct.mdwn
git-annex-drop.mdwn
git-annex-dropkey.mdwn
git-annex-dropunused.mdwn
git-annex-edit.mdwn
git-annex-enable-tor.mdwn	enable-tor: No longer needs to be run as root.	2016-12-20 17:40:36 -04:00
git-annex-enableremote.mdwn
git-annex-examinekey.mdwn
git-annex-expire.mdwn
git-annex-export.mdwn	export --fast sets up but does not populate export	2017-09-19 14:26:03 -04:00
git-annex-find.mdwn
git-annex-findref.mdwn
git-annex-fix.mdwn
git-annex-forget.mdwn
git-annex-fromkey.mdwn
git-annex-fsck.mdwn	fsck: Support --json.	2017-06-26 13:40:57 -04:00
git-annex-fuzztest.mdwn
git-annex-get.mdwn	document get -J's behavior of spreading load amoung same-cost remotes	2017-03-08 14:26:29 -04:00
git-annex-group.mdwn	The second argument of "git annex group" is optional, fix the man page	2017-05-31 12:09:15 +02:00
git-annex-groupwanted.mdwn
git-annex-import.mdwn	documentation for export	2017-08-29 13:25:48 -04:00
git-annex-importfeed.mdwn
git-annex-indirect.mdwn
git-annex-info.mdwn
git-annex-init.mdwn
git-annex-initremote.mdwn	initremote: When a uuid= parameter is passed, use the specified UUID for the new special remote, instead of generating a UUID.	2017-02-07 15:10:41 -04:00
git-annex-list.mdwn
git-annex-lock.mdwn
git-annex-log.mdwn
git-annex-lookupkey.mdwn
git-annex-map.mdwn
git-annex-matchexpression.mdwn
git-annex-matching-options.mdwn	securehash matching	2017-02-27 15:02:44 -04:00
git-annex-merge.mdwn	configuration to disable automatic merge conflict resolution	2017-06-01 12:51:01 -04:00
git-annex-metadata.mdwn	metadata: Added --remove-all.	2017-09-28 12:36:10 -04:00
git-annex-migrate.mdwn
git-annex-mirror.mdwn
git-annex-move.mdwn	move, copy: Support --batch.	2017-08-15 12:39:10 -04:00
git-annex-multicast.mdwn	multicast: New command, uses uftp to multicast annexed files, for eg a classroom setting.	2017-03-30 19:35:30 -04:00
git-annex-numcopies.mdwn	vicfg: Include the numcopies configuation.	2017-01-30 15:27:25 -04:00
git-annex-p2p.mdwn
git-annex-post-receive.mdwn	post-recive hook to make updateInstead work in direct mode and adjusted branches	2017-02-17 14:04:43 -04:00
git-annex-pre-commit.mdwn
git-annex-preferred-content.mdwn	securehash matching	2017-02-27 15:02:44 -04:00
git-annex-proxy.mdwn
git-annex-readpresentkey.mdwn
git-annex-registerurl.mdwn
git-annex-reinit.mdwn	reusing repository uuid cannot result in data loss AFAIK	2017-01-30 13:18:50 -04:00
git-annex-reinject.mdwn
git-annex-rekey.mdwn
git-annex-remotedaemon.mdwn	Added remote.<name>.annex-push and remote.<name>.annex-pull	2017-04-05 13:22:35 -04:00
git-annex-repair.mdwn
git-annex-required.mdwn
git-annex-resolvemerge.mdwn	expand docs	2017-06-01 11:56:56 -04:00
git-annex-rmurl.mdwn
git-annex-schedule.mdwn
git-annex-semitrust.mdwn
git-annex-setkey.mdwn
git-annex-setpresentkey.mdwn
git-annex-shell.mdwn	correct spelling mistake	2017-02-11 09:14:27 +00:00
git-annex-smudge.mdwn
git-annex-status.mdwn	status: Pass --ignore-submodules=when option on to git status.	2017-02-20 17:01:24 -04:00
git-annex-sync.mdwn	sync: Added --cleanup, which removes local and remote synced/ branches.	2017-09-28 14:58:48 -04:00
git-annex-test.mdwn
git-annex-testremote.mdwn
git-annex-transferkey.mdwn
git-annex-transferkeys.mdwn
git-annex-trust.mdwn
git-annex-unannex.mdwn
git-annex-undo.mdwn
git-annex-ungroup.mdwn
git-annex-uninit.mdwn
git-annex-unlock.mdwn
git-annex-untrust.mdwn
git-annex-unused.mdwn
git-annex-upgrade.mdwn
git-annex-vadd.mdwn
git-annex-vcycle.mdwn
git-annex-version.mdwn
git-annex-vfilter.mdwn
git-annex-vicfg.mdwn	vicfg: Include the numcopies configuation.	2017-01-30 15:27:25 -04:00
git-annex-view.mdwn
git-annex-vpop.mdwn	expand manpages cross-references significantly	2015-05-29 12:12:55 -04:00
git-annex-wanted.mdwn
git-annex-watch.mdwn
git-annex-webapp.mdwn
git-annex-whereis.mdwn
git-annex.mdwn	configuration and docs for tracking exports	2017-09-19 13:05:43 -04:00
git-remote-tor-annex.mdwn
git-union-merge.mdwn
how_it_works.mdwn	add some links to new workflow page	2016-12-20 15:13:44 -04:00
index.mdwn
install.mdwn	remove links to trees of files, as they confuse users and prevent them reading the instructions before downloading	2017-08-28 13:05:03 -04:00
internals.mdwn	change export.log format to support multiple export remotes	2017-09-12 17:45:52 -04:00
license.mdwn
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn	Fix markup that caused broken link.	2017-10-03 02:52:24 +00:00
polls.mdwn
preferred_content.mdwn	securehash matching	2017-02-27 15:02:44 -04:00
privacy.mdwn
publicrepos.mdwn	Added datalad "super-dataset".	2017-09-05 17:00:38 +00:00
related_software.mdwn
repomap.png
required_content.mdwn
scalability.mdwn
shortcuts.mdwn
sidebar.mdwn
sitemap.mdwn
special_remotes.mdwn	add link to git-annex-remote-gvfs for smb / sftp	2017-09-16 13:57:37 -04:00
stickers.mdwn
submodules.mdwn
summary.mdwn
sync.mdwn	add some links to new workflow page	2016-12-20 15:13:44 -04:00
testimonials.mdwn
thanks.mdwn	name update	2017-07-06 09:33:43 -04:00
tips.mdwn
todo.mdwn	limit rss/atom feeds to 10 pages to avoid enormous files	2017-09-29 12:42:09 -04:00
transferring_data.mdwn
trust.mdwn
tuning.mdwn
upgrades.mdwn
users.mdwn
videos.mdwn
Void.mdwn
walkthrough.mdwn	remove recent addition of receive.denyNonFastForwards to walkthrough	2017-02-13 16:02:53 -04:00
workflow.mdwn	link to the assistant page here	2017-04-24 13:56:08 +00:00