git-annex/doc/design
Joey Hess 6ecd55a9fa
Fixed some other potential hangs in the P2P protocol
Finishes the start made in 983c9d5a53, by
handling the case where `transfer` fails for some other reason, and so the
ReadContent callback does not get run. I don't know of a case where
`transfer` does fail other than the locking dealt with in that commit, but
it's good to have a guarantee.

StoreContent and StoreContentTo had a similar problem.
Things like `getViaTmp` may decide not to run the transfer action.
And `transfer` could certianly fail, if another transfer of the same
object was in progress. (Or a different object when annex.pidlock is set.)

If the transfer action was not run, the content of the object would
not all get consumed, and so would get interpreted as protocol commands,
which would not go well.

My approach to fixing all of these things is to set a TVar only
once all the data in the transfer is known to have been read/written.
This way the internals of `transfer`, `getViaTmp` etc don't matter.

So in ReadContent, it checks if the transfer completed.
If not, as long as it didn't throw an exception, send empty and Invalid
data to the callback. On an exception the state of the protocol is unknown
so it has to raise ProtoFailureException and close the connection,
same as before.

In StoreContent, if the transfer did not complete
some portion of the DATA has been read, so the protocol is in an unknown
state and it has to close the conection as well.

(The ProtoFailureMessage used here matches the one in Annex.Transfer, which
is the most likely reason. Not ideal to duplicate it..)

StoreContent did not ever close the protocol connection before. So this is
a protocol change, but only in an exceptional circumstance, and it's not
going to break anything, because clients already need to deal with the
connection breaking at any point.

The way this new behavior looks (here origin has annex.pidlock = true so will
only accept one upload to it at a time):

git annex copy --to origin -J2
copy x (to origin...) ok
copy y (to origin...)
  Lost connection (fd:25: hGetChar: end of file)

This work is supported by the NIH-funded NICEMAN (ReproNim TR&D3) project.
2018-11-06 14:52:32 -04:00
..
adjusted_branches Added a comment: adjusted branche to "focus" on a specific subtree 2016-08-22 14:19:57 +00:00
assistant close this old poll 2018-09-11 13:29:19 -04:00
encryption
exporting_trees_to_special_remotes Added a comment 2018-02-07 20:01:53 +00:00
external_special_remote_protocol response 2018-10-29 15:40:52 -04:00
git-remote-daemon Added a comment: Rolling hash chunking 2014-04-04 14:16:25 +00:00
iabackup Added a comment: 14 of 21PB, actually 2015-04-30 02:58:05 +00:00
metadata followup 2015-04-09 14:33:11 -04:00
new_repo_versions devblog 2016-05-04 14:39:53 -04:00
p2p_protocol response 2018-03-12 18:32:58 -04:00
requests_routing Added a comment: design phase only 2015-08-18 19:02:30 +00:00
adjusted_branches.mdwn link to the adjust manpage 2016-06-23 14:39:49 +00:00
assistant.mdwn clarify that this is mostly done (i think?) 2014-04-07 04:41:56 +00:00
balanced_preferred_content.mdwn Fix a typo 2016-02-08 10:54:08 -04:00
caching_database.mdwn correct spelling mistakes 2017-02-12 17:30:23 -04:00
encryption.mdwn clean up 2015-12-11 11:03:22 -04:00
exporting_trees_to_special_remotes.mdwn update 2017-09-18 14:51:32 -04:00
external_special_remote_protocol.mdwn add GETINFO to external protocol (for ronnypfa) 2018-06-08 11:56:24 -04:00
gcrypt.mdwn automatically derive an annex-uuid from a gcrypt-uuids 2013-09-05 16:02:39 -04:00
git-remote-daemon.mdwn update 2015-01-15 15:58:56 -04:00
iabackup.mdwn Fix spelling in doc/design/iabackup.mdwn 2018-06-03 12:28:26 +00:00
metadata.mdwn update for v6 unlocked files 2015-12-26 14:59:06 -04:00
new_repo_versions.mdwn comment 2016-04-20 14:24:26 -04:00
p2p_protocol.mdwn Fixed some other potential hangs in the P2P protocol 2018-11-06 14:52:32 -04:00
preferred_content.mdwn preferred content stability analysis 2014-01-22 15:55:44 -04:00
requests_routing.mdwn keep track of satisfied requests, and summarize 2014-05-09 16:41:05 -03:00
roadmap.mdwn updates 2016-12-13 14:35:58 -04:00