mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

external protocol VERSION 2

Browse source 
Support VERSION 2 in the external special remote protocol, which is
identical to VERSION 1, but avoids external remote programs neededing to
work around the above bug. External remote program that support
exporttree=yes are recommended to be updated to send VERSION 2.

Sponsored-by: Kevin Mueller on Patreon
This commit is contained in:
Joey Hess 2023-03-28 17:00:08 -04:00
parent 82c65b7951
commit 18d326cb6f
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
11 changed files with 80 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

44
doc/bugs/external_remote_export_sent_to_wrong_process/comment_1_831bdc33451de9ef5c54592a7455683f._comment
View file

@ -18,17 +18,7 @@ but am confident that I fixed it in
----
I'm looking forward to export support in rclone!
I'm wondering about the data loss part of this. You said:
> As a result, one process receives two `EXPORT`s in a row, the first of which it ignores,
> while some other process receives a `TRANSFEREXPORT` without a prior `EXPORT`,
> reusing whatever filename was set in the previous transaction,
> and either ovewriting the last accessed remote file with the wrong content,
> or retrieving the content of the last accessed remote file instead of the one git-annex wanted.
So in your implementation, you're keeping EXPORT set after handling TRANSFEREXPORT
In your implementation, you're keeping EXPORT set after handling TRANSFEREXPORT
and similar commands, and so if you receive a TRANSFEREXPORT not prefixed by an
EXPORT, it can be bad. A natural way to write things, indeed
`doc/special_remotes/external/example.sh` does the same.
@ -37,26 +27,20 @@ An alternative implementation would be to clear the EXPORT after handling a
TRANSFEREXPORT, CHECKPRESENTEXPORT, REMOVEEXPORT, or RENAMEEXPORT.
And have those error out if no EXPORT was received before them.
So that's one way we can avoid the data loss problem if your external remote
is used with an old git-annex that has this bug. You might want to do that now.
But, that does not fully avoid the data loss problem. Because this might
happen:
----
EXPORT foo
EXPORT bar
TRANSFEREXPORT STORE $foo_key $foo_file
But, that requires defensive coding in every external remote... Maybe it
would be worth changing the protocol in some way, that avoids the problem.
Then if external remotes were updated to the new protocol, they'd not work
with the buggy git-annex versions, and avoid data loss. I'm leaving this bug
open for now to consider such a protocol change..
So even ignoring the first EXPORT can result in the external special remote
doing the wrong thing. To fully guard against it, you'd have to error out
if multiple EXPORT were received before a request that uses one, rather
than ignoring the first EXPORT.
Weighing the costs and benefits of such a change, the extent of the data
loss is fairly limited. exporttree=yes remotes are always untrusted, so
if a file on one is overwritten with the wrong content, git-annex will be preserving
the right content elsewhere. And if the wrong file is retrieved from one,
git-annex will notice it has the wrong content (so long as its key uses a checksum).
Still, whatever that export is used for would unexpectedly have the wrong file
content, which could still be a bad day for somebody.
(If I decide against the protocol change, I should fix
`doc/special_remotes/external/example.sh` to defend against the bug,
and tell others who have externals that support exporttree too..)
So, fixing this in your remote and others would need significant defensive
coding. Too much to make sense, IMHO. I think git-annex needs to chnge the
protocol in some way instead, to make it easy for you to avoid speaking to
a buggy git-annex.
"""]]

37
doc/bugs/external_remote_export_sent_to_wrong_process/comment_2_8ff0638891c38f423d792f5c3cbfb64f._comment
View file

@ -12,40 +12,5 @@ git-annex could either continue to also support VERSION 1, or it could
refuse to work with externals that don't use VERSION 2. The latter would
force externals to get updated. But then those externals would have no way to work
with old git-annex even if they wanted to. I think forcing an update is not called
for.
Note that git-annex's handling of an external that sends VERSION 2 is not
stellar currently:
joey@darkstar:~/tmp/b>git-annex initremote t type=external externaltype=test encryption=none
initremote t
external special remote protocol error, unexpectedly received "CONFIG directory store data here" (command not allowed at this time)
git-annex: unable to use special remote due to protocol error
joey@darkstar:~/tmp/b>git-annex copy --to t
external special remote protocol error, unexpectedly received "UNSUPPORTED-REQUEST" (command not allowed at this time)
external special remote protocol error, unexpectedly received "UNSUPPORTED-REQUEST" (command not allowed at this time)
copy x
external special remote protocol error, unexpectedly received "UNSUPPORTED-REQUEST" (command not allowed at this time)
(unable to use special remote due to protocol error) failed
copy: 1 failed
Protocol debug shows what's happening:
[2023-03-28 16:00:09.361602472] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] --> VERSION 2
[2023-03-28 16:00:09.361764519] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] <-- ERROR unsupported VERSION
[2023-03-28 16:00:09.361897595] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] <-- EXTENSIONS INFO GETGITREMOTENAME ASYNC
[2023-03-28 16:00:09.362112912] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] --> UNSUPPORTED-REQUEST
[2023-03-28 16:00:09.362212948] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] <-- PREPARE
[2023-03-28 16:00:09.362332494] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-remote-test[3] --> UNSUPPORTED-REQUEST
Not really ideal behavior for an old version of git-annex when used with
an external that wants to send VERSION 2 to ensure it does not need to deal with
this bug. But better than nothing I suppose.
(That output really ought to be fixed going forward, but old versions of git-annex
of course can't be fixed now.)
for. So git-annex will keep supporting both versions.
"""]]

12
doc/bugs/external_remote_export_sent_to_wrong_process/comment_3_d774a7319bd1a1778adb2004229f057e._comment Normal file
View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2023-03-28T20:51:40Z"
content="""
I've implemented support for `VERSION 2`. Recommend any external special
remotes that support exporttree=yes use it to avoid talking with a buggy
git-annex version.
(See [[todo/better_message_for_external_special_remote_protocol_mismatch]]
for related todo.)
"""]]