mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

design work

Browse source
This commit is contained in:
Joey Hess 2020-12-02 14:31:24 -04:00
parent 439e3f8e24
commit 1858b65d88
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 52 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment
View file

@ -30,5 +30,6 @@ Also, all parts of Messages that remotes use will need to be serialized
over the transferkeys protocol. Eg, Messages.prompt when the transferkeys
process is doing something like ssh that is prompting for a password.
So transferkeys will need its own OutputType to make Messages communicate
over the pipe.
over the pipe. (This is a hard thing, so added a todo for it:
[[todo/serialize_Messages_for_transferkeys]]
"""]]

50
doc/todo/serialize_Messages_for_transferkeys.mdwn Normal file
View file

@ -0,0 +1,50 @@
For [[todo/more_extensive_retries_to_mask_transient_failures]],
multiple transferkeys processes need to be run. So all console IO needs to
be serialized and sent back to the main git-annex process from them,
to avoid concurrent output.
Using --json and --json-progress and --json-error-messages is pretty close
to what would be needed. That handles progress of transfers, and overall
success/failure.
Using --json-error-messages makes most error messages be output in json.
However, that uses an error-messages array in the json object,
which error messages are added to as an action runs, and it's only sent at
the end. So a problem for eg, a relay of stderr output from a command
(eg ssh password prompt), or for anything that displays a warning that
should be displayed before the transfer is complete.
Since the existing json is not a perfect fit, it might be better to use a
custom protocol, implemented as a new output type. Then anything in
Messages and child modules that looks at output types will support it.
(Although perhaps some of that stuff is not used by remotes.)
A few notes on implementing that:
* Messages.Progress.mkOutputHandler, which uses mkStderrEmitter,
outputs to stderr directly no matter the output type currently.
It would need to be changed to support the new output type.
(And probably should for concurrent output mode too actually!)
* So does warningIO, though it's only used in a couple of remotes
and rarely. It would be good to find a way to eliminate it.
* Messages.prompt. Which is used by remotes, and would need to
communicate over the pipe to the parent git-annex bidirectionally.
Eg, send a message saying the parent needs to prepare for prompt,
wait for it to reply saying it has, and then send a message when the
prompting is done. (Note that the parent would need to detect if the child
process crashed to avoid being locked waiting for the prompt.)
* Messages.Internal.outputMessage is used by several things, and
includes some special parameters used in json mode. Since the parent
git-annex might itself have json mode enabled, those parameters will need
to be included in the serialization. But those parameters are currently
actually functions that manipulate the json object that will be outputted
later. So cannot be serialized. Uuuuh.
Maybe the thing to do is, pass along the --json options to transferkeys,
and have a message type for json objects, which it uses to send them
to git-annex, which can then output them. outputMessage can handle the
new output type by sending the message through the pipe, and also
building any json object, and sending it through the pipe once it's done.