33 lines
1.5 KiB
Markdown
33 lines
1.5 KiB
Markdown
[[!meta title="streaming for transitive transfers"]]
|
|
|
|
This todo was originally about `git-annex copy --from --to`, which got
|
|
implemented, but without the ability to stream content between the two
|
|
remotes.
|
|
|
|
So this todo remains open, but is now only concerned with
|
|
streaming an object that is being received from one remote out to another
|
|
repository without first needing to buffer the whole object on disk.
|
|
|
|
git-annex's remote interface does not currently support that.
|
|
`retrieveKeyFile` stores the object into a file. And `storeKey`
|
|
sends an object file to a remote.
|
|
|
|
The interface would need to be extended to support this, and the external
|
|
special remote interface as well. As well as git remotes and special
|
|
remotes needing to be updated to support the new interface.
|
|
|
|
There's also a question of buffering. If a object is being received from a
|
|
remote faster than it is being sent to the other remote, it has to be
|
|
buffered somewhere, eg not in memory. Or the receive interface needs to
|
|
include a way to get the sender to pause.
|
|
|
|
----
|
|
|
|
Recieving to a file, and sending from the same file as it grows is one
|
|
possibility, since that would handle buffering, and it might avoid needing
|
|
to change interfaces as much. It would still need a new interface since the
|
|
current one does not guarantee the file is written in-order.
|
|
|
|
A fifo is a possibility, but would certianly not work with remotes
|
|
that don't write to the file in-order. Also resuming a download would not
|
|
work with a fifo, the sending remote wouldn't know where to resume from.
|