mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

streaming uploads

Browse source
This commit is contained in:
Joey Hess 2024-10-15 16:02:19 -04:00
parent 6eb32468bc
commit c4dfeaef53
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
2 changed files with 13 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
doc/design/passthrough_proxy.mdwn
View file

@ -411,6 +411,15 @@ sending SUCCESS, leading to a perhaps long delay on the client before an
upload finishes. Perhaps extend the P2P protocol with progress information
for the uploads?
To stream uploads via the proxy, storeKey would need its interface changed
to not read the object file itself, but read from eg a lazy ByteString.
Chunking and encryption would complicate that. Chunking seems fairly
straightforward since it uses a lazy ByteString internally.
storeExport would change similarly. The external special remote protocol
would also need a change if it was to support that.
----
Both of those file-based approaches need the proxy to have enough free disk
space to buffer the largest file, times the number of concurrent
uploads+downloads. So the proxy will need to check annex.diskreserve

5
doc/todo/git-annex_proxies.mdwn
View file

@ -28,9 +28,12 @@ Planned schedule of work:
## work notes
* Currently working on streaming download via proxy from special remote
* Currently working on streaming special remotes via proxy
in the `streamproxy` branch.
* Downloads from special remotes can stream (though using a temp file on
the proxy). Next: Streaming uploads via the proxy.
## completed items for September's work on proving behavior of preferred content
* Static analysis to detect "not present", "not balanced", and similar