From c4dfeaef539ff6d1bf941d1a7d27c51aa4b6d58c Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 15 Oct 2024 16:02:19 -0400 Subject: [PATCH] streaming uploads --- doc/design/passthrough_proxy.mdwn | 9 +++++++++ doc/todo/git-annex_proxies.mdwn | 5 ++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/doc/design/passthrough_proxy.mdwn b/doc/design/passthrough_proxy.mdwn index 578b79a427..c6b5f57b27 100644 --- a/doc/design/passthrough_proxy.mdwn +++ b/doc/design/passthrough_proxy.mdwn @@ -411,6 +411,15 @@ sending SUCCESS, leading to a perhaps long delay on the client before an upload finishes. Perhaps extend the P2P protocol with progress information for the uploads? +To stream uploads via the proxy, storeKey would need its interface changed +to not read the object file itself, but read from eg a lazy ByteString. +Chunking and encryption would complicate that. Chunking seems fairly +straightforward since it uses a lazy ByteString internally. +storeExport would change similarly. The external special remote protocol +would also need a change if it was to support that. + +---- + Both of those file-based approaches need the proxy to have enough free disk space to buffer the largest file, times the number of concurrent uploads+downloads. So the proxy will need to check annex.diskreserve diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn index 662b8c7a5d..50d1561073 100644 --- a/doc/todo/git-annex_proxies.mdwn +++ b/doc/todo/git-annex_proxies.mdwn @@ -28,9 +28,12 @@ Planned schedule of work: ## work notes -* Currently working on streaming download via proxy from special remote +* Currently working on streaming special remotes via proxy in the `streamproxy` branch. +* Downloads from special remotes can stream (though using a temp file on + the proxy). Next: Streaming uploads via the proxy. + ## completed items for September's work on proving behavior of preferred content * Static analysis to detect "not present", "not balanced", and similar