design page for rate limiting

doc/design/assistant.mdwn

@@ -11,10 +11,10 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]]
 * Month 4 "cloud": [[!traillink cloud]] [[!traillink transfer_control]]
 * Month 5 "cloud continued": [[!traillink xmpp]] [[!traillink more_cloud_providers]]
 * Month 6 "9k bonus round": [[!traillink desymlink]]
-* Month 7: user-driven features and polishing; 
+* Month 7: user-driven features and polishing;
   [presentation at LCA2013](http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4)
 * Month 8: [[!traillink Android]]
-* Month 9: user-driven features and polishing
+* Month 9: [[screencasts|videos]] and polishing
 
 We are, approximately, here:
 
@@ -29,6 +29,7 @@ We are, approximately, here:
 
 ## not yet on the map:
 
+* [[rate_limiting]]
 * [[partial_content]]
 * encrypted git remotes using [git-remote-gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt)
 * [[deltas]]

doc/design/assistant/rate_limiting.mdwn

@@ -0,0 +1,57 @@
+Webapp needs a simple speed control knob, especially to avoid saturating 
+bandwidth on uploads.
+
+We have basic throttling support in git-annex for rsync,
+but none for any special remotes. A good first step would be to expose
+this in the webapp, and ensure that `git-annex-shell` also honors it when
+sending/receiving data.
+
+We actually need two speed controls, one for upload and one for download.
+
+It is probably not necessary to throttle git push/pull operations, as the
+data transfers tend to be small. Only throttling file transfers is
+essential.
+
+## possibility: trickle
+
+Since `git-annex transferkeys` is a separate process, one easy option would
+be to run it inside `trickle`. If the user changes the bandwidth limits,
+it could kill the transfer and restart it with different trickle options.
+
+Problem: Not all special remotes support resuming transfers, so this is
+suboptimal. (So too are the pause/resume buttons, when using those
+remotes!)
+
+`trickle` is available for OSX as well as Linux and BSDs.
+<http://hints.macworld.com/comment.php?mode=view&cid=65362>
+<http://mac.softpedia.com/get/Network-Admin/Trickle.shtml>
+It is probably not easily available for Android, as it uses `LD_PRELOAD`.
+
+## possibility: built in IO limiting
+
+A cleaner method would be to do the limiting inside git-annex. We already
+have metered file IO. It should be possible to make the meter not only report
+on the transfer speed, but detect when it's going too fast, and delay. This
+will delay the file streaming through the special remote's transfer code,
+so should work for a variety of special remotes. (Not for rsync or bup
+or git-annex-shell though, so those need to be handled separately.)
+
+Should work well for uploads at least. I don't know how well it would work
+for throttling downloads; the sender may just keep sending data and the
+data buffer before it gets to the IO meter. Maybe once the buffers fill the
+OS would have the TCP throttled down. Needs investigation; trickle claims
+to throttle downloads.
+
+## communications channels
+
+There would need to be a communication channel for the assistant to tell
+`git annex transferkeys` when the rate limit has changed. It could for
+example send it a SIGUSR1, and then leave it up to the process to reload
+the git config. Inside the IO meter, we could have an MVar that contains
+the current throttle value, so the IO meter could check it each time it's
+called and adjust its throttling appropriately.
+
+Ideally, the assistant could also communicate in the same way with
+`git-annex-shell` to tell it when the limit has changed. Since
+`git-annex-shell` uses rsync, it would need to abort the transfer, and rely
+on the other side retrying to start it up with the new limit.

doc/design/assistant/syncing.mdwn

@@ -25,11 +25,6 @@ syncs?
 ## TODO
 
 * Test MountWatcher on LXDE.
-* git-annex needs a simple speed control knob, which can be plumbed
-  through to, at least, rsync. A good job for an hour in an
-  airport somewhere.
-* Find a way to probe available outgoing bandwidth, to throttle so
-  we don't bufferbloat the network to death.
 * Add a hook, so when there's a change to sync, a program can be run
    and do its own signaling.
 * --debug will show often unnecessary work being done. Optimise.