design page for rate limiting
This commit is contained in:
Joey Hess
parent
2ded1b8c75
commit
e4dfe84f4c
3 changed files with 60 additions and 7 deletions
|
|
@ -11,10 +11,10 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]]
|
|||
* Month 4 "cloud": [[!traillink cloud]] [[!traillink transfer_control]]
|
||||
* Month 5 "cloud continued": [[!traillink xmpp]] [[!traillink more_cloud_providers]]
|
||||
* Month 6 "9k bonus round": [[!traillink desymlink]]
|
||||
* Month 7: user-driven features and polishing;
|
||||
* Month 7: user-driven features and polishing;
|
||||
[presentation at LCA2013](http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4)
|
||||
* Month 8: [[!traillink Android]]
|
||||
* Month 9: user-driven features and polishing
|
||||
* Month 9: [[screencasts|videos]] and polishing
|
||||
|
||||
We are, approximately, here:
|
||||
|
||||
|
|
@ -29,6 +29,7 @@ We are, approximately, here:
|
|||
|
||||
## not yet on the map:
|
||||
|
||||
* [[rate_limiting]]
|
||||
* [[partial_content]]
|
||||
* encrypted git remotes using [git-remote-gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt)
|
||||
* [[deltas]]
|
||||
|
|
|
|||
57
doc/design/assistant/rate_limiting.mdwn
Normal file
57
doc/design/assistant/rate_limiting.mdwn
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
Webapp needs a simple speed control knob, especially to avoid saturating
|
||||
bandwidth on uploads.
|
||||
|
||||
We have basic throttling support in git-annex for rsync,
|
||||
but none for any special remotes. A good first step would be to expose
|
||||
this in the webapp, and ensure that `git-annex-shell` also honors it when
|
||||
sending/receiving data.
|
||||
|
||||
We actually need two speed controls, one for upload and one for download.
|
||||
|
||||
It is probably not necessary to throttle git push/pull operations, as the
|
||||
data transfers tend to be small. Only throttling file transfers is
|
||||
essential.
|
||||
|
||||
## possibility: trickle
|
||||
|
||||
Since `git-annex transferkeys` is a separate process, one easy option would
|
||||
be to run it inside `trickle`. If the user changes the bandwidth limits,
|
||||
it could kill the transfer and restart it with different trickle options.
|
||||
|
||||
Problem: Not all special remotes support resuming transfers, so this is
|
||||
suboptimal. (So too are the pause/resume buttons, when using those
|
||||
remotes!)
|
||||
|
||||
`trickle` is available for OSX as well as Linux and BSDs.
|
||||
<http://hints.macworld.com/comment.php?mode=view&cid=65362>
|
||||
<http://mac.softpedia.com/get/Network-Admin/Trickle.shtml>
|
||||
It is probably not easily available for Android, as it uses `LD_PRELOAD`.
|
||||
|
||||
## possibility: built in IO limiting
|
||||
|
||||
A cleaner method would be to do the limiting inside git-annex. We already
|
||||
have metered file IO. It should be possible to make the meter not only report
|
||||
on the transfer speed, but detect when it's going too fast, and delay. This
|
||||
will delay the file streaming through the special remote's transfer code,
|
||||
so should work for a variety of special remotes. (Not for rsync or bup
|
||||
or git-annex-shell though, so those need to be handled separately.)
|
||||
|
||||
Should work well for uploads at least. I don't know how well it would work
|
||||
for throttling downloads; the sender may just keep sending data and the
|
||||
data buffer before it gets to the IO meter. Maybe once the buffers fill the
|
||||
OS would have the TCP throttled down. Needs investigation; trickle claims
|
||||
to throttle downloads.
|
||||
|
||||
## communications channels
|
||||
|
||||
There would need to be a communication channel for the assistant to tell
|
||||
`git annex transferkeys` when the rate limit has changed. It could for
|
||||
example send it a SIGUSR1, and then leave it up to the process to reload
|
||||
the git config. Inside the IO meter, we could have an MVar that contains
|
||||
the current throttle value, so the IO meter could check it each time it's
|
||||
called and adjust its throttling appropriately.
|
||||
|
||||
Ideally, the assistant could also communicate in the same way with
|
||||
`git-annex-shell` to tell it when the limit has changed. Since
|
||||
`git-annex-shell` uses rsync, it would need to abort the transfer, and rely
|
||||
on the other side retrying to start it up with the new limit.
|
||||
|
|
@ -25,11 +25,6 @@ syncs?
|
|||
## TODO
|
||||
|
||||
* Test MountWatcher on LXDE.
|
||||
* git-annex needs a simple speed control knob, which can be plumbed
|
||||
through to, at least, rsync. A good job for an hour in an
|
||||
airport somewhere.
|
||||
* Find a way to probe available outgoing bandwidth, to throttle so
|
||||
we don't bufferbloat the network to death.
|
||||
* Add a hook, so when there's a change to sync, a program can be run
|
||||
and do its own signaling.
|
||||
* --debug will show often unnecessary work being done. Optimise.
|
||||
|
|
|
|||
Loading…
Reference in a new issue