Merge branch 'master' into assistant

This commit is contained in:
Joey Hess 2012-07-07 11:23:28 -06:00
commit 208e96deef
4 changed files with 58 additions and 19 deletions

View file

@ -18,6 +18,17 @@ available!
I may need to fork off multiple watcher processes to handle this.
See [[bugs/Issue_on_OSX_with_some_system_limits]].
## todo
* Run niced and ioniced? Seems to make sense, this is a background job.
* configurable option to only annex files meeting certian size or
filename criteria
* option to check files not meeting annex criteria into git directly,
automatically
* honor .gitignore, not adding files it excludes (difficult, probably
needs my own .gitignore parser to avoid excessive running of git commands
to check for ignored files)
## beyond Linux
I'd also like to support OSX and if possible the BSDs.
@ -65,17 +76,6 @@ I'd also like to support OSX and if possible the BSDs.
* Windows has a Win32 ReadDirectoryChangesW, and perhaps other things.
## todo
- Run niced and ioniced? Seems to make sense, this is a background job.
- configurable option to only annex files meeting certian size or
filename criteria
- option to check files not meeting annex criteria into git directly,
automatically
- honor .gitignore, not adding files it excludes (difficult, probably
needs my own .gitignore parser to avoid excessive running of git commands
to check for ignored files)
## the races
Many races need to be dealt with by this code. Here are some of them.

View file

@ -1,23 +1,25 @@
Once files are added (or removed or moved), need to send those changes to
all the other git clones, at both the git level and the key/value level.
## action items
## immediate action items
* Check that download transfer triggering code works (when a symlink appears
and the remote does *not* upload to us.
* Investigate why transfers seem to block other git-annex assistant work.
* At startup, and possibly periodically, look for files we have that
location tracking indicates remotes do not, and enqueue Uploads for
them. Also, enqueue Downloads for any files we're missing.
* Find a way to probe available outgoing bandwidth, to throttle so
we don't bufferbloat the network to death.
* git-annex needs a simple speed control knob, which can be plumbed
through to, at least, rsync. A good job for an hour in an
airport somewhere.
* file transfer processes are not waited for, contain the zombies.
* The TransferWatcher does not notice ongoing transfers, because inotify is
waiting for the info file to be closed, but that never happens, it's left
open to keep it locked. May need to separate the transfer info files
into an info file, and a lock file.
## longer-term TODO
* git-annex needs a simple speed control knob, which can be plumbed
through to, at least, rsync. A good job for an hour in an
airport somewhere.
* Find a way to probe available outgoing bandwidth, to throttle so
we don't bufferbloat the network to death.
* Investigate the XMPP approach like dvcs-autosync does, or other ways of
signaling a change out of band.
* Add a hook, so when there's a change to sync, a program can be run

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmWQTrnPloMWiPFg8Y2Y5g-2IYe26D0KKw"
nickname="Jim"
subject="comment 6"
date="2012-07-07T16:18:06Z"
content="""
It's also possible you got a one-time DRAM corruption. You have to expect those to happen every so often unless you're using ECC memory.
"""]]

View file

@ -0,0 +1,29 @@
The [[design/assistant]] would be better if git-annex used ghc's threaded
runtime (`ghc -threaded`).
Currently, whenever the assistant code runs some external command, all
threads are blocked waiting for it to finish.
For transfers, the assistant works around this problem by forking separate
upload processes, and not waiting on them until it sees an indication that
they have finished the transfer. While this works, it's messy.. threaded
would be better.
When pulling, pushing, and merging, the assistant runs external git
commands, and this does block all other threads. The threaded runtime would
really help here.
---
Currently, git-annex seems unstable when built with the threaded runtime.
The test suite tends to hang when testing add. `git-annex` occasionally
hangs, apparently in a futex lock. This is not the assistant hanging, and
git-annex does not otherwise use threads, so this is surprising. --[[Joey]]
---
It would be possible to not use the threaded runtime. Instead, we could
have a child process pool, with associated continuations to run after a
child process finishes. Then periodically do a nonblocking waitpid on each
process in the pool in turn (waiting for any child could break anything not
using the pool!). This is probably a last resort...