Merge branch 'master' into assistant

doc/design/assistant/inotify.mdwn

@@ -18,6 +18,17 @@ available!
   I may need to fork off multiple watcher processes to handle this.
   See [[bugs/Issue_on_OSX_with_some_system_limits]].
 
+## todo
+
+* Run niced and ioniced? Seems to make sense, this is a background job.
+* configurable option to only annex files meeting certian size or
+  filename criteria
+* option to check files not meeting annex criteria into git directly,
+  automatically
+* honor .gitignore, not adding files it excludes (difficult, probably
+  needs my own .gitignore parser to avoid excessive running of git commands
+  to check for ignored files)
+
 ## beyond Linux
 
 I'd also like to support OSX and if possible the BSDs.
@@ -65,17 +76,6 @@ I'd also like to support OSX and if possible the BSDs.
 
 * Windows has a Win32 ReadDirectoryChangesW, and perhaps other things.
 
-## todo
-
-- Run niced and ioniced? Seems to make sense, this is a background job.
-- configurable option to only annex files meeting certian size or
-  filename criteria
-- option to check files not meeting annex criteria into git directly,
-  automatically
-- honor .gitignore, not adding files it excludes (difficult, probably
-  needs my own .gitignore parser to avoid excessive running of git commands
-  to check for ignored files)
-
 ## the races
 
 Many races need to be dealt with by this code. Here are some of them.

doc/design/assistant/syncing.mdwn

@@ -1,23 +1,25 @@
 Once files are added (or removed or moved), need to send those changes to
 all the other git clones, at both the git level and the key/value level.
 
-## action items
+## immediate action items
 
 * Check that download transfer triggering code works (when a symlink appears
   and the remote does *not* upload to us.
-* Investigate why transfers seem to block other git-annex assistant work.
 * At startup, and possibly periodically, look for files we have that
   location tracking indicates remotes do not, and enqueue Uploads for
   them. Also, enqueue Downloads for any files we're missing.
-* Find a way to probe available outgoing bandwidth, to throttle so
+* The TransferWatcher does not notice ongoing transfers, because inotify is
-  we don't bufferbloat the network to death.
+  waiting for the info file to be closed, but that never happens, it's left
-* git-annex needs a simple speed control knob, which can be plumbed
+  open to keep it locked. May need to separate the transfer info files
-  through to, at least, rsync. A good job for an hour in an
+  into an info file, and a lock file.
-  airport somewhere.
-* file transfer processes are not waited for, contain the zombies.
 
 ## longer-term TODO
 
+* git-annex needs a simple speed control knob, which can be plumbed
+  through to, at least, rsync. A good job for an hour in an
+  airport somewhere.
+* Find a way to probe available outgoing bandwidth, to throttle so
+  we don't bufferbloat the network to death.
 * Investigate the XMPP approach like dvcs-autosync does, or other ways of
    signaling a change out of band.
 * Add a hook, so when there's a change to sync, a program can be run

doc/forum/fsck_gives_false_positives/comment_6_1d4fbbd212fa92967abda346323031f4._comment

@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmWQTrnPloMWiPFg8Y2Y5g-2IYe26D0KKw"
+ nickname="Jim"
+ subject="comment 6"
+ date="2012-07-07T16:18:06Z"
+ content="""
+It's also possible you got a one-time DRAM corruption.  You have to expect those to happen every so often unless you're using ECC memory.
+"""]]

doc/todo/threaded_runtime.mdwn

@@ -0,0 +1,29 @@
+The [[design/assistant]] would be better if git-annex used ghc's threaded
+runtime (`ghc -threaded`).
+
+Currently, whenever the assistant code runs some external command, all
+threads are blocked waiting for it to finish.
+
+For transfers, the assistant works around this problem by forking separate
+upload processes, and not waiting on them until it sees an indication that
+they have finished the transfer. While this works, it's messy.. threaded
+would be better.
+
+When pulling, pushing, and merging, the assistant runs external git
+commands, and this does block all other threads. The threaded runtime would
+really help here.
+
+---
+
+Currently, git-annex seems unstable when built with the threaded runtime.
+The test suite tends to hang when testing add. `git-annex` occasionally
+hangs, apparently in a futex lock. This is not the assistant hanging, and
+git-annex does not otherwise use threads, so this is surprising. --[[Joey]]
+
+---
+
+It would be possible to not use the threaded runtime. Instead, we could
+have a child process pool, with associated continuations to run after a
+child process finishes. Then periodically do a nonblocking waitpid on each
+process in the pool in turn (waiting for any child could break anything not
+using the pool!). This is probably a last resort...