Merge branch 'master' into assistant
This commit is contained in:
commit
208e96deef
4 changed files with 58 additions and 19 deletions
|
@ -18,6 +18,17 @@ available!
|
|||
I may need to fork off multiple watcher processes to handle this.
|
||||
See [[bugs/Issue_on_OSX_with_some_system_limits]].
|
||||
|
||||
## todo
|
||||
|
||||
* Run niced and ioniced? Seems to make sense, this is a background job.
|
||||
* configurable option to only annex files meeting certian size or
|
||||
filename criteria
|
||||
* option to check files not meeting annex criteria into git directly,
|
||||
automatically
|
||||
* honor .gitignore, not adding files it excludes (difficult, probably
|
||||
needs my own .gitignore parser to avoid excessive running of git commands
|
||||
to check for ignored files)
|
||||
|
||||
## beyond Linux
|
||||
|
||||
I'd also like to support OSX and if possible the BSDs.
|
||||
|
@ -65,17 +76,6 @@ I'd also like to support OSX and if possible the BSDs.
|
|||
|
||||
* Windows has a Win32 ReadDirectoryChangesW, and perhaps other things.
|
||||
|
||||
## todo
|
||||
|
||||
- Run niced and ioniced? Seems to make sense, this is a background job.
|
||||
- configurable option to only annex files meeting certian size or
|
||||
filename criteria
|
||||
- option to check files not meeting annex criteria into git directly,
|
||||
automatically
|
||||
- honor .gitignore, not adding files it excludes (difficult, probably
|
||||
needs my own .gitignore parser to avoid excessive running of git commands
|
||||
to check for ignored files)
|
||||
|
||||
## the races
|
||||
|
||||
Many races need to be dealt with by this code. Here are some of them.
|
||||
|
|
|
@ -1,23 +1,25 @@
|
|||
Once files are added (or removed or moved), need to send those changes to
|
||||
all the other git clones, at both the git level and the key/value level.
|
||||
|
||||
## action items
|
||||
## immediate action items
|
||||
|
||||
* Check that download transfer triggering code works (when a symlink appears
|
||||
and the remote does *not* upload to us.
|
||||
* Investigate why transfers seem to block other git-annex assistant work.
|
||||
* At startup, and possibly periodically, look for files we have that
|
||||
location tracking indicates remotes do not, and enqueue Uploads for
|
||||
them. Also, enqueue Downloads for any files we're missing.
|
||||
* Find a way to probe available outgoing bandwidth, to throttle so
|
||||
we don't bufferbloat the network to death.
|
||||
* git-annex needs a simple speed control knob, which can be plumbed
|
||||
through to, at least, rsync. A good job for an hour in an
|
||||
airport somewhere.
|
||||
* file transfer processes are not waited for, contain the zombies.
|
||||
* The TransferWatcher does not notice ongoing transfers, because inotify is
|
||||
waiting for the info file to be closed, but that never happens, it's left
|
||||
open to keep it locked. May need to separate the transfer info files
|
||||
into an info file, and a lock file.
|
||||
|
||||
## longer-term TODO
|
||||
|
||||
* git-annex needs a simple speed control knob, which can be plumbed
|
||||
through to, at least, rsync. A good job for an hour in an
|
||||
airport somewhere.
|
||||
* Find a way to probe available outgoing bandwidth, to throttle so
|
||||
we don't bufferbloat the network to death.
|
||||
* Investigate the XMPP approach like dvcs-autosync does, or other ways of
|
||||
signaling a change out of band.
|
||||
* Add a hook, so when there's a change to sync, a program can be run
|
||||
|
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawmWQTrnPloMWiPFg8Y2Y5g-2IYe26D0KKw"
|
||||
nickname="Jim"
|
||||
subject="comment 6"
|
||||
date="2012-07-07T16:18:06Z"
|
||||
content="""
|
||||
It's also possible you got a one-time DRAM corruption. You have to expect those to happen every so often unless you're using ECC memory.
|
||||
"""]]
|
29
doc/todo/threaded_runtime.mdwn
Normal file
29
doc/todo/threaded_runtime.mdwn
Normal file
|
@ -0,0 +1,29 @@
|
|||
The [[design/assistant]] would be better if git-annex used ghc's threaded
|
||||
runtime (`ghc -threaded`).
|
||||
|
||||
Currently, whenever the assistant code runs some external command, all
|
||||
threads are blocked waiting for it to finish.
|
||||
|
||||
For transfers, the assistant works around this problem by forking separate
|
||||
upload processes, and not waiting on them until it sees an indication that
|
||||
they have finished the transfer. While this works, it's messy.. threaded
|
||||
would be better.
|
||||
|
||||
When pulling, pushing, and merging, the assistant runs external git
|
||||
commands, and this does block all other threads. The threaded runtime would
|
||||
really help here.
|
||||
|
||||
---
|
||||
|
||||
Currently, git-annex seems unstable when built with the threaded runtime.
|
||||
The test suite tends to hang when testing add. `git-annex` occasionally
|
||||
hangs, apparently in a futex lock. This is not the assistant hanging, and
|
||||
git-annex does not otherwise use threads, so this is surprising. --[[Joey]]
|
||||
|
||||
---
|
||||
|
||||
It would be possible to not use the threaded runtime. Instead, we could
|
||||
have a child process pool, with associated continuations to run after a
|
||||
child process finishes. Then periodically do a nonblocking waitpid on each
|
||||
process in the pool in turn (waiting for any child could break anything not
|
||||
using the pool!). This is probably a last resort...
|
Loading…
Add table
Add a link
Reference in a new issue