Commit graph

1700 commits

Author SHA1 Message Date
Joey Hess
6af319d8cd enqueue Downloads when new symlinks appear to content we don't have 2012-07-05 10:58:49 -06:00
Joey Hess
c8135ea0a8 split logic for uploads and downloads 2012-07-05 10:44:07 -06:00
Joey Hess
83c66ccaf8 queue Uploads of newly added files to remotes
Added knownRemotes to DaemonStatus. This list is not entirely trivial to
calculate, and having it here should make it easier to add/remove remotes
on the fly later on. It did require plumbing the daemonstatus through to
some more threads.
2012-07-05 10:21:22 -06:00
Joey Hess
b4917bd18f add transfer watching thread
Worked the 1st try!
2012-07-03 10:58:40 -04:00
Joey Hess
32e5e02e43 added currentTransfers 2012-07-02 16:12:55 -04:00
Joey Hess
ad0b827957 added 2012-07-02 16:07:39 -04:00
Joey Hess
397117429c simplify
modifyMVar_ catches exceptions, so no need to roll my own
2012-06-28 23:40:16 -04:00
Joey Hess
a3636602ab MVar deadlock problem seems to be fixed by previous commit 2012-06-28 13:47:15 -04:00
Joey Hess
4888c5b042 improve thread termination handling
The reason the DirWatcher had to wait for program termination was because
it used withINotify, so when it finished, its watcher threads were killed.
But since I have two DirWatcher threads now, that was not good, and could
perhaps explain the MVar problem I saw yesterday. In any case, fixed this
part of the code by making the DirWatcher return a handle that can be used
to stop it, and now the main Assistant thread is the only one calling
waitForTermination.
2012-06-28 13:37:03 -04:00
Joey Hess
40f357fdcf tweak 2012-06-28 13:04:02 -04:00
Joey Hess
fb51d99951 merge conflict resolution now working
Avoid MVar deadlock issue, which I don't understand.
Have not taken the time to debug it fully, because it turns out I don't
need to resolve merge conflicts when a new branch ref is written... I
think.

Ensure the git-annex branch is merged when doing a manual pull.
Otherwise it can get out of sync, since git-annex normally only merges it
once per run.
2012-06-27 20:50:50 -04:00
Joey Hess
59b5266ad1 actually fetch from remote when doing a manual pull
forgot to do this
2012-06-27 20:30:04 -04:00
Joey Hess
783bee285f automatic conflict resolution for assistant 2012-06-27 20:06:21 -04:00
Joey Hess
67c8ef7de2 use a TMVar
SampleMVar won't work; between getting the current value and changing
it, another thread could made a change, which would get lost.

TMVar works well; this update situation is handled by atomic transactions.
2012-06-26 19:21:44 -04:00
Joey Hess
f2bce89055 better data type for push records
Not yet plumbed thru
2012-06-26 12:36:42 -04:00
Joey Hess
05c4dfb941 fixup merges now done when needed 2012-06-25 20:16:30 -04:00
Joey Hess
5cfe91f06d add a push retry thread 2012-06-25 16:38:12 -04:00
Joey Hess
0b146f9ecc reorg threads 2012-06-25 16:10:24 -04:00
Joey Hess
19eee6a1df noninteractive merge 2012-06-23 10:29:46 -04:00
Joey Hess
d707f2dce5 cleanup 2012-06-23 01:35:40 -04:00
Joey Hess
38df4ed44f license 2012-06-23 01:20:40 -04:00
Joey Hess
e699ce1841 added a merger thread
Wow! I can create a file in repo a, and it instantly* shows up in repo b!

* under 1 second anyway
2012-06-22 17:01:08 -04:00
Joey Hess
e9630e90de the syncer now pushes out changes to remotes, in parallel
Note that, since this always pushes branch synced/master to the remote, it
assumes that master has already gotten all the commits that are on the
remote merged in. Otherwise, fast-forward prevention may prevent the push.

That's probably ok, because the next stage is to automatically detect
incoming pushes and merge.
2012-06-22 15:49:48 -04:00
Joey Hess
28e28bc043 stub syncer thread and commit channel 2012-06-22 14:10:25 -04:00
Joey Hess
75dba7f7bc belt and suspenders check
It's possible for there to be multiple queued changes all adding the same
file, and for those changes to be reordered. Maybe. This check will guard
against that ending up adding the wrong version of the file last.
2012-06-20 20:05:40 -04:00
Joey Hess
33b914bcf1 pending adds now retried for kqueue
Rethought how to keep track of pending adds that need to be retried later.
The commit thread already run up every second when there are changes,
so let's keep pending adds queued as changes until they're safe to add.

Also, the committer is now smarter about avoiding empty commits when
all the adds are currently unsafe, or in the rare case that an add event
for a symlink is not received in time. It may avoid them entirely.

This seems to work as before for inotify, and is untested for kqueue.

(Actually commit batching seems to be improved for inotify, although I'm
not sure why. I'm seeing only two commits made during large batch
operations, and the first of those is the non-batch mode commit.)
2012-06-20 19:29:53 -04:00
Joey Hess
e0fdfb2e70 maintain set of files pendingAdd
Kqueue needs to remember which files failed to be added due to being open,
and retry them. This commit gets the data in place for such a retry thread.

Broke KeySource out into its own file, and added Eq and Ord instances
so it can be stored in a Set.
2012-06-20 16:31:46 -04:00
Joey Hess
627504744c inverted logic 2012-06-19 09:17:06 -04:00
Joey Hess
57cf65eb6d fix kevent symlink creation 2012-06-19 02:40:21 -04:00
Joey Hess
a5cceb7d4f make --force really bypass lsof check 2012-06-19 00:23:14 -04:00
Joey Hess
7a09d74319 lifted out the kqueue and inotify to a generic DirWatcher interface
Kqueue code for dispatching events is not tested and probably doesn't
build.
2012-06-18 23:49:07 -04:00
Joey Hess
22b563408b refactor 2012-06-19 02:13:39 +00:00
Joey Hess
3d163f5ff9 fix build 2012-06-19 01:52:07 +00:00
Joey Hess
d680ff7ef0 kqueue code compiles on debian kfreebsd 2012-06-18 16:34:08 -04:00
Joey Hess
a39b73d118 recurse dirTree and open the directories for kqueue to watch 2012-06-18 13:01:58 -04:00
Joey Hess
3c8a9043b6 skeleton C library for calling kqueue 2012-06-18 12:25:20 -04:00
Joey Hess
91567ab8f6 make inotify a build flag etc 2012-06-17 17:15:56 -04:00
Joey Hess
1863185693 startup check fixes
Move lsof check, and display a message before daemon startup if on an
unsupported OS.
2012-06-17 14:21:29 -04:00
Joey Hess
5d63c2a4bb check files with lsof in batches before adding
I've tested both cases where this is necessary, and it works great!
A file with multiple writers is not added until the last one closes.
2012-06-15 22:36:42 -04:00
Joey Hess
7ee300a47f race avoidance
When there are duplicate add events for the same file, only add it once.
2012-06-15 20:55:03 -04:00
Joey Hess
59abd787c9 can't wait for LinkChanges specifically
There is indeed a race waiting for LinkChanges:

1. file annexed, link made
2. link deleted
3. inotify event for link creation runs, but as link is gone, handler is not run
2012-06-15 19:17:21 -04:00
Joey Hess
679ef4c858 continued work on deferred addding 2012-06-15 19:00:06 -04:00
Joey Hess
c27c751b34 preliminary deferring of file adds to commit time
Defer adding files to the annex until commit time, when during a batch
operation, a bundle of files will be available. This will allow for
checking a them all with a single lsof call.

The tricky part is that adding the file causes a symlink change inotify.
So I made it wait for an appropriate number of symlink changes to be
received before continuing with the commit. This avoids any delay
in the commit process. It is possible that some unrelated symlink change is
made; if that happens it'll commit it and delay committing the newly added
symlink for 1 second. This seems ok. I do rely on the expected symlink
change event always being received, but only when the add succeeds.

Another way to do it might be to directly stage the symlink, and then
ignore the redundant symlink change event. That would involve some
redundant work, and perhaps an empty commit, but if this code turns
out to have some bug, that'd be the best way to avoid it.

FWIW, this change seems to, as a bonus, have produced better grouping
of batch changes into single commits. Before, a large batch change would
result in a series of commits, with the first containing only one file,
and each of the rest bundling a number of files. Now, the added wait for
the symlink changes to arrive gives time for additional add changes to
be processed, all within the same commit.
2012-06-15 19:00:06 -04:00
Joey Hess
8c7dfc93b5 catch IO exceptions in runThreadState
A few places catch IO errors after calling runThreadState,
but since the MVar was not restored, it'd later deadlock trying to read
from it.

I'd like to catch all exceptions here, but I could not get the types
to unify.
2012-06-15 19:00:06 -04:00
Joey Hess
8919c2e4da check for unstaged old symlinks in the sanity checker 2012-06-13 19:25:47 -04:00
Joey Hess
4b9b9b4947 add sanity checker thread
Currently wakes up once a day, and does nothing. :)
2012-06-13 17:54:23 -04:00
Joey Hess
7575c5bb56 tweak 2012-06-13 14:19:21 -04:00
Joey Hess
59a7b3a51a finish daemon status thread 2012-06-13 14:02:40 -04:00
Joey Hess
ff2414427b implement daemon status serialization to a file
Also afterLastDaemonRun, with 10 minute slop to handle majority of clock
skew issues.
2012-06-13 13:35:15 -04:00
Joey Hess
ccc5005245 reorganize 2012-06-13 12:46:39 -04:00