Commit graph

601 commits

Author SHA1 Message Date
Joey Hess
791c8535b5 fix stale git locks as part of repo repair 2013-10-29 13:52:19 -04:00
Joey Hess
fabb0c50b7 move code around and rename thread; no functional changes 2013-10-29 13:41:44 -04:00
Joey Hess
a7821c0581 automatically launch git repository repair
Added a RemoteChecker thread, that waits for problems to be reported with
remotes, and checks if their git repository is in need of repair.

Currently, only failures to sync with the remote cause a problem to be
reported. This seems enough, but we'll see.

Plugging in a removable drive with a repository on it that is corrupted
does automatically repair the repository, as long as the corruption causes
git push or git pull to fail. Some types of corruption do not, eg
missing/corrupt objects for blobs that git push doesn't need to look at.

So, this is not really a replacement for scheduled git repository fscking.
But it does make the assistant more robust.

This commit is sponsored by Fernando Jimenez.
2013-10-27 16:42:13 -04:00
Joey Hess
7ed8e87a34 assistant: Support repairing git remotes that are locally accessible
(eg, on removable drives)

gcrypt remotes are not yet handled.

This commit was sponsored by Sören Brunk.
2013-10-27 15:38:59 -04:00
Joey Hess
b48aaa22d0 assistant: Automatically repair damanged git repository, if it can be done without losing data. 2013-10-26 17:16:29 -04:00
Joey Hess
a1b1b5ef52 moved code out of webapp
No code changes, aside from some changes to lifting in code that turned out
to be able to run in Assistant rather than Handler.
2013-10-26 16:58:16 -04:00
Joey Hess
2233ddd5a2 assistant: When autostarted, wait 5 seconds before running the startup scan, to avoid contending with the user's desktop login process. 2013-10-26 12:42:58 -04:00
Joey Hess
1eaec2f9aa UI tweaks 2013-10-22 16:30:23 -04:00
Joey Hess
d345e5b52f add git fsck to cronner, and UI for repository repair (not yet wired up) 2013-10-22 16:02:52 -04:00
Joey Hess
4f871f89ba git-recover-repository 1/2 done 2013-10-20 17:50:51 -04:00
Joey Hess
7a9daefea2 update for LsTree type change
in the config monitor, we want files relative to the top of the working directory
2013-10-17 14:51:39 -04:00
Joey Hess
25462f125d cronner: run jobs triggered by remotes becoming connected (untested) 2013-10-13 17:14:56 -04:00
Joey Hess
1ffb3bb0ba add remote fsck interface
Currently only implemented for local git remotes. May try to add support
to git-annex-shell for ssh remotes later. Could concevably also be
supported by some special remote, although that seems unlikely.

Cronner user this when available, and when not falls back to
fsck --fast --from remote

git annex fsck --from does not itself use this interface.
To do so, I would need to pass --fast and all other options that influence
fsck on to the git annex fsck that it runs inside the remote. And that
seems like a lot of work for a result that would be no better than
cd remote; git annex fsck
This may need to be revisited if git-annex-shell gets support, since it
may be the case that the user cannot ssh to the server to run git-annex
fsck there, but can run git-annex-shell there.

This commit was sponsored by Damien Diederen.
2013-10-11 16:03:18 -04:00
Joey Hess
e9745f2da2 add config page for fsck, and alert with button when a fsck is running 2013-10-10 18:05:53 -04:00
Joey Hess
18f4d1b400 queue downloads of keys that fsck finds with bad content 2013-10-10 17:27:00 -04:00
Joey Hess
82083658cf stop fsck when scheduled activity is removed 2013-10-10 16:22:55 -04:00
Joey Hess
6a331d1261 got delay calculation backwards 2013-10-10 12:55:30 -04:00
Joey Hess
5b70eac659 fix option name 2013-10-10 12:49:54 -04:00
Joey Hess
c80bc53960 cronner builds, should work (untested)
I probably need to improve handling of the PleaseTerminate exception to
kill the fsck process. Also, if fsck finds bad files, something needs
to requeue downloads of them. Otherwise, this should work, but is probably
quite buggy since I have only tested the pure code over the past 2 days.
2013-10-08 18:13:08 -04:00
Joey Hess
af5e1d0494 half way complete cronner thread to run scheduled activities 2013-10-08 11:48:28 -04:00
Joey Hess
45aed381df import: Skip .git directories. 2013-10-07 13:03:05 -04:00
Joey Hess
635c9a1549 assistant: Detect stale git lock files at startup time, and remove them.
Extends the index.lock handling to other git lock files. I surveyed
all lock files used by git, and found more than I expected. All are
handled the same in git; it leaves them open while doing the operation,
possibly writing the new file content to the lock file, and then closes
them when done.

The gc.pid file is excluded because it won't affect the normal operation
of the assistant, and waiting for a gc to finish on startup wouldn't be
good.

All threads except the webapp thread wait on the new startup sanity checker
thread to complete, so they won't try to do things with git that fail
due to stale lock files. The webapp thread mostly avoids doing that kind of
thing itself. A few configurators might fail on lock files, but only if the
user is explicitly trying to run them. The webapp needs to start
immediately when the user has opened it, even if there are stale lock
files.

Arranging for the threads to wait on the startup sanity checker was a bit
of a bear. Have to get all the NotificationHandles set up before the
startup sanity checker runs, or they won't see its signal. Perhaps
the NotificationBroadcaster is not the best interface to have used for
this. Oh well, it works.

This commit was sponsored by Michael Jakl
2013-10-05 17:04:21 -04:00
Joey Hess
93dbb7842e watcher: Detect at startup time when there is a stale .git/lock, and remove it so it does not interfere with the automatic commits of changed files. 2013-10-03 16:57:21 -04:00
Joey Hess
3ac9c4e672 hlint 2013-10-02 22:59:07 -04:00
Joey Hess
98fc7e8a19 add, import, assistant: Better preserve the mtime of symlinks, when when adding content that gets deduplicated.
Note that this turned out to remove a syscall, not add any expense.
Otherwise, I would not have done it.
2013-09-25 16:07:11 -04:00
Joey Hess
4dc4a9a385 assistant: Clear the list of failed transfers when doing a full transfer scan. This prevents repeated retries to download files that are not available, or are not referenced by the current git tree.
This is motivated by a user report that the assistant was repeatedly
retrying transfers of files that had been deleted (in direct mode, so
removing the only copy).

Note that the glacier code retries failed transfers after a while to retry
downloads that have aged long enough to be available. This is ok; if we're
doing a full transfer scan we'll retry on every file that is still in the
git tree.

Also note that this makes the assistant less likely to get every file
referenced by old revs of the git tree. Not something the assistant tries
to ensure anyway, so I feel this is acceptable.
2013-09-25 11:46:17 -04:00
Joey Hess
5fe49b98f8 Support hot-swapping of removable drives containing gcrypt repositories.
To support this, a core.gcrypt-id is stored by git-annex inside the git
config of a local gcrypt repository, when setting it up.

That is compared with the remote's cached gcrypt-id. When different, a
drive has been changed. git-annex then looks up the remote config for
the uuid mapped from the core.gcrypt-id, and tweaks the configuration
appropriately. When there is no known config for the uuid, it will refuse to
use the remote.
2013-09-12 15:54:35 -04:00
Joey Hess
9dc2373977 only retry every 60 seconds
Retying every second is a bit much, especially given the current leak
https://github.com/audreyt/network-multicast/issues/4
2013-08-24 14:37:34 -04:00
Joey Hess
8587485994 clarify notifyNetMessagerRestart 2013-08-24 13:49:04 -04:00
Joey Hess
b191d5c595 gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.

Thanks to Adam Spiers, for getting the necessary support into git for this.

A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.

However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.

Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!

So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.

(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)

This commit was sponsored by Francois Marier. Thanks!
2013-08-02 20:37:03 -04:00
Joey Hess
672cfc3923 better git version checking 2013-08-02 18:32:26 -04:00
Joey Hess
869c638b82 assistant: Fix bug that caused it to stall when adding a very large number of files at once (around 5 thousand).
This bug was introduced in 82a6db8fe8,
which improved handling of adding very large numbers of files by ensuring
that a minimum number of max size commits (5000 files each) were done.

I accidentially made it wait for another change to appear after such a max
size commit, even if a lot of queued changes were already accumulated.
That resulted in a stall when it got to the end. Now fixed to not wait
any longer than necessary to ensure the watcher has had time to wake back
up after the max size commit.

This commit was sponsored by Michael Linksvayer. Thanks!
2013-07-27 17:42:18 -04:00
Joey Hess
ec4d974dcf assistant: Fix deadlock that could occur when adding a lot of files at once in indirect mode.
This is a laziness problem. Despite the bang pattern on newfiles, the list
was not being fully evaluated before cleanup was called. Moving cleanup out
to after the list is actually used fixes this.

More evidence that I should be using ResourceT or pipes, if any was needed.
2013-07-26 18:42:22 -04:00
Joey Hess
97f3aecb17 assistant: Fix NetWatcher to not sync with remotes that have remote.<name>.annex-sync set to false.
This affected both the hourly NetWatcherFallback thread and the syncing
when network connection is detected.

It was a reversion of sorts, introduced in
8861e270be, when annex-ignore was changed to
not control git syncing. I forgot to make it check annex-sync at that
point.
2013-07-26 16:54:20 -04:00
Joey Hess
dba1e29949 webapp: Better display of added files. 2013-07-10 15:37:40 -04:00
Joey Hess
5e48aa4d4b assistant: Daily sanity check thread is run niced. 2013-06-21 13:29:42 -04:00
Joey Hess
6e309b63f8 assistant: On Linux, the expensive transfer scan is run niced.
This is a compromise. I would like to nice every thread except for the
webapp thread, but it's not practical to do so. That would need every
thread to run as a bound thread, which could add significant overhead.
And any forkIO would escape the nice level.
2013-06-20 22:25:41 -04:00
Joey Hess
c7b493fecb log local pairing messages received when debugging is enabled 2013-06-11 00:25:44 -04:00
Joey Hess
c46b263fde Android: Make the "Open webapp" menu item open the just created repository when a new repo is made. 2013-06-10 23:55:53 -04:00
Joey Hess
b42fe2283a remove unnecessary haskell extensions 2013-06-04 21:13:20 -04:00
Joey Hess
853ec68253 avoid debug logging unknown xmpp messages, which may contain sensative information 2013-05-27 15:00:07 -04:00
Joey Hess
ed4febb170 remove debug print 2013-05-25 00:02:39 -04:00
Joey Hess
99324eac9a XMPP: Send pings and use them to detect when contact with the server is lost.
I noticed that when my modem hung up and redialed, my xmpp client was left
sending messages into the void. This will also handle any idle
disconnection issues.
2013-05-22 17:03:46 -04:00
Joey Hess
e2b67e0bc4 add two long-running XMPP push threads, no more inversion of control
I hope this will be easier to reason about, and less buggy. It was
certianly easier to write!

An immediate benefit is that with a traversable queue of push requests to
select from, the threads can be a lot fairer about choosing which client to
service next.
2013-05-22 15:13:31 -04:00
Joey Hess
9efde46cdd per-client inboxes for push messages
This will avoid losing any messages received from 1 client when a push
involving another client is running.

Additionally, the handling of push initiation is improved,
it's no longer allowed to run multiples of the same type of push to
the same client.

Still stalls sometimes :(
2013-05-21 11:08:08 -04:00
Joey Hess
14d96b8e06 XMPP: Be better at responding to CanPush messages when busy with something else.
Observed: With 2 xmpp clients, one would sometimes stop responding
to CanPush messages. Often it was in the middle of a receive-pack
of its own (or was waiting for a failed one to time out).

Now these are always immediately responded to, which is fine; the point
of CanPush is to find out if there's another client out there that's
interested in our push.

Also, in queueNetPushMessage, queue push initiation messages when
we're already running the side of the push they would initiate.
Before, these messages were sent into the netMessagesPush channel,
which was wrong. The xmpp send-pack and receive-pack code discarded
such messages.

This still doesn't make XMPP push 100% robust. In testing, I am seeing
it sometimes try to run two send-packs, or two receive-packs at once
to the same client (probably because the client sent two requests).

Also, I'm seeing rather a lot of cases where it stalls out until it
runs into the 120 second timeout and cancels a push.

And finally, there seems to be a bug in runPush. I have logs that
show it running its setup action, but never its cleanup action.
How is this possible given its use of E.bracket? Either some exception
is finding its way through, or the action somehow stalls forever.
When this happens, one of the 2 clients stops syncing.
2013-05-21 00:59:38 -04:00
Joey Hess
b8e5b9c645 test suite passes in direct mode
This fixes a bug with git annex add in direct mode. If some files already
existed in the tree pointing at the same key as a file that was just added,
and their content was not present, add neglected to copy the content to
those files.

I also changed the behavior of moveAnnex slightly: When content is moved
into the annex in direct mode, it does not overwrite any content already
present in direct mode files. That content may be modified after all.
2013-05-17 15:59:37 -04:00
Joey Hess
25a8d4b11c rename module 2013-05-12 19:19:28 -04:00
Joey Hess
ce47f4cb20 --listen is not supported on Android 2013-05-02 16:47:42 -04:00
Joey Hess
50a8ea4cdc avoid auto-accepting pair requests from friends already paired with
Unless the request is for repo uuid we already know. This way, if A1 pairs
with friend B1, and B1 pairs with device B2, then B1 can request A1 pair
with it and no confirmation is needed. (In future, may want to try to do
that automatically, to make a more robust network.)
2013-04-30 17:38:58 -04:00
Joey Hess
e363cefcb3 assistant: Fix bug that could cause incoming pushes to not get merged into the local tree.
Observed that the pushed refs were received, but not merged into master.
The merger never saw an add event for these refs. Either git is not writing
to a new file and renaming it into place, or the inotify code didn't notice
that. Changed it to also watch for modify events and that seems to have
fixed it!
2013-04-30 16:37:13 -04:00
Joey Hess
e06a4e12b5 more xmpp debugging 2013-04-30 15:56:33 -04:00
Joey Hess
2810807ca5 Internet Archive!
* Add public repository group.
* webapp: Can now set up Internet Archive repositories.

TODO: Enabling IA repositories.
2013-04-25 12:23:36 -04:00
Joey Hess
c6da464051 use a DList for the deferred downloads queue 2013-04-25 01:26:23 -04:00
Joey Hess
82a6db8fe8 committer tweak to wait for Watcher to resume after a max-size commit
Without this, a very large batch add has commits of sizes approx
5000, 2500, 1250, etc down to 10, and then starts over at 5000.
This fixes it so it's 5000+ every time.
2013-04-25 00:48:09 -04:00
Joey Hess
a9081ae473 optimise direct mode startup scan
A recent change made existing symlinks be re-staged. That does not need to
be done during the startup scan though.
2013-04-24 21:20:29 -04:00
Joey Hess
46529c0129 assistant: Sanitize XMPP presence information logged for debugging. 2013-04-24 21:13:10 -04:00
Joey Hess
ebee93a837 get rid of need to run pre-commit hook when assistant commits in direct mode
That hook updates associated file bookkeeping info for direct mode.
But, everything already called addAssociatedFile when adding/changing a
file. It only needed to also call removeAssociatedFile when deleting a file,
or a directory.

This should make bulk adds faster, by some possibly significant amount.
Bulk removals may be a little slower, since it has to use catKeyFile now
on each removed file, but will still be faster than adds.
2013-04-24 18:04:59 -04:00
Joey Hess
b8e45ec9d7 refactoring and minor performance tweak 2013-04-24 17:46:46 -04:00
Joey Hess
7fa2d255da remove last use of TSet 2013-04-24 17:16:04 -04:00
Joey Hess
cd7055631f batch commit every 5 thousand changes, not 10 thousand
There's a tradeoff between making less frequent commits, and
needing to use memory to store all the changes that are coming
in. At 10 thousand, it needs 150 mb of memory. 5 thousand drops
that down to 90 mb or so.

This also turns out to have significant imact on total run time.
I benchmarked 10k changes taking 27 minutes. But two 5k batches
took only 21 minutes.
2013-04-24 16:40:35 -04:00
Joey Hess
bda237f14a convert PendingAddChange back to Change when an add fails
If an add failed, we should lose the KeySource, since it, presumably,
differs due to a change that was made to the file.

(The locked down file is already deleted.)
2013-04-24 16:29:25 -04:00
Joey Hess
a929e6641a show one alert when bulk adding files
Turns out that a lot of the time spent in a bulk add was just updating the
add alert to rotate through each file that was added. Showing one alert
makes for a significant speedup.

Also, when the webapp is open, this makes it take quite a lot less cpu
during bulk adds.

Also, it lets the user know when a bulk add happened, which is sorta
nice..
2013-04-24 13:04:46 -04:00
Joey Hess
ca72b1ac7b assistant: when an add fails, requeue it for later
See analysis in bug report for one way this could happen.
2013-04-23 18:23:04 -04:00
Joey Hess
2c42e70f6c rename module 2013-04-23 11:38:52 -04:00
Joey Hess
8861e270be sync, assistant: Sync with remotes that have annex-ignore set
This is so git remotes on servers without git-annex installed can be used
to keep clients' git repos in sync.

This is a behavior change, but since annex-sync can be set to disable
syncing with a remote, I think it's acceptable.
2013-04-22 14:57:09 -04:00
Joey Hess
780ed8ebf1 fix warning 2013-04-20 18:52:46 -04:00
Joey Hess
090a69f00c assistant: Work around misfeature in git 1.8.2 that makes git commit --alow-empty -m "" run an editor.
See http://git-annex.branchable.com/bugs/assistant_hangs_during_commit/
2013-04-18 16:27:17 -04:00
Joey Hess
824c95376a refactor static route definition -- android webapp build fix
Incidentially should work around the last problem that prevented the webapp
building on Android. (Except for a few places I need to clean up after
hand-fixing the spliced TH code.)
2013-04-17 01:37:08 -04:00
Joey Hess
2c365b8b74 remove debug 2013-04-11 16:36:45 -04:00
Joey Hess
1eb3fff787 addurl: Register transfer so the webapp can see it.
* addurl: Register transfer so the webapp can see it.
* addurl: Automatically retry downloads that fail, as long as some
  additional content was downloaded.
2013-04-11 16:14:17 -04:00
Joey Hess
04a27ad926 assistant: Bug fix to avoid annexing the files that git uses to stand in for symlinks on FAT and other filesystem not supporting symlinks.
also, blog for the day..
2013-04-10 19:57:26 -04:00
Joey Hess
5e2e4347a3 webapp: New --listen= option allows running the webapp on one computer and connecting to it from another.
Does not yet use HTTPS. I'd need to generate a certificate, and I'm not
sure what's the best way to do that.
2013-04-08 15:04:35 -04:00
Joey Hess
602baae12e Bugfix: Direct mode no longer repeatedly checksums duplicated files.
Fixed by storing a list of cached inodes for a key, instead of just one.

Backwards compatability note: An old git-annex version will fail to parse
an inode cache file that has been written by a new version, and has
multiple items. It will succees if just one. So old git-annexes will have
even worse behavior when there are duplicated files, if that is possible.
I don't think it will be a problem. (Famous last words.)

Also, note that it doesn't expire old and unused inode caches for a key.
It would be possible to add this if needed; just look through the
associated files for a key and if there are more cached inodes, throw out
any not corresponding to associated files. Unless a file is being copied
repeatedly and the old copy deleted, this lack of expiry should not be a
problem.
2013-04-06 16:07:25 -04:00
Joey Hess
f1b0a4b404 Use lower case hash directories for storing files on crippled filesystems, same as is already done for bare repositories.
* since this is a crippled filesystem anyway, git-annex doesn't use
  symlinks on it
* so there's no reason to use the mixed case hash directories that we're
  stuck using to avoid breaking everyone's symlinks to the content
* so we can do what is already done for all bare repos, and make non-bare
  repos on crippled filesystems use the all-lower case hash directories
* which are, happily, all 3 letters long, so they cannot conflict with
  mixed case hash directories
* so I was able to 100% fix this and even resuming `git annex add` in the
  test case will recover and it will all just work.
2013-04-04 15:46:33 -04:00
Joey Hess
8b329c0317 refactor alert button creation code 2013-04-04 01:48:26 -04:00
Joey Hess
2c3aeec19b check for unused keys on an unwanted remote, and move them off, before deleting it 2013-04-03 19:03:16 -04:00
Joey Hess
021c564319 clean up urlrenderer handling when the webapp is not built 2013-04-03 17:48:54 -04:00
Joey Hess
9a5f421768 detect when unwanted remote is empty and remove it
Needs fixes to build when the webapp is disabled.
2013-04-03 17:01:40 -04:00
Joey Hess
47950cdf31 more efficient uuid to remote lookup 2013-04-02 16:39:11 -04:00
Joey Hess
6e7842475b convert "./file" from inotify to just "file"
This just prettifies some display.
2013-04-02 16:20:23 -04:00
Joey Hess
38d61f934d Update working tree files fully atomically
This avoids commit churn by the assistant when eg,
replacing a file with a symlink.

But, just as importantly, it prevents the working tree being left with a
deleted file if git-annex, or perhaps the whole system, crashes at the
wrong time.

(It also probably avoids confusing displays in file managers.)
2013-04-02 15:02:00 -04:00
Joey Hess
8c52b20cc7 optimise last commit
Rather than re-adding a direct mode file unnecessarily when it's not
changed, just re-stage the symlink.
2013-04-02 12:58:56 -04:00
Joey Hess
31cbde8190 assistant: Fix bug that could cause direct mode files to be unstaged from git.
My test case for this bug is to have the assistant running and syncing to
a remote, and create a file in the annex. Then at the command line run
git annex drop. The assistant sees that the file is gone, sees it's a wanted
file, and downloads it from the remote.

With a directory special remote and a small file, I was seeing around 1
time in 3, a race where the file got unstaged from git after it got
downloaded.

Looking at what direct mode content managing code does in this case, it
deletes the symlink, and then adds the file content back. It would be
possible, sometimes, to avoid removing the symlink and do this atomically.
And I probably should.. but in some cases, particularly where the file
needs to be run through `cp` (multiple direct mode files with same
content), there's no way to atomically replace the symlink with the
content.

Anyway, the bug turns out to be something that the watcher does right for
indirect mode, but not for direct mode. When it got an add event, it
checked to see if this was a new file, or one we've already added. In the
latter case, no add event was queued. But that means that only the rm event
is queued, and so it unstages the file.

Fixed by queueing an add event even when the file is already in git.

Tested by running hundreds of drops in a loop; file remained staged.
2013-04-02 12:45:31 -04:00
Joey Hess
c57baaaa30 webapp: Added UI to delete repositories. Closes: #689847 2013-03-31 16:38:05 -04:00
Joey Hess
5771cfce02 assistant: Check small files into git directly. 2013-03-29 16:54:59 -04:00
Joey Hess
67e817c6a1 New annex.largefiles setting, which configures which files git annex add and the assistant add to the annex.
I would have sort of liked to put this in .gitattributes, but it seems
it does not support multi-word attribute values. Also, making this a single
config setting makes it easy to only parse the expression once.

A natural next step would be to make the assistant `git add` files that
are not annex.largefiles. OTOH, I don't think `git annex add` should
`git add` such files, because git-annex command line tools are
not in the business of wrapping git command line tools.
2013-03-29 16:17:13 -04:00
Joey Hess
577128e9b8 avoid removing a transfer being made by another process
When another process is running a transfer, if we try to run a dup it'll
fail, and we should not remove the transfer from the transfer display.
2013-03-28 15:16:45 -04:00
Joey Hess
1d0b692198 webapp: Fix a race that sometimes caused alerts or other notifications to be missed if they occurred while a page was loading.
When a page is loaded, the javascript requests an notification url, and
does long polling on the url to be informed of changes. But if a change
occured before the notification url was requested, it would not be notified
of that change, and so the page display would not update.

I fixed this by *always* updating the page display after it gets
the notification url. This is extra work, but the overhead is not noticable
in the other overhead of loading a page.

(A nicer way would be to somehow record the version of a page initially
loaded, and then compare it with the current version when getting the
notification url, and only force an update if it's changed. But getting
the "version" of the different parts of the page that use long polling
is difficult.)
2013-03-27 14:56:20 -04:00
Joey Hess
3ef6b6a200 fix build with xmpp and w/o webapp 2013-03-24 18:55:19 -04:00
Joey Hess
b6d691aff7 maintain pools of running transferkeys processes (untested) 2013-03-19 18:46:29 -04:00
Joey Hess
bb284cd980 move display of transfer scan in progress to transfers section of dashboard
This way it's only visible when transfers are not running, which is about
what a user would expect.
2013-03-19 13:03:41 -04:00
Joey Hess
a30768cf7f new alert while scanning
Like the old one, but does not mention which remotes are scanned.
I think this is less confusing, as it does not imply the remotes
were somehow accessed (which they are not; inaccessible remotes
can be scanned.)
2013-03-18 23:15:48 -04:00
Joey Hess
aadb9069b3 deal with transferkey crashing
If transferkey crashes or even fails to run, the TransferWatcher will not
see the transfer info file be created, so will not remove the transfer
from the list of active transfers. This causes the list to grow
continually, and all active transfers are displayed in the webapp. So, put
in a guard.

I assume that transferkey will not exit 0 while neglecting to clean up.
2013-03-18 22:58:14 -04:00
Joey Hess
29d603b72a ensure 2013-03-18 22:41:16 -04:00
Joey Hess
b543842a7f optimisation for transfers to drives that are not plugged in
Rather than forking a git-annex transferkey only to have it fail,
just immediately record the failed transfer (so when the drive is plugged
in, the scan will retry it).
2013-03-18 20:40:24 -04:00
Joey Hess
cdb21649d0 webapp: Improved alerts displayed when syncing with remotes, and when syncing with a remote fails. 2013-03-18 17:23:47 -04:00
Joey Hess
35a0ae334c assistant: Fix OSX bug that prevented committing changed files to a repository when in indirect mode. 2013-03-17 17:01:43 -04:00
Joey Hess
746ffa773a assistant: Avoid syncing with annex-ignored remotes when reconnecting to the network, or connecting a drive. 2013-03-17 15:59:03 -04:00
Joey Hess
db2fe522ba xmpp: Re-enable XA flag, since disabling it did not turn out to help with the problems Google Talk has with not always sending presence messages to clients. 2013-03-16 16:00:37 -04:00
Joey Hess
55f20ae099 xmpp: send a presence query when there's an important message to send
This may work around google talk's horrible presence handling, in which
clients often don't learn about other clients, at least when using the same
account. This way, every time we start a git push over xmpp, we'll waste
bandwidth asking clients to please try again to identify themselves.
2013-03-16 15:36:47 -04:00
Joey Hess
e3354cf19c xmpp: --debug now enables a sanitized dump of the XMPP protocol
So I can debug these damn google talk presence issues.
2013-03-16 15:29:51 -04:00
Joey Hess
c94c99942b make liftAnnex and liftAssistant polymorphic, like liftIO 2013-03-16 00:19:56 -04:00
Joey Hess
77c82de4ea webapp: Display an alert when there are XMPP remotes, and a cloud transfer repository needs to be configured. 2013-03-15 17:52:41 -04:00
Joey Hess
39e979fb65 webapp: Improved UI for pairing your own devices together using XMPP. 2013-03-15 15:19:07 -04:00
Joey Hess
9cf4701a8f no longer need webapp state storage! excellent 2013-03-15 01:01:25 -04:00
Joey Hess
02facde154 assistant: Be smarter about avoiding unncessary transfers.
Just before starting a transfer, do one last check that it's still
preferred content.

I was just doing this for uploads, as part of the smarter flood filling
bug, but realized it's also possible for a download that was preferred
content to change to not be before the download begins, so check that too.
2013-03-13 13:36:02 -04:00
Joey Hess
60760cb430 tweak 2013-03-13 13:11:49 -04:00
Joey Hess
0ef8d806ac gratuitous rename HomeR -> DashboardR 2013-03-12 22:18:36 -04:00
Joey Hess
8221c2b4ed split repolist out of configuration, into its own tab (temporarily) 2013-03-12 21:51:03 -04:00
Joey Hess
393340dc3b better handling of batch renames
Rather than wait a full second, which may be longer than needed, or too
short to get all the rename events, we start a mode where we wait 1/10th of
a second, and if there are Changes received, wait again. Basically we're
back in batch mode when this happens.
2013-03-11 15:46:09 -04:00
Joey Hess
14fcfced48 detect directory rename and wait up to 1 second to get all the changes 2013-03-11 15:24:13 -04:00
Joey Hess
f340fd324c synthesize RmChange when a directory is deleted
This gets directory renames closer to being fully detected.
There's close to no extra overhead to doing it this way.
2013-03-11 15:14:42 -04:00
Joey Hess
06046a0d2b finish fast direct mode rename handling. wow, it's fast 2013-03-11 14:14:45 -04:00
Joey Hess
87cba71d5a fix changeFile to not be partial
That led to runtime crashes, without even a warning from -Wall. Yipes!
2013-03-11 13:55:36 -04:00
Joey Hess
61c5e8736c detect renames during commit, and .. um, do nothing special because it's lunch time
But I'm well set up to fast-track direct mode adds for renames now.
2013-03-11 12:56:47 -04:00
Joey Hess
74f723bb50 let's put type modules under the parent module, not in a Types directory 2013-03-10 22:24:13 -04:00
Joey Hess
2762ab03b4 assistant: generate better commits for renames 2013-03-10 22:10:26 -04:00
Joey Hess
b2c7ee5551 tweak 2013-03-10 20:20:58 -04:00
Joey Hess
f27c21eb0c avoid ugly alert caused by trying to push to unavailable removable drive 2013-03-10 18:42:28 -04:00
Joey Hess
65a4c7966f moved transfer queueing out of watcher and into committer
This cleaned up the code quite a bit; now the committer just looks at the
Change to see if it's a change that needs to have a transfer queued for it.
If I later want to add dropping keys for files that were removed, or
something like that, this should make it straightforward.

This also fixes a bug. In direct mode, moving a file out of an archive
directory failed to start a transfer to get its content. The problem
was that the file had not been committed to git yet, and so the transfer
code didn't want to touch it, since fileKey failed to get its key.
Only starting transfers after a commit avoids this problem.
2013-03-10 18:16:03 -04:00
Joey Hess
0e508f860a assistant: Sync with all git remotes on startup. 2013-03-08 13:48:27 -04:00
Joey Hess
243e99717d empty buddy list when client is connecting
This is not perfect, because on loss of connection, we do not currently
immediately detect it and stop the client. It has to time out, and then
the buddy list will clear.

The NetWatcher should detect disconnects too..
2013-03-07 03:50:21 -04:00
Joey Hess
f8c2dc82d8 show when not connected to xmpp server 2013-03-06 22:02:47 -04:00
Joey Hess
c16adc25c4 assistant: XMPP git pull and push requests are cached and sent when presence of a new client is detected.
Noticed that, At startup or network reconnect, git push messages were sent,
often before presence info has been gathered, so were not sent to any
buddies.

To fix this, keep track of which buddies have seen such messages,
and when new presence is received from a buddy that has not yet seen it,
resend.

This is done only for push initiation messages, so very little data needs
to be stored.
2013-03-06 21:38:01 -04:00
Joey Hess
060119fdc4 better xmpp debugging 2013-03-06 18:28:34 -04:00
Joey Hess
aaec2cbf03 avoid false alert about syncing with xmpp remote 2013-03-06 17:54:45 -04:00
Joey Hess
cbb6e1fae4 tag xmpp pushes with jid
This fixes the issue mentioned in the last commit.

Turns out just collecting UUID of clients behind a XMPP remote is
insufficient (although I should probably still do it for other reasons),
because a single remote repo might be connected via both XMPP and local
pairing. So a way is needed to know when a push was received from any
client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
Joey Hess
c23ea9e311 assistant: Get back in sync with XMPP remotes after network reconnection, and on startup.
Make manualPull send push requests over XMPP.

When reconnecting with remotes, those that are XMPP remotes cannot
immediately be pulled from and scanned, so instead maintain a set of
(probably) desynced remotes, and put XMPP remotes on it. (This set could be
used in other ways later, if we can detect we're out of sync with other
types of remotes.)

The merger handles detecting when a XMPP push is received from a desynced
remote, and triggers a scan then, if they have in fact diverged.

This has one known bug: A single XMPP remote can have multiple clients
behind it. When this happens, only the UUID of one client is recorded
as the UUID of the XMPP remote. Pushes from the other XMPP clients will not
trigger a scan. If the client whose UUID is expected responds to the push
request, it'll work, but when that client is offline, we're SOL.
2013-03-06 15:09:31 -04:00
Joey Hess
907b0c0d78 better liftAnnex, avoid using runAnnex undefined 2013-03-04 16:36:38 -04:00
Joey Hess
c908672f3d fix another potential race with the watcher and direct mode
Watcher wants to rewrite symlink to fix it. But in direct mode, the symlink
could be replaced at any time with file content that has finished being
transferred by some other process. So, just don't touch it.

FWIW, I audited the rest of the assistant for places where it removes
files, and the rest is ok. I have not audited the rest of git-annex.
2013-03-04 15:09:32 -04:00
Joey Hess
1d388d5579 fixed the race breaking moving files from archive in direct mode
assistant: Fix bug in direct mode that could occur when a symlink is moved
out of an archive directory, and resulted in the file not being set to
direct mode when it was transferred.

The bug was that the direct mode mapping was not up-to-date when the
transferrer finished. So, finding no direct mode place to store the object,
it was put into .git/annex in indirect mode.

To fix this, just make the watcher update the direct mode mapping to
include the new file before it starts the transfer. (Seems we don't need to
update it to remove the old file if the link was moved, because the direct
mode code will notice it's not present and the mapping gets updated for its
removal later.)

The reason this was a race, and was probably not seen often is because
the committer came along and updated the direct mode mapping as part of
adding the moved symlink. But when the file was sufficiently small or
the remote sufficiently fast, this could happen after the transfer
finished.
2013-03-04 14:25:22 -04:00
Joey Hess
08bdea7e52 webapp: New preferences page allows enabling/disabling debug logging at runtime, as well as configuring numcopies and diskreserve. 2013-03-03 17:07:27 -04:00
Joey Hess
724711e4b7 fix 2013-03-03 15:18:24 -04:00
Joey Hess
789ca15012 better prevention of auto repack
Looking through the git sources (documentation is unclear),
it seems commit doesn't ever trigger git-gc, mostly fetching and merging
seems to. I cannot easily override the setting in all those places, so
instead set gc.auto in git config when initializing a repository with
the assistant.

This does mean that the user cannot set gc.auto=0 and completely avoid
repacks, as the assistant does it daily. But, it only does it after there
are 100x the default number of loose objects, so this is probably not going
to be too annoying.
2013-03-03 14:20:07 -04:00
Joey Hess
6dea43831e assistant: Prevent automatic commits from causing git-gc runs, as that can make things quite slow. Instead, git-gc --auto is run once a day. (This can be disabled by the usual gc.auto=0 setting.) 2013-03-03 13:44:35 -04:00
Joey Hess
0c13d3065e git subcommand cleanup
Pass subcommand as a regular param, which allows passing git parameters
like -c before it. This was already done in the pipeing set of functions,
but not the command running set.
2013-03-03 13:39:07 -04:00
Joey Hess
7a8833c81d remove excess log rotation; openLog rotates 2013-03-01 16:56:56 -04:00
Joey Hess
0d4b513ec2 assistant: Fix dropping content when a file is moved to an archive directory.
A transfer is queued, but if the file has already been transferred to the
remote before, the transfer is skipped. In this case, it needs to perform
any actions it would normally take after finishing the transfer, like
dropping the local object.
2013-03-01 16:46:36 -04:00
Joey Hess
4d33423067 assistant: Avoid noise in logs from git commit about typechanged files in direct mode repositories. 2013-03-01 16:21:29 -04:00
Joey Hess
a733271a9c add additional debug info about reasons for drops 2013-03-01 15:58:44 -04:00
Joey Hess
46c9cbeb1e add additional debug info about reasons for transfers 2013-03-01 15:23:59 -04:00
Joey Hess
1865b28094 assistant: Logs are rotated to avoid them using too much disk space.
This cannot completely guard against a runaway log event, and only runs
every hour anyway, but it should avoid most problems with very
long-running, active assistants using up too much space.
2013-03-01 13:30:48 -04:00
Joey Hess
08854afa10 fix inverted logic 2013-02-22 17:01:48 -04:00
Joey Hess
2a4dad8bd4 remove debug prints 2013-02-19 23:18:15 -04:00
Joey Hess
d7c93b8913 fully support core.symlinks=false in all relevant symlink handling code
Refactored annex link code into nice clean new library.

Audited and dealt with calls to createSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:  ( liftIO $ createSymbolicLink linktarget file
  only when core.symlinks=true
Assistant/WebApp/Configurators/Local.hs:                createSymbolicLink link link
  test if symlinks can be made
Command/Fix.hs: liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/FromKey.hs:     liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/Indirect.hs:                    liftIO $ createSymbolicLink l f
  refuses to run if core.symlinks=false
Init.hs:                createSymbolicLink f f2
  test if symlinks can be made
Remote/Directory.hs:    go [file] = catchBoolIO $ createSymbolicLink file f >> return True
  fast key linking; catches failure to make symlink and falls back to copy
Remote/Git.hs:          liftIO $ catchBoolIO $ createSymbolicLink loc file >> return True
  ditto
Upgrade/V1.hs:                          liftIO $ createSymbolicLink link f
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to readSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:		( liftIO $ catchMaybeIO $ readSymbolicLink file
  only when core.symlinks=true
Assistant/Threads/Watcher.hs:		ifM ((==) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file))
  code that fixes real symlinks when inotify sees them
  It's ok to not fix psdueo-symlinks.
Assistant/Threads/Watcher.hs:		mlink <- liftIO (catchMaybeIO $ readSymbolicLink file)
  ditto
Command/Fix.hs:	stopUnless ((/=) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file)) $ do
  command only works in indirect mode
Upgrade/V1.hs:	getsymlink = takeFileName <$> readSymbolicLink file
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to isSymbolicLink.
(Typically used with getSymbolicLinkStatus, but that is just used because
getFileStatus is not as robust; it also works on pseudolinks.)
Remaining calls are all safe, because:

Assistant/Threads/SanityChecker.hs:                             | isSymbolicLink s -> addsymlink file ms
  only handles staging of symlinks that were somehow not staged
  (might need to be updated to support pseudolinks, but this is
  only a belt-and-suspenders check anyway, and I've never seen the code run)
Command/Add.hs:         if isSymbolicLink s || not (isRegularFile s)
  avoids adding symlinks to the annex, so not relevant
Command/Indirect.hs:                            | isSymbolicLink s -> void $ flip whenAnnexed f $
  only allowed on systems that support symlinks
Command/Indirect.hs:            whenM (liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f) $ do
  ditto
Seek.hs:notSymlink f = liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f
  used to find unlocked files, only relevant in indirect mode
Utility/FSEvents.hs:                    | Files.isSymbolicLink s = runhook addSymlinkHook $ Just s
Utility/FSEvents.hs:                                            | Files.isSymbolicLink s ->
Utility/INotify.hs:                             | Files.isSymbolicLink s ->
Utility/INotify.hs:                     checkfiletype Files.isSymbolicLink addSymlinkHook f
Utility/Kqueue.hs:              | Files.isSymbolicLink s = callhook addSymlinkHook (Just s) change
  all above are lower-level, not relevant

Audited and dealt with calls to isSymLink.
Remaining calls are all safe, because:

Annex/Direct.hs:			| isSymLink (getmode item) =
  This is looking at git diff-tree objects, not files on disk
Command/Unused.hs:		| isSymLink (LsTree.mode l) = do
  This is looking at git ls-tree, not file on disk
Utility/FileMode.hs:isSymLink :: FileMode -> Bool
Utility/FileMode.hs:isSymLink = checkMode symbolicLinkMode
  low-level

Done!!
2013-02-17 16:43:14 -04:00
Joey Hess
630f4531a7 fix assistant's use of lsof in crippled filesystem mode 2013-02-15 13:08:22 -04:00
Joey Hess
47477b2807 crippled filesystem support, probing and initial support
git annex init probes for crippled filesystems, and sets direct mode, as
well as `annex.crippledfilesystem`.

Avoid manipulating permissions of files on crippled filesystems.
That would likely cause an exception to be thrown.

Very basic support in Command.Add for cripped filesystems; avoids the lock
down entirely since doing it needs both permissions and hard links.
Will make this better soon.
2013-02-14 14:15:26 -04:00
Joey Hess
5737c49804 support Android's crippled lsof 2013-02-11 17:33:45 -04:00
Joey Hess
547d7745fb pre-commit: Update direct mode mappings.
Making the pre-commit hook look at git diff-index to find changed direct
mode files and update the mappings works pretty well.

One case where it does not work is when a file is git annex added, and then
git rmed, and then this is committed. That's a no-op commit, so the hook
probably doesn't even run, and it certianly never notices that the file
was deleted, so the mapping will still have the original filename in it.

For this and other reasons, it's important that the mappings still be
treated as possibly inconsistent.

Also, the assistant now allows the pre-commit hook to run when in direct
mode, so the mappings also get updated there.
2013-02-06 12:44:19 -04:00