Commit graph

962 commits

Author SHA1 Message Date
Joey Hess
b42fe2283a remove unnecessary haskell extensions 2013-06-04 21:13:20 -04:00
Joey Hess
1198b5444d now builds with both yesod 1.2 and 1.1 2013-06-03 18:14:22 -04:00
Joey Hess
31753bad46 add liftH shim between yesod versions, to avoid needing zillions of ifdefs 2013-06-03 13:51:54 -04:00
Joey Hess
79fd677805 WIP yesod 1.2 2013-06-03 11:25:23 -04:00
Joey Hess
853ec68253 avoid debug logging unknown xmpp messages, which may contain sensative information 2013-05-27 15:00:07 -04:00
Joey Hess
d8d46c3ba3 when xmpp connection fails, show the host(s) it tried to connect to 2013-05-27 14:36:20 -04:00
Joey Hess
5a678244e2 avoid duplicate srv lookup when testing xmpp 2013-05-27 13:48:24 -04:00
Joey Hess
c15a0b796e 3 more fd leaks 2013-05-26 21:27:44 -04:00
Joey Hess
c092b4fc16 XMPP: Fix a file descriptor leak. 2013-05-26 20:49:16 -04:00
Joey Hess
bd8f358e6d typo 2013-05-26 17:12:03 -04:00
Joey Hess
f857c44b45 avoid redundant CanPush messages with different shas being queued up 2013-05-26 17:09:23 -04:00
Joey Hess
ed4febb170 remove debug print 2013-05-25 00:02:39 -04:00
Joey Hess
99324eac9a XMPP: Send pings and use them to detect when contact with the server is lost.
I noticed that when my modem hung up and redialed, my xmpp client was left
sending messages into the void. This will also handle any idle
disconnection issues.
2013-05-22 17:03:46 -04:00
Joey Hess
4effef3176 fix minor memory leak caused by recent CanPush change
Putting the UUID in meant that equivilant CanPush messages no longer are ==
2013-05-22 15:47:06 -04:00
Joey Hess
e2b67e0bc4 add two long-running XMPP push threads, no more inversion of control
I hope this will be easier to reason about, and less buggy. It was
certianly easier to write!

An immediate benefit is that with a traversable queue of push requests to
select from, the threads can be a lot fairer about choosing which client to
service next.
2013-05-22 15:13:31 -04:00
Joey Hess
cc68b340ff avoid building up a lot of threads all waiting for their chance to run a push
Only 2 threads are needed, one running, and one waiting to push something
new. Any more is redundant and wasteful.
2013-05-22 00:27:12 -04:00
Joey Hess
6e04194e84 perhaps this fixes the stall tbd 2013-05-21 21:40:37 -04:00
Joey Hess
a600471a23 include HEAD in CanPush shas 2013-05-21 20:04:38 -04:00
Joey Hess
08c03b2af3 XMPP: Avoid redundant and unncessary pushes. Note that this breaks compatibility with previous versions of git-annex, which will refuse to accept any XMPP pushes from this version. 2013-05-21 18:24:29 -04:00
Joey Hess
9efde46cdd per-client inboxes for push messages
This will avoid losing any messages received from 1 client when a push
involving another client is running.

Additionally, the handling of push initiation is improved,
it's no longer allowed to run multiples of the same type of push to
the same client.

Still stalls sometimes :(
2013-05-21 11:08:08 -04:00
Joey Hess
14d96b8e06 XMPP: Be better at responding to CanPush messages when busy with something else.
Observed: With 2 xmpp clients, one would sometimes stop responding
to CanPush messages. Often it was in the middle of a receive-pack
of its own (or was waiting for a failed one to time out).

Now these are always immediately responded to, which is fine; the point
of CanPush is to find out if there's another client out there that's
interested in our push.

Also, in queueNetPushMessage, queue push initiation messages when
we're already running the side of the push they would initiate.
Before, these messages were sent into the netMessagesPush channel,
which was wrong. The xmpp send-pack and receive-pack code discarded
such messages.

This still doesn't make XMPP push 100% robust. In testing, I am seeing
it sometimes try to run two send-packs, or two receive-packs at once
to the same client (probably because the client sent two requests).

Also, I'm seeing rather a lot of cases where it stalls out until it
runs into the 120 second timeout and cancels a push.

And finally, there seems to be a bug in runPush. I have logs that
show it running its setup action, but never its cleanup action.
How is this possible given its use of E.bracket? Either some exception
is finding its way through, or the action somehow stalls forever.
When this happens, one of the 2 clients stops syncing.
2013-05-21 00:59:38 -04:00
Joey Hess
230aed671f XMPP: Ignore duplicate messages received when pushing. 2013-05-20 21:58:59 -04:00
Joey Hess
345ee4f37c Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
  .. Which can result in cached values being redundantly calculated, or other
     possibly worse bugs when the annex state gets out of sync with reality.

This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.

The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.

This is a deep level change. It passes the test suite! What could it break?

Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.

Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 14:16:36 -04:00
Joey Hess
b8e5b9c645 test suite passes in direct mode
This fixes a bug with git annex add in direct mode. If some files already
existed in the tree pointing at the same key as a file that was just added,
and their content was not present, add neglected to copy the content to
those files.

I also changed the behavior of moveAnnex slightly: When content is moved
into the annex in direct mode, it does not overwrite any content already
present in direct mode files. That content may be modified after all.
2013-05-17 15:59:37 -04:00
Joey Hess
25a8d4b11c rename module 2013-05-12 19:19:28 -04:00
Joey Hess
92910f5107 fix build 2013-05-12 19:13:38 -04:00
Joey Hess
1e2ddcb68a use setCurrentDirectory
On POSIX, this just calls changeWorkingDirectory.
2013-05-11 19:14:30 -04:00
Joey Hess
18bdff3fae clean up from windows porting 2013-05-11 18:23:41 -04:00
Joey Hess
7c2a90b0e8 fix for Android 2013-05-09 15:23:40 -04:00
Joey Hess
d63a0a77f2 fix build w/o DAV 2013-05-08 12:20:10 -04:00
Joey Hess
69bd8d0462 fix android build 2013-05-06 16:45:49 -04:00
Joey Hess
749c197947 fix use of wrong shebang when android is installing git-annex-shell wrapper on server 2013-05-06 15:58:13 -04:00
Joey Hess
ad6fee422a fix warning 2013-05-04 16:36:51 -04:00
Joey Hess
001d82f953 fix warning 2013-05-04 16:35:54 -04:00
Joey Hess
d35132810a Got removable media mount detection working on Android.
Bionic has an amusing stub for `getmntent` that prints out
"FIX ME! implement getmntent()"

But, `/proc/mounts` is there, so I just parse it.
2013-05-04 16:19:25 -04:00
Joey Hess
d994f35937 disable hostname validation and canonicalization on android 2013-05-04 14:26:49 -04:00
Joey Hess
33334bfe2e added special camera repository to start screen for android 2013-05-03 15:21:31 -04:00
Joey Hess
ce47f4cb20 --listen is not supported on Android 2013-05-02 16:47:42 -04:00
Joey Hess
e92f04f9cc build fix for -XMPP 2013-05-02 13:04:49 -04:00
Joey Hess
d9e288b0fd fix build with S3 disabled (for Android) 2013-05-01 13:27:51 -04:00
Joey Hess
50a8ea4cdc avoid auto-accepting pair requests from friends already paired with
Unless the request is for repo uuid we already know. This way, if A1 pairs
with friend B1, and B1 pairs with device B2, then B1 can request A1 pair
with it and no confirmation is needed. (In future, may want to try to do
that automatically, to make a more robust network.)
2013-04-30 17:38:58 -04:00
Joey Hess
e363cefcb3 assistant: Fix bug that could cause incoming pushes to not get merged into the local tree.
Observed that the pushed refs were received, but not merged into master.
The merger never saw an add event for these refs. Either git is not writing
to a new file and renaming it into place, or the inotify code didn't notice
that. Changed it to also watch for modify events and that seems to have
fixed it!
2013-04-30 16:37:13 -04:00
Joey Hess
e06a4e12b5 more xmpp debugging 2013-04-30 15:56:33 -04:00
Joey Hess
017f62654f stop using UltDest, doesn't seem to reliably work 2013-04-30 15:21:49 -04:00
Joey Hess
df88c51334 add uuid to all xmpp messages
(Except for the actual streaming of receive-pack through XMPP, which
can only run once we've gotten an appropriate uuid in a push initiation
message.)

Pushes are now only initiated when the initiation message comes from a
known uuid. This allows multiple distinct repositories to use the same xmpp
address.

Note: This probably breaks initial push after xmpp pairing, because at that
point we may not know about the paired uuid, and so reject the push from
it. It won't break in simple cases, because the annex-uuid of the remote
is checked. However, when there are multiple clients behind a single xmpp
address, only uuid of the first is recorded in annex-uuid, and so any
pushes from the others will be rejected (unless the first remote pushes their
uuids to us beforehand.
2013-04-30 13:22:55 -04:00
Joey Hess
c3498042fd webapp: Now automatically fills in any creds used by an existing remote when creating a new remote of the same type. Done for Internet Archive, S3, Glacier, and Box.com remotes. 2013-04-27 15:16:06 -04:00
Joey Hess
85d83e7756 To enable an existing special remote, the new enableremote command must be used. The initremote command now is used only to create new special remotes. 2013-04-26 18:22:52 -04:00
Joey Hess
8a7e059f54 fix build w/o s3 2013-04-26 16:12:23 -04:00
Joey Hess
440d6e331f tweak
This works around a bug in the EvilSplicer
2013-04-26 15:20:31 -04:00
Joey Hess
b4f4ed7d93 fix build with newer yesod 2013-04-26 15:06:18 -04:00
Joey Hess
8603109294 allow configuring the preferreddir 2013-04-26 13:01:06 -04:00
Joey Hess
867cba52a0 better default collections 2013-04-26 01:46:07 -04:00
Joey Hess
0ae8c82c53 per-IA-item content directories 2013-04-25 23:44:55 -04:00
Joey Hess
e3ea36174b webapp: Display some additional information about a repository on its edit page. 2013-04-25 16:42:17 -04:00
Joey Hess
8284b310a7 support enabling IA repositories 2013-04-25 13:14:49 -04:00
Joey Hess
2810807ca5 Internet Archive!
* Add public repository group.
* webapp: Can now set up Internet Archive repositories.

TODO: Enabling IA repositories.
2013-04-25 12:23:36 -04:00
Joey Hess
362ed9f0e3 use DList for the transfer queue
Some nice efficiency gains here for list appending, although mostly
the small size of the transfer queue makes them irrelivant.
2013-04-25 01:33:44 -04:00
Joey Hess
c6da464051 use a DList for the deferred downloads queue 2013-04-25 01:26:23 -04:00
Joey Hess
82a6db8fe8 committer tweak to wait for Watcher to resume after a max-size commit
Without this, a very large batch add has commits of sizes approx
5000, 2500, 1250, etc down to 10, and then starts over at 5000.
This fixes it so it's 5000+ every time.
2013-04-25 00:48:09 -04:00
Joey Hess
a9081ae473 optimise direct mode startup scan
A recent change made existing symlinks be re-staged. That does not need to
be done during the startup scan though.
2013-04-24 21:20:29 -04:00
Joey Hess
46529c0129 assistant: Sanitize XMPP presence information logged for debugging. 2013-04-24 21:13:10 -04:00
Joey Hess
ebee93a837 get rid of need to run pre-commit hook when assistant commits in direct mode
That hook updates associated file bookkeeping info for direct mode.
But, everything already called addAssociatedFile when adding/changing a
file. It only needed to also call removeAssociatedFile when deleting a file,
or a directory.

This should make bulk adds faster, by some possibly significant amount.
Bulk removals may be a little slower, since it has to use catKeyFile now
on each removed file, but will still be faster than adds.
2013-04-24 18:04:59 -04:00
Joey Hess
cb575cbe91 fix bda237f14a
Converted back to the wrong type, oops.
2013-04-24 17:50:11 -04:00
Joey Hess
b8e45ec9d7 refactoring and minor performance tweak 2013-04-24 17:46:46 -04:00
Joey Hess
7fa2d255da remove last use of TSet 2013-04-24 17:16:04 -04:00
Joey Hess
cd7055631f batch commit every 5 thousand changes, not 10 thousand
There's a tradeoff between making less frequent commits, and
needing to use memory to store all the changes that are coming
in. At 10 thousand, it needs 150 mb of memory. 5 thousand drops
that down to 90 mb or so.

This also turns out to have significant imact on total run time.
I benchmarked 10k changes taking 27 minutes. But two 5k batches
took only 21 minutes.
2013-04-24 16:40:35 -04:00
Joey Hess
bda237f14a convert PendingAddChange back to Change when an add fails
If an add failed, we should lose the KeySource, since it, presumably,
differs due to a change that was made to the file.

(The locked down file is already deleted.)
2013-04-24 16:29:25 -04:00
Joey Hess
76431520e4 use TList for ChangePool 2013-04-24 16:13:22 -04:00
Joey Hess
a929e6641a show one alert when bulk adding files
Turns out that a lot of the time spent in a bulk add was just updating the
add alert to rotate through each file that was added. Showing one alert
makes for a significant speedup.

Also, when the webapp is open, this makes it take quite a lot less cpu
during bulk adds.

Also, it lets the user know when a bulk add happened, which is sorta
nice..
2013-04-24 13:04:46 -04:00
Joey Hess
01bedb54c6 improve display of multiline messages in alerts
This better handles error messages formatted for console display, by
adding a <br> after each line.

Hmm, I wonder if it'd be worth pulling in a markdown formatter, and running
the messages through it?
2013-04-24 11:45:41 -04:00
Joey Hess
e72f0b2898 only show latest warning, rather than combining warning alerts
In the case of the inotify limit warning, particularly, if it happens once
it will be happening repeatedly, and so combining alerts resulted in a
much too large alert message that took up a lot of memory and was too
large for the webapp to display.
2013-04-24 11:27:30 -04:00
Joey Hess
ca72b1ac7b assistant: when an add fails, requeue it for later
See analysis in bug report for one way this could happen.
2013-04-23 18:23:04 -04:00
Joey Hess
daecfd5bd7 more efficient data type for ChangeChan
Making this a tset of lists of Changes, rather than a tset of Changes
makes refilling it, in batch mode, much more efficient. Rather than needing
to add every Change it's collected one at a time, it can add them in one
fast batch operation.

It would be more efficient yet to use a Set, but that would need an Eq
instance for InodeCache.
2013-04-23 14:54:54 -04:00
Joey Hess
2c42e70f6c rename module 2013-04-23 11:38:52 -04:00
Joey Hess
8a2d1988d3 expose Control.Monad.join
I think I've been looking for that function for some time.
Ie, I remember wanting to collapse Just Nothing to Nothing.
2013-04-22 20:24:53 -04:00
Joey Hess
450baa02e9 assistant: When built with git before 1.8.0, use git remote rm to delete a remote. Newer git uses git remote remove. 2013-04-22 16:53:09 -04:00
Joey Hess
e92a207df5 webapp: When told to add a git repository on a remote server, and the repository already exists as a non-bare repository, use it, rather than initializing a bare repository in the same directory. 2013-04-22 16:21:37 -04:00
Joey Hess
9cb223a8b3 Detect systems that have no user name set in GECOS, and also don't have user.name set in git config, and put in a workaround so that commits to the git-annex branch (and the assistant) will still succeed despite git not liking the system configuration. 2013-04-22 15:36:34 -04:00
Joey Hess
8861e270be sync, assistant: Sync with remotes that have annex-ignore set
This is so git remotes on servers without git-annex installed can be used
to keep clients' git repos in sync.

This is a behavior change, but since annex-sync can be set to disable
syncing with a remote, I think it's acceptable.
2013-04-22 14:57:09 -04:00
Joey Hess
1579c98d4d fix osx build 2013-04-22 11:24:48 -04:00
Joey Hess
8df5be7804 fix build with xmpp enabled but not local pairing 2013-04-21 13:24:56 -04:00
Joey Hess
580f22f45d forgot to commit this new file 2013-04-20 19:27:36 -04:00
Joey Hess
844e772910 fix warnings 2013-04-20 19:25:50 -04:00
Joey Hess
70e251547e fix menu file installation directory 2013-04-20 19:02:14 -04:00
Joey Hess
4ddf9b7467 webapp: Don't default to making repository in cwd when started from within a directory containing a git-annex file (eg, standalone tarball directory). 2013-04-20 18:53:04 -04:00
Joey Hess
780ed8ebf1 fix warning 2013-04-20 18:52:46 -04:00
Joey Hess
d2e92e2494 Install FDO desktop menu file when webapp is started in standalone mode. 2013-04-20 18:48:05 -04:00
Joey Hess
090a69f00c assistant: Work around misfeature in git 1.8.2 that makes git commit --alow-empty -m "" run an editor.
See http://git-annex.branchable.com/bugs/assistant_hangs_during_commit/
2013-04-18 16:27:17 -04:00
Joey Hess
824c95376a refactor static route definition -- android webapp build fix
Incidentially should work around the last problem that prevented the webapp
building on Android. (Except for a few places I need to clean up after
hand-fixing the spliced TH code.)
2013-04-17 01:37:08 -04:00
Joey Hess
eeaf471f2f minor tweak to fix a build issue when TH is splices 2013-04-16 22:46:20 -04:00
Joey Hess
c1b682cb8f another no-xmpp fix 2013-04-16 19:30:09 -04:00
Joey Hess
d0336a9475 build fix for no xmpp 2013-04-16 19:19:30 -04:00
Joey Hess
7956177445 webapp: Include the repository directory in the mangled hostname and ssh key name, so that a locked down ssh key for one repository is not re-used when setting up additional repositories on the same server. 2013-04-16 13:49:39 -04:00
Joey Hess
21f93de33c use git-annex for ssh directory name 2013-04-14 20:00:32 -04:00
Joey Hess
aca9e4f0b4 random ssh keys (╯°□°)╯︵ ┻━┻ gnome-keyring
assistant: Work around horrible, terrible, very bad behavior of
gnome-keyring, by not storing special-purpose ssh keys in ~/.ssh/*.pub.

Apparently gnome-keyring apparently will load and indiscriminately use such
keys in some cases, even if they are not using any of the standard ssh key
names. Instead store the keys in ~/.ssh/annex/, which gnome-keyring will
not check.

Note that neither I nor #debian-devel were able to quite reproduce this
problem, but I believe it exists, and that this fixes it. And it certianly
won't hurt anything..
2013-04-14 15:34:59 -04:00
Joey Hess
11d106a18c turn on PackageImports globally
This will make it easier to use the Evil Splicer, when it needs to add
package qualified imports

And there's no real downside.
2013-04-13 18:12:44 -04:00
Joey Hess
dbb791a13b avoid warning 2013-04-13 15:46:24 -04:00
Joey Hess
2c365b8b74 remove debug 2013-04-11 16:36:45 -04:00
Joey Hess
1eb3fff787 addurl: Register transfer so the webapp can see it.
* addurl: Register transfer so the webapp can see it.
* addurl: Automatically retry downloads that fail, as long as some
  additional content was downloaded.
2013-04-11 16:14:17 -04:00
Joey Hess
04a27ad926 assistant: Bug fix to avoid annexing the files that git uses to stand in for symlinks on FAT and other filesystem not supporting symlinks.
also, blog for the day..
2013-04-10 19:57:26 -04:00