Commit graph

962 commits

Author SHA1 Message Date
Joey Hess
5fe49b98f8 Support hot-swapping of removable drives containing gcrypt repositories.
To support this, a core.gcrypt-id is stored by git-annex inside the git
config of a local gcrypt repository, when setting it up.

That is compared with the remote's cached gcrypt-id. When different, a
drive has been changed. git-annex then looks up the remote config for
the uuid mapped from the core.gcrypt-id, and tweaks the configuration
appropriately. When there is no known config for the uuid, it will refuse to
use the remote.
2013-09-12 15:54:35 -04:00
Joey Hess
b64f5baf2d sync: support gcrypt 2013-09-09 10:02:15 -04:00
Joey Hess
7c1a9cdeb9 partially complete gcrypt remote (local send done; rest not)
This is a git-remote-gcrypt encrypted special remote. Only sending files
in to the remote works, and only for local repositories.

Most of the work so far has involved making initremote work. A particular
problem is that remote setup in this case needs to generate its own uuid,
derivied from the gcrypt-id. That required some larger changes in the code
to support.

For ssh remotes, this will probably just reuse Remote.Rsync's code, so
should be easy enough. And for downloading from a web remote, I will need
to factor out the part of Remote.Git that does that.

One particular thing that will need work is supporting hot-swapping a local
gcrypt remote. I think it needs to store the gcrypt-id in the git config of the
local remote, so that it can check it every time, and compare with the
cached annex-uuid for the remote. If there is a mismatch, it can change
both the cached annex-uuid and the gcrypt-id. That should work, and I laid
some groundwork for it by already reading the remote's config when it's
local. (Also needed for other reasons.)

This commit was sponsored by Daniel Callahan.
2013-09-07 18:38:00 -04:00
Joey Hess
9dc2373977 only retry every 60 seconds
Retying every second is a bit much, especially given the current leak
https://github.com/audreyt/network-multicast/issues/4
2013-08-24 14:37:34 -04:00
Joey Hess
8587485994 clarify notifyNetMessagerRestart 2013-08-24 13:49:04 -04:00
Joey Hess
b191d5c595 gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.

Thanks to Adam Spiers, for getting the necessary support into git for this.

A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.

However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.

Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!

So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.

(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)

This commit was sponsored by Francois Marier. Thanks!
2013-08-02 20:37:03 -04:00
Joey Hess
672cfc3923 better git version checking 2013-08-02 18:32:26 -04:00
Joey Hess
eba74e3940 fix build with old yesod 2013-08-02 12:32:16 -04:00
Joey Hess
d1ed337035 webapp: Improve handling of remotes whose setup has stalled.
This includes recovery from the ssh-agent problem that led to many reporting
http://git-annex.branchable.com/bugs/Internal_Server_Error:_Unknown_UUID/
(Including fixing up .ssh/config to set IdentitiesOnly.)

Remotes that have no known uuid are now displayed in the webapp as
"unfinished". There's a link to check their status, and if the remote
has been set annex-ignore, a retry button can be used to unset that and
try again to set up the remote.

As this bug has shown, the process of adding a ssh remote has some failure
modes that are not really ideal. It would certianly be better if, when
setting up a ssh remote it would detect if it's failed to get the UUID,
and handle that in the remote setup process, rather than waiting until
later and handling it this way.

However, that's hard to do, particularly for local pairing, since the
PairListener runs as a background thread. The best it could do is pop up an
alert if there's a problem. This solution is not much different.

Also, this solution handles cases where the user has gotten their repo into
a mess manually and let's the assistant help with cleaning it up.

This commit was sponsored by Chia Shee Liang. Thanks!
2013-07-31 16:36:29 -04:00
Joey Hess
ab62ae357a hide NoUUID remotes from repolist
A ssh remote will breifly have NoUUID when it's just being set up and
git-annex-shell has not yet been queried for the UUID. So it doesn't make
sense to display any kind of error message in this case. The UI doesn't
work when there's NoUUID, and it can even crash the ajax long polling code.
So hiding NoUUID repositories is the right thing to do.

I've tested and the automatic refresh of the repolist causes the remote
to show up as soon as a UUID is recorded, when doing local pairing, and
when adding a ssh remote.
2013-07-31 14:31:05 -04:00
Joey Hess
cbfdf3ab21 set IdentitiesOnly
When setting up a dedicated ssh key to access the annex on a host,
set IdentitiesOnly to prevent the ssh-agent from forcing use of a different
ssh key.

That behavior could result in unncessary password prompts. I remember
getting a message or two from people who got deluged with password
prompts and I couldn't at the time see why.

Also, it would prevent git-annex-shell from being run on the remote host,
when git-annex was installed there by unpacking the standalone tarball,
since the authorized_keys line for the dedicated ssh key, which sets
up calling git-annex-shell when it's not in path, wouldn't be used.

This fixes
http://git-annex.branchable.com/bugs/Internal_Server_Error:_Unknown_UUID
but I've not closed that bug yet since I should still:

1. Investigate why the ssh remote got set up despite being so broken.
2. Make the webapp not handle the NoUUID state in such an ugly way.
3. Possibly add code to fix up systems that encountered the problem.
   Although since it requires changes to .ssh/config this may be one for
   the release notes.

Thanks to TJ for pointing me in the right direction to understand what
was happening here.
2013-07-31 13:30:49 -04:00
Joey Hess
869c638b82 assistant: Fix bug that caused it to stall when adding a very large number of files at once (around 5 thousand).
This bug was introduced in 82a6db8fe8,
which improved handling of adding very large numbers of files by ensuring
that a minimum number of max size commits (5000 files each) were done.

I accidentially made it wait for another change to appear after such a max
size commit, even if a lot of queued changes were already accumulated.
That resulted in a stall when it got to the end. Now fixed to not wait
any longer than necessary to ensure the watcher has had time to wake back
up after the max size commit.

This commit was sponsored by Michael Linksvayer. Thanks!
2013-07-27 17:42:18 -04:00
Joey Hess
2d7b4517ab got hdevtools working again
Fix comment to not make cpp unhappy
2013-07-27 13:00:49 -04:00
Joey Hess
ec4d974dcf assistant: Fix deadlock that could occur when adding a lot of files at once in indirect mode.
This is a laziness problem. Despite the bang pattern on newfiles, the list
was not being fully evaluated before cleanup was called. Moving cleanup out
to after the list is actually used fixes this.

More evidence that I should be using ResourceT or pipes, if any was needed.
2013-07-26 18:42:22 -04:00
Joey Hess
97f3aecb17 assistant: Fix NetWatcher to not sync with remotes that have remote.<name>.annex-sync set to false.
This affected both the hourly NetWatcherFallback thread and the syncing
when network connection is detected.

It was a reversion of sorts, introduced in
8861e270be, when annex-ignore was changed to
not control git syncing. I forgot to make it check annex-sync at that
point.
2013-07-26 16:54:20 -04:00
Joey Hess
6967f539d0 add status tag to all presence messages
Necessary for push messages to not override the previously sent tag.
2013-07-25 14:58:30 -04:00
Joey Hess
54e0bd55b4 fun with exec 2013-07-23 17:31:51 -04:00
Joey Hess
0962c43bf7 fix build 2013-07-23 13:40:27 -04:00
Joey Hess
5c82c99c76 webapp: When creating a repository on a removable drive, set core.fsyncobjectfiles, to help prevent data loss when the drive is yanked. 2013-07-23 13:38:05 -04:00
Joey Hess
381637e4c8 Add status message to XMPP presence tag, to identify to others that the client is a git-annex client.
I only added this to the presense messages that are really intended for
presence. The ones used for tunneling git etc don't have the tag, because
that would waste bandwidth.
2013-07-23 12:41:41 -04:00
Joey Hess
780efc775c When an XMPP server has SRV records, try them, but don't then fall back to the regular host if they all fail.
gmail.com has some XMPP SRV records, but does not itself respond to XMPP
traffic, although it does accept connections on port 5222. So if a user
entered the wrong password, it would try all the SRVs and fall back to
trying gmail, and hang at that point.

This seems the right thing to do, not just a workaround.
2013-07-20 21:18:55 -04:00
Joey Hess
9fc1448947 webapp: Differentiate between creating a new S3/Glacier/WebDav remote, and initializing an existing remote. When creating a new remote, avoid conflicts with other existing (or deleted) remotes with the same name. 2013-07-20 18:15:16 -04:00
Joey Hess
dba1e29949 webapp: Better display of added files. 2013-07-10 15:37:40 -04:00
Joey Hess
00e6663128 linux standalone auto-install icons 2013-07-09 20:50:41 -04:00
Joey Hess
19b8bcbe30 Install XDG desktop icon files.
The icon files will be installed when running make install or cabal
install. Did not try to run update-icon-caches, since I think it's debian
specific, and dh_icons will take care of that for the Debian package.

Using the favicon as a 16x16 icon. At 24x24 the svg displays pretty well,
although the dotted lines are rather faint. The svg is ok at all higher
resolutions.

The standalone linux build auto-installs the desktop and autostart files
when run. I have not made it auto-install the icon file too, because
a) that would take more work to include them in the tarball and find them
b) it would need to be an install to ~/.icons/, and I don't know if that
   really works!
2013-07-09 19:56:30 -04:00
Joey Hess
b6d857da08 fix build 2013-07-08 12:53:08 -04:00
Joey Hess
57f5c9f41a webapp: Fix authorized_keys line added when setting up a rsync remote on a server that also supports git-annex, to not force running git-annex-shell. 2013-07-08 12:51:07 -04:00
Joey Hess
96ded21ae5 fix display of expected sequence number 2013-07-08 12:31:32 -04:00
Joey Hess
d84a000e92 detect system with no dot in FQDN, where git commit will fail, and workaround
Sigh, git is so *fragile*. Or rather, across the set of systems that use
git-annex, where are no many horribly broken systems..
2013-07-05 12:24:28 -04:00
Joey Hess
7a7e426352 moved AssociatedFile definition 2013-07-04 02:36:02 -04:00
Joey Hess
1b395dcb76 webapp: Fix ssh setup with nonstandard port, broken in last release. 2013-07-02 15:46:02 -04:00
Joey Hess
ff4f008591 clean up build warnings with yesod 1.2, while still building with 1.1 2013-06-27 01:15:28 -04:00
Joey Hess
f1fe8edbbb webapp: Fix bug setting up ssh repo if the user enters "~/" at the start of the path. 2013-06-25 15:43:46 -04:00
Joey Hess
c8e6947693 webapp: Ensure that ssh keys generated for different directories on a server are always different. 2013-06-25 15:31:04 -04:00
Joey Hess
10072157cb webapp: Fix bug when setting up a remote ssh repo repeatedly on the same server.
The ssh setup first runs ssh to the real hostname, to probe if a ssh key is
needed. If one is, it generates a mangled hostname that uses a key. This
mangled hostname was being used to ssh into the server to set up the key.
But if the server already had the key set up, and it was locked down, the
setup would fail. This changes it to use the real hostname when sshing in
to set up the key, which avoids the problem.

Note that it will redundantly set up the key on the ssh server. But it's
the same key; the ssh key generation code uses the key if it already
exists.
2013-06-25 14:46:20 -04:00
Joey Hess
ce9be5ec14 fix crash in drop scan when there are no associated files
maximum is partial, so need to ensure the list is not empty.

This lack of associated files would most likely be a problem -- and fsck
fixes it. It could also occur legitimately due to a race.
2013-06-24 12:44:51 -04:00
Joey Hess
5e48aa4d4b assistant: Daily sanity check thread is run niced. 2013-06-21 13:29:42 -04:00
Joey Hess
9b61d6a82b fix warning building w/o xmpp 2013-06-20 22:25:45 -04:00
Joey Hess
6e309b63f8 assistant: On Linux, the expensive transfer scan is run niced.
This is a compromise. I would like to nice every thread except for the
webapp thread, but it's not practical to do so. That would need every
thread to run as a bound thread, which could add significant overhead.
And any forkIO would escape the nice level.
2013-06-20 22:25:41 -04:00
Joey Hess
e8827c29c6 Make field readonly, not disabled
Working on Windows horriblness all day sure did nice things to my mood,
didn't it?
2013-06-18 17:08:37 -04:00
Joey Hess
f651b87bbd revert broken change that prevented saving changes to local repo
People who get annoyed at not being able to edit "here" can send me a
patch, I don't have time for this nonsense.
2013-06-18 17:02:58 -04:00
Joey Hess
fa1c1e0f65 annex.debug can now be set to enable debug logging by default. The webapp's debugging check box does this. 2013-06-17 20:41:27 -04:00
Joey Hess
0527c74c0f assistant: In direct mode, objects are now only dropped when all associated files are unwanted. This avoids a repreated drop/get loop of a file that has a copy in an archive directory, and a copy not in an archive directory. (Indirect mode still has some buggy behavior in this area, since it does not keep track of associated files.) Closes: #712060 2013-06-15 14:44:43 -04:00
Joey Hess
c9f3082b99 disable text field for repository name for current repository 2013-06-13 13:38:46 -04:00
Joey Hess
81d6f96cd9 sanity check 2013-06-11 23:12:01 -04:00
Joey Hess
c7b493fecb log local pairing messages received when debugging is enabled 2013-06-11 00:25:44 -04:00
Joey Hess
680d3b5120 webapp: When the user switches to display a different repository, that repository becomes the default repository to be displayed next time the webapp gets started. 2013-06-11 00:06:06 -04:00
Joey Hess
768f7c87f2 display any illegal character found in ssh comment 2013-06-10 23:55:54 -04:00
Joey Hess
c46b263fde Android: Make the "Open webapp" menu item open the just created repository when a new repo is made. 2013-06-10 23:55:53 -04:00
Joey Hess
0b11c71740 Android: Add .thumbnails to .gitignore when setting up a camera repository. 2013-06-10 17:10:12 -04:00