Commit graph

1159 commits

Author SHA1 Message Date
Joey Hess
5279c4d1df tested transferkeys restarting; fix some bugs 2014-01-06 17:07:08 -04:00
Joey Hess
d356069f3e use strict version of map 2014-01-06 16:04:09 -04:00
Joey Hess
e5b4d447b6 assistant: Start a new git-annex transferkeys process after a network connection change
So that remotes that use a persistent network connection are restarted.

A remote might keep open a long duration network connection, and could
fail to deal well with losing the connection. This is particularly a
concern now that we have external special reotes. An external
special remote that is implemented naively might open the connection only
when PREPARE is sent, and if it loses connection, throw errors on each
request that is made.

(Note that the ssh connection caching should not have this problem; if the
long-duration ssh process loses connection, the named pipe is disconnected
and the next ssh attempt will reconnect. Also, XMPP already deals with
disconnection robustly in its own way.)

There's no way for git-annex to know if a lost network connection actually
affects a given remote, which might have a transfer in process. It does not
make sense to force kill the transferkeys process every time the NetWatcher
detects a change. (Especially because the NetWatcher sometimes polls 1
change per hour.)

In any case, the NetWatcher only detects connection to a network, not
disconnection. So if a transfer is in progress over the network, and the
network goes down, that will need to time out on its own.

An alternate approch that was considered is to use a separate transferkeys
process for each remote, and detect when a request fails, and assume that
means that process is in a failing state and restart it. The problem with
that approach is that if a resource is not available and a remote fails
every time, it degrades to starting a new transferkeys process for every
file transfer, which is too expensive.

Instead, this commit only handles the network reconnection case, and restarts
transferkeys only once the network has reconnected and another transfer needs
to be made. So, a transferkeys process will be reused for 1 hour, or until the
next network connection.

----

The NotificationBroadcaster was rewritten to use TMVars rather than MSampleVars,
to allow checking without blocking if a notification has been received.

----

This commit was sponsored by Tobias Brunner.
2014-01-06 16:03:39 -04:00
Joey Hess
b92b54bd42 assistant: Fixed several minor memory leaks that manifested when adding a large number of files. 2014-01-05 21:30:48 -04:00
Joey Hess
fdcc04096c assistant: Ensure that .ssh/config and .ssh/authorized_keys are not group or world writable when writing to those files, as that can make ssh refuse to use them, if it allows another user to write to them. 2014-01-03 17:44:12 -04:00
Joey Hess
f7727d2df1 Remotes can now be made read-only, by setting remote.<name>.annex-readonly 2014-01-02 13:12:32 -04:00
Joey Hess
f83e5c75dc Assistant: Remove stale MERGE_HEAD files in lockfile cleanup.
See http://git-annex.branchable.com/forum/How_to_resolve_a_interrupted_merge_in_direct_mode__63__/
2014-01-01 22:05:33 -04:00
Joey Hess
079f463d51 mirror: Support --all (and --unused). 2014-01-01 17:39:33 -04:00
Joey Hess
fab7adb161 Windows: Avoid eating stdin when running ssh to add a authorized key, since this is used for password prompting. 2014-01-01 15:49:51 -04:00
Joey Hess
5c834daf59 avoid empty env vars when setting up clean environment 2013-12-31 14:55:55 -04:00
Joey Hess
6c565ec905 external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!

The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)

Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo

Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:

* Need to lock the remote when using it to eg make a request, because
  in theory git-annex could have multiple threads that each try to use
  a remote at the same time. I don't think that git-annex ever does
  that currently, but better safe than sorry.

* Rather than starting up every external special remote program when
  git-annex starts, they are started only on demand, when first used.
  This will avoid slowdown, especially when running fast git-annex query
  commands. Once started, they keep running until git-annex stops, currently,
  which may not be ideal, but it's hard to know a better time to stop them.

* Bit of a chicken and egg problem with caching the cost of the remote,
  because setting annex-cost in the git config needs the remote to already
  be set up. Managed to finesse that.

This commit was sponsored by Lukas Anzinger.
2013-12-26 18:23:13 -04:00
Joey Hess
8e58ff7f71 check that ~/Desktop/annex can be written to; fall back to ~/annex when not 2013-12-24 17:04:07 -04:00
Joey Hess
3a6e0d1215 assistant: Set StrictHostKeyChecking yes when creating ssh remotes, and add it to the configuration for any ssh remotes previously created by the assistant. This avoids repeated prompts by ssh if the host key changes, instead syncing with such a remote will fail. Closes: #732602 2013-12-20 20:58:36 -04:00
Richard Hartmann
974fe009bf Another round of s/amoung/among/ 2013-12-19 12:30:53 -04:00
Joey Hess
adca65b31a remove some unused vars
Warnings for these are normally hidden somehow by TH, but show up when
building with EvilSplicer.
2013-12-18 22:39:15 -04:00
Joey Hess
c99d6a8151 assistant: Fix OSX-specific bug that caused the startup scan to try to follow symlinks to other directories, and add their contents to the annex. 2013-12-18 15:05:29 -04:00
Joey Hess
b7e3fe2ebd flip for clarity 2013-12-16 16:24:57 -04:00
Joey Hess
58c7b0a56d assistant: Always batch changes found in startup scan.
Batch detection is heuristic, so can sometimes fail. I observed one such
failure while starting up in a repository with 87000 files. After the first
several batches of ~5000 files, it fell out of batch mode, and never
re-entered it, and so made many more commits of a few files at a time
than necessary.

So, let's always use batch mode when in the startup scan. This avoids the
heuristic there, at least.

There is clearly also room to improve the heuristic. Possibly 10 files is
too high a bar to be found during a commit, on a system that can commit
quickly.
2013-12-16 16:16:19 -04:00
Joey Hess
0bc4a78db5 squash warning in OSX build 2013-12-15 17:10:24 -04:00
Joey Hess
13f7e51724 add a note about only writable drives being listed 2013-12-12 14:25:50 -04:00
Joey Hess
011b8bc7ec pull in Win32-extras, to be able to get current process id in Windows
Fixed up a number of things that had worked around there not being a way to
get that.

Most notably, transfer info files on windows now include the process id,
since no locking is currently done. This means the file format varies
between windows and unix.
2013-12-11 00:15:10 -04:00
Joey Hess
c01af6285f typo 2013-12-10 23:55:15 -04:00
Joey Hess
ecd42aef8e different PID types for Unix and Windows
Windows has a larger (unsigned) PID space, so cannot use the unix CInt
there.

Note that TransferInfo does not yet ever get the TransferPid populated,
as there is missing locking.
2013-12-10 23:48:42 -04:00
Joey Hess
2fd63f3cfa port transferkeys to windows; make stopping in progress transfers work too (probably)
transferkeys had used special FDs for communication, but that would be
quite annoying to do in Windows.

Instead, use stdin and stdout. But, to avoid commands like rsync stomping
on them and messing up the communications channel, they're duplicated to a
different handle; stdin is replaced with a null handle, and stdout is
replaced with a copy of stderr. This should all work in windows too.

Stopping in progress transfers may work on windows.. if the types unify
anyway. ;) May need some more porting.
2013-12-10 23:19:18 -04:00
Joey Hess
e6c4f550d8 repair: Remove damaged git-annex sync branches. 2013-12-10 16:17:49 -04:00
Joey Hess
ce045a51af Improve repair of git-annex index file.
Fixes a test case I received where a corrupted repo was repaired, but the
git-annex branch was not. The root of the problem was that the
MissingObject returned by the repair code was not necessarily a complete
set of all objects that might have been deleted during the repair.

So, stop trying to return that at all, and instead make the index file
checking code explicitly verify that each object the index uses is present.
2013-12-10 15:40:01 -04:00
Joey Hess
7c5261efb1 import fixes 2013-12-10 01:26:52 -04:00
Joey Hess
063a183a61 windows disk free space checking 2013-12-10 01:18:04 -04:00
Joey Hess
0bb9ace195 fix drive path so fileAccess will work with it 2013-12-09 17:58:49 -04:00
Joey Hess
98bd6e1f89 Revert "add /dashboard route as an alias for /"
This reverts commit a1880a152c.

Didn't help firefox problem.
2013-12-09 17:26:25 -04:00
Joey Hess
6f8693c59a redirect to /dashboard 2013-12-09 17:24:53 -04:00
Joey Hess
a1880a152c add /dashboard route as an alias for / 2013-12-09 17:14:45 -04:00
Joey Hess
d961985679 fix 2013-12-09 17:13:37 -04:00
Joey Hess
86a4b56db3 fix 2013-12-09 17:11:40 -04:00
Joey Hess
a268ac6c1d comment 2013-12-09 16:59:16 -04:00
Joey Hess
4866ec84df try all drive letters 2013-12-09 16:56:52 -04:00
Joey Hess
2066e90421 avoid needing --force on windows despite no lsof
Note that I still need to think this through and make sure handling of open
files is safe. This is just for testing purposes.
2013-12-09 16:56:15 -04:00
Joey Hess
51b43f2b68 better default annex location on windows 2013-12-09 16:39:23 -04:00
Joey Hess
fc3a9a8adb make opening file browser work on windows 2013-12-09 16:39:10 -04:00
Joey Hess
8fb1973c76 move all static site files out of subdirectories
Works around yesod-static path bug on windows.
https://github.com/yesodweb/yesod/issues/626
2013-12-09 14:38:22 -04:00
Joey Hess
a38abecd66 close tmp file handle
May fix permission problem on windows
2013-12-07 11:45:01 -04:00
Joey Hess
621ca07119 squash warning 2013-12-05 14:16:07 -04:00
Joey Hess
7a8e47d178 allow building on windows 2013-12-05 14:10:49 -04:00
Joey Hess
9d323a98e2 avoid trying to use lsof when it's not in path and --forced 2013-12-04 17:39:44 -04:00
Joey Hess
aef1d7e76d stuf on Windows 2013-12-04 15:37:53 -04:00
Joey Hess
8c55003ae4 don't shown non-working file browser opening UI on android 2013-12-03 13:41:23 -04:00
Joey Hess
e425a966ed Deal with box.com changing the url of their webdav endpoint.
Use new url when making new remotes.

Transparently rewrite old url to new for existing remotes.
2013-12-02 16:01:20 -04:00
Joey Hess
0fd6078865 avoid repeatedly searching path to make batch command when running transferkeys 2013-12-01 15:37:51 -04:00
Joey Hess
2711d13d55 assistant: Run transferkeys as batch jobs. 2013-12-01 14:56:37 -04:00
Joey Hess
03932212ec Avoid using git commit in direct mode, since in some situations it will read the full contents of files in the tree.
The assistant's commit code also always avoids git commit, for simplicity.
Indirect mode sync still does a git commit -a to catch unstaged changes.

Note that this means that direct mode sync no longer runs the pre-commit
hook or any other hooks git commit might call. The git annex pre-commit
hook action for direct mode is however explicitly run. (The assistant
already ran git commit with hooks disabled, so no change there.)
2013-12-01 13:59:45 -04:00