Commit graph

134 commits

Author SHA1 Message Date
Joey Hess
be2e9427ad
refactor 2016-02-25 13:46:31 -04:00
Joey Hess
737e45156e
remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Joey Hess
aaf1ef268d
convert from Utility.LockPool to Annex.LockPool everywhere 2015-11-12 18:13:37 -04:00
Joey Hess
7c741302cc
assistant: Pass ssh-options through 3 more git pull/push calls that were missed before.
It was used for regular pull, but not for regular push, tagged push, or the
fallback fetching.
2015-11-10 16:52:30 -04:00
Joey Hess
f7d7995172 clean 2015-08-04 17:07:45 -04:00
Joey Hess
3c971c414e sshopts is never going to be null; the concat of it may be 2015-08-04 16:53:38 -04:00
Joey Hess
a6374b7a3d typo 2015-08-04 15:44:46 -04:00
Joey Hess
f041a65c33 Windows: Fix bug that caused git-annex sync to fail due to missing environment variable.
I think that the problem was caused by windows not having a concept of an
env var that is set, but to the empty string. So, GIT_ANNEX_SSHOPTION
got set to "" and was not seen as set at all.

Easy fix, which also makes git-annex sync a little faster is to not set
GIT_SSH, when GIT_ANNEX_SSHOPTION has no options. Might as well let git use
ssh per usual in this case, no need to run git-annex as the proxy ssh
command..
2015-08-04 15:27:48 -04:00
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
a6d54e49a0 sync, remotedaemon: Pass configured ssh-options even when annex.sshcaching is disabled. 2015-05-30 22:01:52 -04:00
Joey Hess
ecb0d5c087 use lock pools throughout git-annex
The one exception is in Utility.Daemon. As long as a process only
daemonizes once, which seems reasonable, and as long as it avoids calling
checkDaemon once it's already running as a daemon, the fcntl locking
gotchas won't be a problem there.

Annex.LockFile has it's own separate lock pool layer, which has been
renamed to LockCache. This is a persistent cache of locks that persist
until closed.

This is not quite done; lockContent stil needs to be converted.
2015-05-19 14:09:52 -04:00
Joey Hess
450ee53ab6 When re-execing git-annex, use current program location, rather than ~/.config/git-annex/program, when possible.
Most of the time, there will be no discreprancy between programPath and
readProgramFile.

But, the programFile might have been written by an old version of git-annex
that is still installed, while a newer one is currently running. In this
case, we want to run the same one that's currently running.

This is especially important for things like the GIT_SSH=git-annex used for
ssh connection caching.

The only code that still uses readProgramFile directly is the upgrade code,
which needs to know where the standalone git-annex was installed, in order to
upgrade it.
2015-02-28 17:23:13 -04:00
Joey Hess
15107d2c5a propigate ssh-options everywhere ssh caching is used
* sync: Use the ssh-options git config when doing git pull and push.
* remotedaemon: Use the ssh-options git config.

Note that the rename env var means that if a new git-annex calls an old one
for git-annex ssh, or a new calls an old, nothing much will go wrong;
just ssh caching won't happen.
2015-02-12 16:14:53 -04:00
Joey Hess
5be7ba7ee5 The ssh-options git config is now used by gcrypt, rsync, and ddar special remotes that use ssh as a transport. 2015-02-12 15:44:10 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
c9a3e80d32 fixed all remaining build warnings on Windows 2014-12-29 17:30:20 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
1994771215 more lock file refactoring
Also fixes a test suite failures introduced in recent commits, where
inAnnexSafe failed in indirect mode, since it tried to open the lock file
ReadWrite. This is why the new checkLocked opens it ReadOnly.

This commit was sponsored by Chad Horohoe.
2014-08-20 18:58:14 -04:00
Joey Hess
d279180266 reorganize and refactor lock code
Added a convenience Utility.LockFile that is not a windows/posix
portability shim, but still manages to cut down on the boilerplate around
locking.

This commit was sponsored by Johan Herland.
2014-08-20 16:45:58 -04:00
Joey Hess
83dc82c232 forgot some lifts 2014-08-20 11:51:47 -04:00
Joey Hess
092041fab0 Ensure that all lock fds are close-on-exec, fixing various problems with them being inherited by child processes such as git commands.
(With the exception of daemon pid locking.)

This fixes at part of #758630. I reproduced the assistant locking eg, a
removable drive's annex journal lock file and forking a long-running
git-cat-file process that inherited that lock.

This did not affect Windows.

Considered doing a portable Utility.LockFile layer, but git-annex uses
posix locks in several special ways that have no direct Windows equivilant,
and it seems like it would mostly be a complication.

This commit was sponsored by Protonet.
2014-08-20 11:37:02 -04:00
Joey Hess
26ee27915a refactor locking 2014-07-10 00:32:23 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
79cf404e75 support being run by ssh as ssh-askpass replacement
To use, set GIT_ANNEX_SSHASKPASS to point to a fifo or regular file
(FIFO is better, avoids touching disk or multiple readers) that contains
the password. Then set SSH_ASKPASS=git-annex, and when ssh runs it, it will
tell ssh the password.

This is not yet used..
2014-04-29 18:08:10 -04:00
Joey Hess
2807e14904
use a subdir of GIT_ANNEX_TMP for ssh connection caching sockets
To prevent any possible collisions with other, non-socket files, like the
xmppgit directory.
2014-04-20 16:56:01 -04:00
Joey Hess
a0ef99b3f9
don't try to use ssh connection caching for non-ssh urls 2014-04-13 21:39:04 -04:00
Joey Hess
a33b30d0c4 remotedaemon: When network connection is lost, close all cached ssh connections.
This commit was sponsored by Cedric Staub.
2014-04-12 16:32:59 -04:00
Joey Hess
15917ec1a8 sync, assistant, remotedaemon: Use ssh connection caching for git pushes and pulls.
For sync, saves 1 ssh connection per remote. For remotedaemon, the same
ssh connection that is already open to run git-annex-shell notifychanges
is reused to pull from the remote.

Only potential problem is that this also enables connection caching
when the assistant syncs with a ssh remote. Including the sync it does
when a network connection has just come up. In that case, cached ssh
connections are likely to be stale, and so using them would hang.
Until I'm sure such problems have been dealt with, this commit needs to
stay on the remotecontrol branch, and not be merged to master.

This commit was sponsored by Alexandre Dupas.
2014-04-12 15:59:34 -04:00
Joey Hess
ba02cd8a80 Fix ssh connection caching stop method to work with openssh 6.5p1, which broke the old method.
Old ssh did not check the hostname passed to -O stop, so I had used "any".
But now ssh does check it! I think this happened as part of the client-side
hostname canonicalization changes in 6.5p1, but have not verified that
introduced the problem.

The symptom was that it would try to dns lookup "any", which often caused a
bit of a delay at shutdown. And the old ssh connection kept running, so
it would do it over and over again.

Fixed by using localhost, which hopefully reliably resolves to some address
that ssh will accept.. Also nukeFile the socket after ssh has been asked to
shutdown, just in case.
2014-03-13 19:35:06 -04:00
Joey Hess
8e2997aa69 only run sshCleanup when the command actually used ssh connection caching
Optimises query commands that do not. More importantly, avoids any ssh
connection cleanup delay causing problems at the end of such commands.
2014-03-13 19:30:13 -04:00
Joey Hess
2d480602aa random hlint (to give the autobuilder something new to build) 2014-02-11 00:41:19 -04:00
Joey Hess
267c124f67 run ssh in the directory with its socket when stopping
This guarantees that stopping an existing socket never fails.

This might be the route out of the mess of needing to worry about socket
lengths in general. However, it would need quite a lot of refactoring
to make every place in git-annex that runs ssh run it with a cwd that was
determined by the location of its connection caching socket. If this
wasn't already such a mess, I'd consider even the thought of that API a bad
idea..
2013-10-06 21:11:39 -04:00
Joey Hess
6f38426cb8 work around ssh brain-damange
The control socket path passed to ssh needs to be 17 characters shorter
than the maximum unix domain socket length, because ssh appends stuff to it
to make a temporary filename. Closes: #725512

Also, take the shorter of the relative and the absolute paths to the
socket. Typically the relative path will be a lot shorter (unless
deep inside a subdirectory of the repository), and so using it will
avoid flirting with the maximum safe socket lenghts in more situations,
and so lead to less breakage if all my attempts at fixing this are
still buggy.
2013-10-06 20:59:36 -04:00
Joey Hess
b405295aee hlint
test suite still passes
2013-09-25 03:09:06 -04:00
Joey Hess
06db8e0bd9 squash compiler warnings on Windows 2013-08-04 13:18:05 -04:00
Joey Hess
93f2371e09 get rid of __WINDOWS__, use mingw32_HOST_OS
The latter is harder for me to remember, but avoids build failures in code
used by the configure program.
2013-08-02 12:27:32 -04:00
Joey Hess
6ae2637eb1 For long hostnames, use a hash of the hostname to generate the socket file for ssh connection caching.
This is ok to do now that the socket filename never needs to be mapped back
to a hostname.

Short hostnames will still appear in the clear, which is less obfuscated.
So this cannot possibly make ssh connection caching fail for a hostname it
used to work for.
2013-07-22 15:09:41 -04:00
Joey Hess
c6a020ad1f stop cached ssh connection w/o needing to look up host and port
Turns out that with -O stop -S socketfile, ssh does not need the real
hostname, or port to be specificed. This is because it simply talks to the
ssh behind the socket and tells it to stop. So, can eliminate the
conversion back from a socketfile to host and port. Which will allow using
shorter filenames for sockets in the future.
2013-07-21 14:14:54 -04:00
Joey Hess
1312cffad0 Revert "Windows: Ssh connection caching is now supported."
Yeah, that didn't actually work. Got error messages like it couldn't read
from the control socket, so probably ssh doesn't really support that on
Windows, at least the cygwin ssh build I'm using.
2013-06-17 22:13:28 -04:00
Joey Hess
07a17f58b7 Windows: Ssh connection caching is now supported.
Turns out the socket stuff just works on windows.
2013-06-17 22:05:49 -04:00
Joey Hess
17952a893e fix imports 2013-05-14 13:53:29 -04:00
Joey Hess
abe8d549df fix permission damage (thanks, Windows) 2013-05-11 23:54:25 -04:00
Joey Hess
18bdff3fae clean up from windows porting 2013-05-11 18:23:41 -04:00
Joey Hess
3c7e30a295 git-annex now builds on Windows (doesn't work) 2013-05-11 15:03:00 -05:00
Joey Hess
763cbda14f fixup #if 0 stubs to use #ifndef mingw32_HOST_OS
That's needed in files used to build the configure program.
For the other files, I'm keeping my __WINDOWS__ define, as I find that much easier to type.
I may search and replace it to use the mingw32_HOST_OS thing later.
2013-05-10 16:57:21 -05:00
Joey Hess
6c74a42cc6 stub out POSIX stuff 2013-05-10 16:29:59 -05:00
guilhem
a1eded8641 Allow rsync to use other remote shells.
Introduced a new per-remote option 'annex-rsync-transport' to specify
the remote shell that it to be used with rsync. In case the value is
'ssh', connections are cached unless 'sshcaching' is unset.
2013-04-13 19:26:24 -04:00
Joey Hess
8a5b397ac4 hlint 2013-04-03 03:52:41 -04:00
Joey Hess
974d075108 Run ssh with -T to avoid tty allocation and any login scripts that may do undesired things with it. 2013-03-04 23:36:07 -04:00
Joey Hess
051476c2a9 squelch warning 2013-02-22 18:22:12 -04:00
Joey Hess
00b465e213 shorter directory to external ssh socket
Before it was too long to be used.
2013-02-19 17:31:08 -04:00
Joey Hess
0f4cc559a7 Android: Support ssh connection caching. 2013-02-19 14:57:45 -04:00
Joey Hess
4008590c68 type based git config handling for remotes
Still a couple of places that use git config ad-hoc, but this is most of it
done.
2013-01-01 13:58:14 -04:00
Joey Hess
7f7c31df1c type based git config handling
Now there's a Config type, that's extracted from the git config at startup.
Note that laziness means that individual config values are only looked up
and parsed on demand, and so we get implicit memoization for all of them.
So this is not only prettier and more type safe, it optimises several
places that didn't have explicit memoization before. As well as getting rid
of the ugly explicit memoization code.

Not yet done for annex.<remote>.* configuration settings.
2012-12-29 23:10:18 -04:00
Joey Hess
f87a781aa6 finished where indentation changes 2012-12-13 00:24:19 -04:00
Joey Hess
b5a9560a1b squelch warning 2012-11-26 16:30:46 -04:00
Joey Hess
9767562f65 rsync special remote: Include annex-rsync-options when running rsync to test a key's presence.
Also, use the new withQuietOutput function to avoid running the shell to
/dev/null stderr in two other places.
2012-10-28 13:51:14 -04:00
Joey Hess
3156febec8 disable ssh connection caching for standalone builds
The standalone build does not bundle its own ssh, so should be built
to support as wide an array of ssh versions as possible, so turn off
connection caching.

Unfortunatly, as implemented this forces a full rebuild when building the
standalone binary, and of course it makes it somewhat slower.

This is not ideal, but neither is probing the ssh version every time it's
run (slow), or once when initializing a repo (fragile).
2012-10-15 14:49:40 -04:00
Joey Hess
e8188ea611 flip catchDefaultIO 2012-09-17 00:18:07 -04:00
Joey Hess
5573911d25 Disable ssh connection caching if the path to the control socket would be too long (and use relative path to minimise path to the control socket). 2012-09-13 19:26:39 -04:00
Joey Hess
bb4f31a0ee Clean up handling of git directory and git worktree.
Baked into the code was an assumption that a repository's git directory
could be determined by adding ".git" to its work tree (or nothing for bare
repos). That fails when core.worktree, or GIT_DIR and GIT_WORK_TREE are
used to separate the two.

This was attacked at the type level, by storing the gitdir and worktree
separately, so Nothing for the worktree means a bare repo.

A complication arose because we don't learn where a repository is bare
until its configuration is read. So another Location type handles
repositories that have not had their config read yet. I am not entirely
happy with this being a Location type, rather than representing them
entirely separate from the Git type. The new code is not worse than the
old, but better types could enforce more safety.

Added support for core.worktree. Overriding it with -c isn't supported
because it's not really clear what to do if a git repo's config is read, is
not bare, and is then overridden to bare. What is the right git directory
in this case? I will worry about this if/when someone has a use case for
overriding core.worktree with -c. (See Git.Config.updateLocation)

Also removed and renamed some functions like gitDir and workTree that
misused git's terminology.

One minor regression is known: git annex add in a bare repository does not
print a nice error message, but runs git ls-files in a way that fails
earlier with a less nice error message. This is because before --work-tree
was always passed to git commands, even in a bare repo, while now it's not.
2012-05-18 17:03:12 -04:00
Joey Hess
f7d8982672 Fix use of several config settings
annex.ssh-options, annex.rsync-options, annex.bup-split-options.

And adjust types to avoid the bugs that broke several config settings
recently. Now "annex." prefixing is enforced at the type level.
2012-05-05 20:16:56 -04:00
Joey Hess
ed79596b75 noop 2012-04-21 23:32:33 -04:00
Joey Hess
bee420bd2d in which I discover void
void :: Functor f => f a -> f () -- ah, of course that's useful :)
2012-04-21 23:06:19 -04:00
Joey Hess
b98b69e8c6 honor core.sharedRepository when making all the other files in the annex
Lock files, directories, etc.
2012-04-21 19:36:03 -04:00
Joey Hess
f1398b5583 use new getConfig 2012-03-22 17:32:47 -04:00
Joey Hess
60ab3d84e1 added ifM and nuked 11 lines of code
no behavior changes
2012-03-14 17:43:34 -04:00
Joey Hess
d08ee1a9d2 syscall optimisation 2012-03-06 13:56:20 -04:00
Joey Hess
12b89a3eb8 configure: Check if ssh connection caching is supported by the installed version of ssh and default annex.sshcaching accordingly. 2012-02-25 19:15:29 -04:00
Joey Hess
a1e52f0ce5 hlint 2012-02-16 00:44:51 -04:00
Joey Hess
d55f3c0716 Fix teardown of stale cached ssh connections. 2012-02-09 21:49:46 -04:00
Joey Hess
146c36ca54 IO exception rework
ghc 7.4 comaplains about use of System.IO.Error to catch exceptions.
Ok, use Control.Exception, with variants specialized to only catch IO
exceptions.
2012-02-03 16:47:24 -04:00
Joey Hess
ba6088b249 rename readMaybe to readish
a stricter (but also partial) readMaybe is getting added to base
2012-01-23 17:00:10 -04:00
Joey Hess
eb9001044f order user provided params after connection caching params
So the user can override them.
2012-01-20 17:32:32 -04:00
Joey Hess
6ef82665de add annex.sshcaching config setting 2012-01-20 17:15:46 -04:00
Joey Hess
47250a153a ssh connection caching
Ssh connection caching is now enabled automatically by git-annex. Only one
ssh connection is made to each host per git-annex run, which can speed some
things up a lot, as well as avoiding repeated password prompts. Concurrent
git-annex processes also share ssh connections. Cached ssh connections are
shut down when git-annex exits.

Note: The rsync special remote does not yet participate in the ssh
connection caching.
2012-01-20 17:14:56 -04:00
Joey Hess
16e7178f20 reorg 2012-01-10 15:29:10 -04:00
Joey Hess
ee3b5b2a42 use Common in a few more modules 2011-12-20 14:37:53 -04:00
Joey Hess
02f1bd2bf4 split more stuff out of Git.hs 2011-12-14 15:43:13 -04:00
Joey Hess
d64132a43a hslint 2011-12-09 01:57:13 -04:00
Joey Hess
63a292324d add a UUID type
Should have done this a long time ago.
2011-11-07 15:59:16 -04:00
Joey Hess
91366c896d clean Annex stuff out of Utility/ 2011-10-16 00:04:26 -04:00
Renamed from Utility/Ssh.hs (Browse further)