Commit graph

1129 commits

Author SHA1 Message Date
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
ade6ed2d71 AMP hack 2015-05-31 16:54:07 -04:00
Joey Hess
bf17af16b7 merge haddock formatting from propellor 2015-05-30 11:27:40 -04:00
Joey Hess
fa38fe0cf6 qualify imports to avoid conflict with regex-tdfa-rc 2015-05-30 02:03:09 -04:00
Joey Hess
8990d4cc68 fsck: When checksumming a file fails due to a hardware fault, the file is now moved to the bad directory, and the fsck proceeds. Before, the fsck immediately failed. 2015-05-27 16:40:03 -04:00
Joey Hess
ff629a1ec0 implment catchHardwareFault 2015-05-27 16:36:54 -04:00
Joey Hess
7cb16b9efb improve error message when checksum command exits nonzero 2015-05-27 15:43:50 -04:00
Joey Hess
ccf307307d merge haddock documentation from propellor 2015-05-27 12:39:04 -04:00
Joey Hess
6c3cea7699 need more polymorphism 2015-05-22 13:50:37 -04:00
Joey Hess
696ed9b3f9 fix windows build 2015-05-22 13:44:24 -04:00
Joey Hess
9de5cd2966 fix crash in stale transfer lockfile cleanup code
Need to differentiate between the lockfile not being locked, and it not
existing.
2015-05-19 23:35:24 -04:00
Joey Hess
846e7db245 unused imports 2015-05-19 15:05:30 -04:00
Joey Hess
b47c9fd587 honor core.sharedRepository settings in lockContent
The content file may not be owned by the user running git-annex, in which
case, setting the owner write bit was not enough to let lockContent
act on the file. However, with some core.sharedRepository configs, the file
should be writable by the user's group. So, the thing to do is to call
thawContent on it.
2015-05-19 14:53:19 -04:00
Joey Hess
1312e721ed convert lockContent to use new LockPools
Also cleaned up the code, avoiding creating a lock file if we're going to
open it for create later anyway.

And, if there's an exception while preparing to lock the file, but not at
the point of actually taking the lock, throw an exception, instead of
silently not locking and pretending to succeed.

And, on Windows, always use lock file, even if the repo somehow got into
indirect mode (maybe with cygwin git..)
2015-05-19 14:12:23 -04:00
Joey Hess
ecb0d5c087 use lock pools throughout git-annex
The one exception is in Utility.Daemon. As long as a process only
daemonizes once, which seems reasonable, and as long as it avoids calling
checkDaemon once it's already running as a daemon, the fcntl locking
gotchas won't be a problem there.

Annex.LockFile has it's own separate lock pool layer, which has been
renamed to LockCache. This is a persistent cache of locks that persist
until closed.

This is not quite done; lockContent stil needs to be converted.
2015-05-19 14:09:52 -04:00
Joey Hess
6915b71c57 lock pools to work around non-concurrency/composition safety of POSIX fcntl 2015-05-18 15:57:17 -04:00
Joey Hess
e9172263e5 comment typos 2015-05-17 14:22:14 -04:00
Joey Hess
8c2dd7d8ee Fix an unlikely race that could result in two transfers of the same key running at once.
As discussed in bug report.
2015-05-12 19:39:28 -04:00
Joey Hess
03667a162a couple of AMP warnings I missed before 2015-05-10 16:51:03 -04:00
Joey Hess
5c7cdbae46 more {-# OPTIONS_GHC -fno-warn-tabs #-} ... Forcing people who have what is merely a difference of opinion to you to do this is a bit of an asshole move. Just saying. 2015-05-10 16:38:49 -04:00
Joey Hess
ed4fe02896 disable horrible tab warning, needed in every file that Setup.hs pulls in
This is certianly a cabal bug for not passing the build options in the
cabal file when building Setup.hs.

And, why oh why did ghc enable this warning by default? So unhappy with
this choice.
2015-05-10 16:31:50 -04:00
Joey Hess
ec267aa1ea rejigger imports for clean build with ghc 7.10's AMP changes
The explict import Prelude after import Control.Applicative is a trick
to avoid a warning.
2015-05-10 16:20:30 -04:00
Joey Hess
6466dbc950 FlexibleContexts needed by ghc 7.10 2015-05-10 15:37:55 -04:00
Joey Hess
fcb8394e5f switch to Control.Monad.ST to avoid build warning in ghc 7.10 2015-05-10 14:42:16 -04:00
Joey Hess
f395ca7a41 Improve quvi 0.4 output parsing to handle cases wher there is no known filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used. 2015-05-08 13:39:00 -04:00
Joey Hess
d9bfc78444 avoid using relative path from temp dir to dest file
That failed on OSX. The temp dir was
/var/folders/fb/pnwjj52n7fg0r9mnvpsfll180000gr/T/downloadurl
and the relative path
../../../../../../Volumes/Visitors/joeyh/git-annex/r/.git/...
Didn't work. I have no clue why, how did OSX manage to break this?

But, the relative path is longer most of the time anyway, so let's
just use the absolute path.
2015-05-07 18:47:24 -04:00
Magnus Therning
8ab56a5cf2 Replace dataenc with sandi.
The library dataenc is unmaintained and has been superseded by sandi.

Signed-off-by: Magnus Therning <magnus@therning.org>
2015-05-07 18:07:05 -04:00
Joey Hess
cf786f42a4 Support checking ftp urls for file presence. 2015-05-05 14:05:02 -04:00
Joey Hess
0b18228516 Work around wget bug #784348 which could cause it to clobber git-annex symlinks when downloading from ftp. 2015-05-05 13:53:06 -04:00
Joey Hess
3d8cf16eec add more generic interface 2015-05-05 13:51:31 -04:00
Joey Hess
38403092ae revert reversion merged from propellor 2015-04-29 14:15:08 -04:00
Joey Hess
d8ffa498ef avoid dupicate bracket import errror
Ambiguous occurrence `bracket'
    It could refer to either `Control.Exception.bracket',
                             imported from `Control.Exception' at Utility/FileMode.hs:14:27-33
                             (and originally defined in `Control.Exception.Base')
                          or `Utility.Exception.bracket',
                             imported from `Utility.Exception' at Utility/FileMode.hs:22:1-24
                             (and originally defined in `Control.Monad.Catch')
2015-04-29 13:43:11 -04:00
Joey Hess
d40dea58e3 merge generalization of writeFileProtected from propellor 2015-04-28 14:58:29 -04:00
Joey Hess
eb8ef44133 Dropped support for older versions of yesod and warp than the ones in Debian Jessie.
466 lines of compat cruft deleted!
2015-04-22 16:19:11 -04:00
Joey Hess
5f73bc3a12 Removed dependency on haskell SHA library, instead using cryptohash >= 0.11.0. 2015-04-19 11:05:32 -04:00
Joey Hess
be3895be45 refactor 2015-04-19 10:57:14 -04:00
Joey Hess
addc82dab7 removed all uses of undefined from code base
It's a code smell, can lead to hard to diagnose error messages.
2015-04-19 00:38:29 -04:00
Joey Hess
8489057e8d fsck --from remote: When bad content is found in the remote, and the local repo does not have a copy of the content, preserve the bad content in .git/annex/bad/ to avoid further data loss. 2015-04-18 14:13:07 -04:00
Joey Hess
cd31b69ff6 don't test with null paths 2015-04-14 15:15:29 -04:00
Joey Hess
f84ccaa4e8 fix relPathDirToFileAbs on windows with different drive letters
Since we started using this for git repos, when a remote was on another
drive, it resulted in a bogus relative path to it being used by git-annex,
which didn't work.
2015-04-14 14:16:44 -04:00
Joey Hess
2be4834822 info: Added --bytes option. 2015-04-12 14:08:40 -04:00
Joey Hess
c65e71e6a5 cleanup 2015-04-09 12:57:30 -04:00
Joey Hess
42281f12d6 bring back --quiet filtering of stdout and stderr, with deadlock fixed
I don't quite understand the cause of the deadlock. It only occurred
when git-annex-shell transferinfo was being spawned over ssh to feed
download transfer progress back. And if I removed this line from
feedprogressback, the deadlock didn't occur:
	bytes <- readSV v

The problem was not a leaked FD, as far as I could see. So what was it?
I don't know.

Anyway, this is a nice clean implementation, that avoids the deadlock.
Just fork off the async threads to handle filtering the stdout and stderr,
and let them clean up their handles whenever they decide to exit.

I've verified that the handles do get promptly closed, although a little
later than I would expect. Presumably that "little later" is what
was making waiting on the threads deadlock.

Despite the late exit, the last line of stdout and stderr appears where
I'd want it to, so I guess this is ok..
2015-04-06 20:20:52 -04:00
Joey Hess
0a89d55269 Fixes a bug in the last release that caused rsync and possibly other commands to hang at the end of a file transfer.
Stderr reader blocks waiting for all stderr, and so blocks the process ever
exiting.

I tried several ways to get around this, but no success yet. For now,
disable the stderr reader entirely.
2015-04-06 17:12:38 -04:00
Joey Hess
30aa902174 relay external special remote stderr through progress suppression machinery (eep!)
It sounds worse than it is. ;)

Some external special remotes may run commands that display progress on
stderr. If git-annex is run with --quiet, this should filter out such
displays while letting the errors through.
2015-04-04 14:54:03 -04:00
Joey Hess
2343f99c85 well along the way to fully quiet --quiet
Came up with a generic way to filter out progress messages while keeping
errors, for commands that use stderr for both.

--json mode will disable command outputs too.
2015-04-04 14:34:03 -04:00
Joey Hess
b2ad3403c6 make downloadQuiet quiet again
This was broken in commit c64ede23cd
2015-04-03 20:38:20 -04:00
Joey Hess
45e97191c3 fix stderr handling 2015-04-03 19:48:56 -04:00
Joey Hess
20fb91a7ad WIP on making --quiet silence progress, and infra for concurrent progress bars 2015-04-03 16:48:30 -04:00
Joey Hess
1c91024978 rename bothHandles -> ioHandles 2015-04-03 15:35:18 -04:00
Joey Hess
294991dacb Significantly sped up processing of large numbers of directories passed to a single git-annex command. (try 2)
New approach is to do it the expensive way for the first 100 paths
on the command line, but then assume the user doesn't care about order too
much and fall back to the cheap way that does not preserve order.
2015-04-02 01:44:32 -04:00
Joey Hess
f79502d377 Revert "Significantly sped up processing of large numbers of directories passed to a single git-annex command."
This reverts commit 705112903e.

Whoops, git ls-files does not always output in the input ordering.
That's why all this work is needed. Urk.
2015-04-02 01:24:37 -04:00
Joey Hess
bf563d5df0 comment 2015-04-02 00:34:06 -04:00
Joey Hess
ee4af54005 minor opt 2015-04-02 00:33:08 -04:00
Joey Hess
7b32e7acb5 make segmentXargs preserve order 2015-04-02 00:31:36 -04:00
Joey Hess
705112903e Significantly sped up processing of large numbers of directories passed to a single git-annex command. 2015-04-02 00:19:49 -04:00
Joey Hess
6dd13d2684 micro-opt 2015-04-01 23:14:59 -04:00
Joey Hess
b8f0b7309f Work around curl bug when asked to download an empty url to a file.
In this situation, curl -o exits successfully without creating the output
file.

There was already a workaround for curl file:/// but I did not realize this
also affected regular url downloads.

To fix it, pre-create the destination file before starting curl.
Since we cannot always know the size of an url before trying to download
it, let's always do this.

Note that since curl is told -C -, we have to consider if this
makes curl try to do a ranged download, which might fail on some servers
where a regular download would have succeeded. My testing indicates
this isn't a problem; since the file is empty, curl seems to not try to
do a ranged download.

Original report: https://github.com/datalad/datalad/issues/79
Curl bug report: https://github.com/bagder/curl/issues/183
2015-03-27 10:22:36 -04:00
Joey Hess
7cb2f91f5b update comment for segmentXargs 2015-03-26 13:48:12 -04:00
Joey Hess
9b93278e8a metadata: Fix encoding problem that led to mojibake when storing metadata strings that contained both unicode characters and a space (or '!') character.
The fix is to stop using w82s, which does not properly reconstitute unicode
strings. Instrad, use utf8 bytestring to get the [Word8] to base64. This
passes unicode through perfectly, including any invalid filesystem encoded
characters.

Note that toB64 / fromB64 are also used for creds and cipher
embedding. It would be unfortunate if this change broke those uses.

For cipher embedding, note that ciphers can contain arbitrary bytes (should
really be using ByteString.Char8 there). Testing indicated it's not safe to
use the new fromB64 there; I think that characters were incorrectly
combined.

For credpair embedding, the username or password could contain unicode.
Before, that unicode would fail to round-trip through the b64.
So, I guess this is not going to break any embedded creds that worked
before.

This bug may have affected some creds before, and if so,
this change will not fix old ones, but should fix new ones at least.
2015-03-04 12:54:30 -04:00
Joey Hess
6726ce8116 fix intentional typo left in after debugging session, broke quvi support (last release is ok) 2015-02-19 14:31:04 -04:00
Joey Hess
a787cead35 bittorrent: Fix mojibake introduced in parsing arai2c progress output.
hGetSomeString reads one byte at a time, so unicode bytes are not composed.
The problem comes when outputting that to the console with hPut; that
tried to apply the handle's encoding, and so we get mojibake.

Instead, use ByteStrings, and only convert it to a string for parsing, not
for display.

Note that there are a couple of other things that use hGetSomeString,
which I've left as-is for now.
2015-02-10 12:34:34 -04:00
Joey Hess
d3b61d69b1 addurl: Avoid crash if quvi is not installed, when git-annex was built with process-1.2
createProcess has been changed to throw an exception if the program is
not in path.
2015-02-10 11:59:59 -04:00
Joey Hess
3e78b83875 Windows: Fix bug in dropping an annexed file, which caused a symlink to be staged that contained backslashes. 2015-02-09 15:37:26 -04:00
Joey Hess
8eb01bc894 Added MD5 and MD5E backends. 2015-02-04 13:47:54 -04:00
Joey Hess
95c1593098 Remove support for building without cryptohash.
This will prevent backporting to wheezy, but it's time to simplify the
code.
2015-02-04 13:41:26 -04:00
Joey Hess
e8c376e0ad import Data.Default in Common 2015-01-28 16:11:28 -04:00
Joey Hess
587f6a919b addurl: When a Content-Disposition header suggests a filename to use, addurl will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this) 2015-01-22 14:52:52 -04:00
Joey Hess
91f1b2bdcf excess indent 2015-01-22 13:47:06 -04:00
Joey Hess
d29bf2f5b2 merge change from propellor
exporting Unit allows custom data units
2015-01-21 23:13:50 -04:00
Joey Hess
09a66f702d Revert "remove absNormPathUnix, using my absPathFrom replacement"
This reverts commit a7f05c007b.

Consider: relPathDirToFile (absPathFrom "/tmp/repo/xxx" "y/bar") "/tmp/repo/.git/annex/objects/xxx"

This needs to always yield "../../../.git/annex/objects/xxx" but on
Windows, it is "..\\..\\/tmp/repo/.git/annex/objects/xxx"
2015-01-21 13:54:47 -04:00
Joey Hess
a7f05c007b remove absNormPathUnix, using my absPathFrom replacement 2015-01-21 13:37:09 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
068aaf943b on second thought, InodeCache should use getFileSize
This is necessary for interop between inode caches created on unix and
windows. Which is more important than supporting inodecaches for large keys
with the wrong size, which are broken anyway.

There should be no slowdown from this change, except on Windows.
2015-01-20 19:35:50 -04:00
Joey Hess
7a14edce5a build fix 2015-01-20 17:27:42 -04:00
Joey Hess
27c24b7407 fix upFrom of "." (and upFrom of "/" on Windows) 2015-01-20 17:23:23 -04:00
Joey Hess
4f657aa14e add getFileSize, which can get the real size of a large file on Windows
Avoid using fileSize which maxes out at just 2 gb on Windows.
Instead, use hFileSize, which doesn't have a bounded size.
Fixes support for files > 2 gb on Windows.

Note that the InodeCache code only needs to compare a file size,
so it doesn't matter it the file size wraps. So it has been
left as-is. This was necessary both to avoid invalidating existing inode
caches, and because the code passed FileStatus around and would have become
more expensive if it called getFileSize.

This commit was sponsored by Christian Dietrich.
2015-01-20 17:09:24 -04:00
Joey Hess
f4de021a54 convert parentDir to be based on takeDirectory, but fixed for trailing / 2015-01-09 14:26:52 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
c70bbdfe9d fix build failure 2015-01-09 11:58:38 -04:00
Joey Hess
858d776352 Merge branch 'master' into relativepaths
Conflicts:
	Locations.hs
	debian/changelog
2015-01-06 19:00:01 -04:00
Joey Hess
9d75709bf7 build fix 2015-01-06 18:59:13 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
cd865c3b8f Switch to using relative paths to the git repository.
This allows the git repository to be moved while git-annex is running in
it, with fewer problems.

On Windows, this avoids some of the problems with the absurdly small
MAX_PATH of 260 bytes. In particular, git-annex repositories should
work in deeper/longer directory structures than before. See
http://git-annex.branchable.com/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/

There are several possible ways this change could break git-annex:

1. If it changes its working directory while it's running, that would
   be Bad News. Good news everyone! git-annex never does so. It would also
   break thread safety, so all such things were stomped out long ago.

2. parentDir "." -> "" which is not a valid path. I had to fix one
   instace of this, and I should probably wipe all calls to parentDir out
   of the git-annex code base; it was never a good idea.

3. Things like relPathDirToFile require absolute input paths,
   and code assumes that the git repo path is absolute and passes it to it
   as-is. In the case of relPathDirToFile, I converted it to not make
   this assumption.

Currently, the test suite has 16 failures.
2015-01-06 16:19:41 -04:00
Joey Hess
5db31e2386 Fix build with process 1.2.1.0. 2015-01-04 13:07:57 -04:00
Joey Hess
e013183280 OSX: Switched away from deprecated statfs64 interface.
Getting rid of build warning

warning: 'statfs64' is deprecated: first deprecated in OS X 10.6
[-Wdeprecated-declarations]

10.6 is much older than the oldest git-annex OSX port, so won't break
anything.
2014-12-31 12:20:27 -04:00
Joey Hess
23051899e6 Windows: Got the rsync special remote working.
More aggressive rsync params fixup for windows. Param may contain a url, or
a file path, so check if it looks like a local file path and if so, fix it
up.

On windows only, rsyncUrlIsPath will treat c:foo as a path, rather than as
a rsyncurl starting with a host "c".
2014-12-30 15:05:59 -04:00
Joey Hess
5b30ba63cd Windows: Fix local rsync filepath munging (fixes 26 test suite failures). 2014-12-30 13:57:49 -04:00
Joey Hess
cdc69d7477 final warning fix 2014-12-29 17:45:36 -04:00
Joey Hess
ef82f0bcef fix 2014-12-29 17:44:58 -04:00
Joey Hess
7c511f900d fix more warnings on BSD/OSX 2014-12-29 17:43:26 -04:00
Joey Hess
66cdc4d612 one more warning 2014-12-29 17:35:19 -04:00
Joey Hess
c9a3e80d32 fixed all remaining build warnings on Windows 2014-12-29 17:30:20 -04:00
Joey Hess
67828b1c86 fix a OSX C compiler warning about null pointer 2014-12-29 17:07:42 -04:00
Joey Hess
415b2c53ed fix OSX build warning 2014-12-29 16:38:52 -04:00
Joey Hess
02f150b33d Work around statfs() overflow on some XFS systems.
statfs(".", 0xffa8ad50)                 = -1 EOVERFLOW (Value too large for defined data type)

Ref <20141222221621.GO7251@onerussian.com>
2014-12-23 11:52:23 -04:00
Joey Hess
1c88b59bd0 refactor 2014-12-17 13:21:55 -04:00
Joey Hess
c64ede23cd Use wget -q --show-progress for less verbose wget output, when built with wget 1.16. 2014-12-16 14:04:40 -04:00
Joey Hess
67c05daf5e sanitize filepaths provided by checkUrl 2014-12-11 20:08:49 -04:00
Joey Hess
0da402e0b9 more general readProcess' merged from propellor 2014-11-19 00:59:45 -04:00
Joey Hess
63fa5c0782 Work around apparent bug in lsof 4.88's -F output format. 2014-11-12 18:00:40 -04:00
Joey Hess
c3390f4c98 generalize monads 2014-11-12 14:59:24 -04:00
Joey Hess
5d946fe3a9 switch from hGetSome to hGet
This should be essentially no-op change for hGetContentsMetered, since it
always gets the entire contents. So the only difference is that each chunk
of the lazy bytestring will always be the full chunk size. So, I'm pretty
sure this is safe. Also, the only current users of hGetContentsMetered are
reading files, so the stream won't block for long in the middle.

The improvement is that hGetUntilMetered will always get some multiple of
the defaultChunkSize. This will allow the S3 multipart code to pick a fixed
size and know that hGetUntilMetered will really get that size.

(cherry picked from commit bd09046291)
2014-11-03 22:11:47 -04:00
Joey Hess
0602b26314 hGetUntilMetered 2014-11-03 18:37:05 -04:00
Joey Hess
0f6aaf8012 Windows: Fix crash when user.name is not set in git config. 2014-10-31 16:14:12 -04:00
Joey Hess
86b326782a getUserEntryForID does not work on windows
Throw an error instead, mentioning the relevant env vars.
2014-10-31 15:46:44 -04:00
Joey Hess
613f396efd more accurate version bounds 2014-10-24 12:11:27 -04:00
Joey Hess
ce9076370e deal with yesod's second gratuitous rename of the same function in a year 2014-10-23 20:26:46 -04:00
Joey Hess
431c1eafe5 typo 2014-10-16 15:39:19 -04:00
Joey Hess
b6be403d3a typos 2014-10-16 15:37:12 -04:00
Joey Hess
1e59df083d Use haskell setenv library to clean up several ugly workarounds for inability to manipulate the environment on windows.
Didn't know that this library existed!

This includes making git-annex not re-exec itself on start on windows, and
making the test suite on Windows run tests without forking.
2014-10-15 20:33:52 -04:00
Joey Hess
9c8abb761f revert reversion
This reverts commit 7bfc4a5442.

Android build is going to have consistent versions again.
2014-10-15 11:49:13 -04:00
Joey Hess
7bfc4a5442 unbreak android build
This reverts commit dd667844b6
and commit e6eff0e951.

Those commits were fine, except the android autobuilder currently has a bit
of a mess of yesod versions and broke. Better to wait on this.
2014-10-14 14:38:38 -04:00
Joey Hess
e6eff0e951 more accurate bounds on yesod boostrap3 ifdefs 2014-10-13 15:33:51 -04:00
Joey Hess
dd667844b6 avoid using Assistant.WebApp.Bootstrap3 when building with current yesod
Only use that when building with ancient yesod, which does not include it.

This also let me remove ifdefs in the file to support building with the new
version of yesod.
2014-10-09 15:19:24 -04:00
Joey Hess
9fd95d9025 indent with tabs not spaces
Found these with:
git grep "^  " $(find -type  f -name \*.hs) |grep -v ':  where'

Unfortunately there is some inline hamlet that cannot use tabs for
indentation.

Also, Assistant/WebApp/Bootstrap3.hs is a copy of a module and so I'm
leaving it as-is.
2014-10-09 15:09:26 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
2abeae7582 Windows: Avoid crashing trying to list gpg secret keys, for gcrypt which is not yet supported on Windows. 2014-09-16 13:40:44 -04:00
Joey Hess
7482166180 Rather than crashing when there's a problem with the requested bloomfilter capacity/accuracy, fall back to a reasonable default bloom filter size. 2014-09-12 12:26:12 -04:00
Joey Hess
6eb5c3f479 Do not preserve permissions and acls when copying files from one local git repository to another. Timestamps are still preserved as long as cp --preserve=timestamps is supported.
This avoids cp -a overriding the default mode acls that the user might have
set in a git repository.

With GNU cp, this behavior change should not be a breaking change, because
git-anex also uses rsync sometimes in the same situation, and has only ever
preserved timestamps when using rsync.

Systems without GNU cp will no longer use cp -a, but instead just cp.
So, timestamps will no longer be preserved. Preserving timestamps when
copying between repos is not guaranteed anyway.

Closes: #729757
2014-08-26 17:10:25 -07:00
Joey Hess
2b234634f6 fix imports for windows 2014-08-23 16:27:24 -07:00
Joey Hess
59eae904b1 final scary locking refactoring (for now)
Note that while before checkTransfer this called getLock with WriteLock,
getLockStatus's use of ReadLock will also notice any exclusive locks.
Since transfer info files are only locked exclusively, never shared,
there is no behavior change.

Also, fixes checkLocked to actually return Just False when the file
exists, but is not locked.
2014-08-20 19:30:40 -04:00
Joey Hess
1994771215 more lock file refactoring
Also fixes a test suite failures introduced in recent commits, where
inAnnexSafe failed in indirect mode, since it tried to open the lock file
ReadWrite. This is why the new checkLocked opens it ReadOnly.

This commit was sponsored by Chad Horohoe.
2014-08-20 18:58:14 -04:00
Joey Hess
ec7dd0446a more lock file refactoring 2014-08-20 17:03:04 -04:00
Joey Hess
d279180266 reorganize and refactor lock code
Added a convenience Utility.LockFile that is not a windows/posix
portability shim, but still manages to cut down on the boilerplate around
locking.

This commit was sponsored by Johan Herland.
2014-08-20 16:45:58 -04:00
Joey Hess
092041fab0 Ensure that all lock fds are close-on-exec, fixing various problems with them being inherited by child processes such as git commands.
(With the exception of daemon pid locking.)

This fixes at part of #758630. I reproduced the assistant locking eg, a
removable drive's annex journal lock file and forking a long-running
git-cat-file process that inherited that lock.

This did not affect Windows.

Considered doing a portable Utility.LockFile layer, but git-annex uses
posix locks in several special ways that have no direct Windows equivilant,
and it seems like it would mostly be a complication.

This commit was sponsored by Protonet.
2014-08-20 11:37:02 -04:00
Joey Hess
8b15af309a add compat cruft for old versions of http-types and http-conduit 2014-08-17 15:39:46 -04:00
Joey Hess
6ab0737a75 work around default Accept-Encoding in http-client 2014-08-15 18:02:17 -04:00
Joey Hess
e0227dfedf memoize construction of the Request -> Request function to apply the UrlOptions 2014-08-15 17:47:21 -04:00
Joey Hess
dd619c7166 Switched from the old haskell HTTP library to http-conduit.
The hoary old HTTP library was only used when checking if an url exists,
when curl was not available. It had many problems, including not supporting
https at all.

Now, this is done using http-conduit for all urls that it supports. Falls
back to curl for any url that http-conduit doesn't like (probably ftp etc,
but could also be an url that its parser chokes on for whatever reason).

This adds a new dependency on http-conduit, but webdav support already
indirectly depended on that, and the s3-aws branch also uses it.

This opens up the possibility of using http-conduit for large file
downloads, but for now I've left it using wget/curl.

This commit was sponsored by Paul Tötterman.
2014-08-15 17:37:42 -04:00
Joey Hess
9193d6e6bd fix build on windows with unix-compat-0.4.1.3
FileID type changed, needs Arbitrary instance.

On the plus side, getFileStatus on Windows now actually gets file id's,
not always 0, so direct mode is safer there now.
2014-08-11 17:25:31 -04:00
Joey Hess
fb66ee8c85 qualify catch and try
got a build failure on android due to there being a Prelude.catch
2014-08-10 19:40:21 -04:00
Joey Hess
c27d9f909b fix windows build 2014-08-10 15:30:55 -04:00
Joey Hess
c784ef4586 unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.

Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.

Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.

However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.

Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-07 22:03:29 -04:00
Joey Hess
8e3d62dd5d generalized using the extensions package 2014-08-07 21:01:07 -04:00
Joey Hess
6c450aad1d move ugly rsync zombie workaround
This reaping of any processes came to cause me problems when redoing the
rsync special remote -- a gpg process that was running gets waited on and
the place that then checks its return code fails.

I cannot reproduce any zombies when using the rsync special remote.
But I still can when using a normal git remote, accessed over ssh.
There is 1 zombie per file downloaded without this horrible hack enabled.

So, move the hack to only be used in that case.
2014-08-03 16:53:29 -04:00
Joey Hess
9720ee9e56 testremote: New command to test uploads/downloads to a remote.
This only performs some basic tests so far; no testing of chunking or
resuming. Also, the existing encryption type of the remote is used; it
would be good later to derive an encrypted and a non-encrypted version of
the remote and test them both.

This commit was sponsored by Joseph Liu.
2014-08-01 15:10:01 -04:00
Joey Hess
1d263e1e7e lift types from IO to Annex
Some remotes like External need to run store and retrieve actions in Annex,
not IO. In order to do that lift, I had to dive pretty deep into the
utilities, making Utility.Gpg and Utility.Tmp be partly converted to using
MonadIO, and Control.Monad.Catch for exception handling.

There should be no behavior changes in this commit.

This commit was sponsored by Michael Barabanov.
2014-07-29 16:28:44 -04:00
Joey Hess
9d4a766cd7 resume interrupted chunked downloads
Leverage the new chunked remotes to automatically resume downloads.
Sort of like rsync, although of course not as efficient since this
needs to start at a chunk boundry.

But, unlike rsync, this method will work for S3, WebDAV, external
special remotes, etc, etc. Only directory special remotes so far,
but many more soon!

This implementation will also properly handle starting a download
from one remote, interrupting, and resuming from another one, and so on.

(Resuming interrupted chunked uploads is similarly doable, although
slightly more expensive.)

This commit was sponsored by Thomas Djärv.
2014-07-27 18:56:32 -04:00
Joey Hess
ab4cce4114 core implementation of new style chunking
Not yet used by any special remotes, but should not be too hard to add it
to most of them.

storeChunks is the hairy bit! It's loosely based on
Remote.Directory.storeLegacyChunked. The object is read in using a lazy
bytestring, which is streamed though, creating chunks as needed, without
ever buffering more than 1 chunk in memory.

Getting the progress meter update to work right was also fun, since
progress meter values are absolute. Finessed by constructing an offset
meter.

This commit was sponsored by Richard Collins.
2014-07-25 16:20:32 -04:00
Joey Hess
cf384cddfb increase sleep 2014-07-16 16:16:25 -04:00
Joey Hess
aacd5c6994 Windows: fix crash after 10 minutes
getLine in waitForTermination doesn't work when stdin is closed..
Just loop forever, there was no reason to getLine here I think.
2014-07-16 15:42:12 -04:00
Joey Hess
339a3b8ab0 Windows: Fix locking issue that prevented the webapp starting (since 5.20140707).
Reversion introduced in 9fb8038166.
The locking code was wrong; the webapp re-ran itself, saw pid was locked,
and so didn't start!
2014-07-14 15:52:00 -04:00
Joey Hess
e36439d4fe deal with process-1.2.0.0 throwing an exception when program DNE (may only fix the tip of the iceberg)
Configure crashed on systems with that process and without eg, sha256sum.

The rest of the code in configure looks to work ok, since it uses sh -c to
probe for commands, and sh is always in path so it works.

Dunno about all the rest of git-annex. Not a huge amount of external
program use, other than git, so perhaps this won't be a large pain.

Note that boolSystem can throw an exception now if the program doesn't
exist. Could easily be changed back to False.
2014-07-13 22:33:28 -04:00
Joey Hess
e5b88713a1 refactor 2014-07-10 00:16:53 -04:00
Joey Hess
d9d76cf98b Fix minor FD leak in journal code.
Minor because normally only 1 FD is leaked per git-annex run. However,
the test suite leaks a few hundred FDs, and this broke it on the Debian
autobuilders, which seem to have a tigher than usual ulimit.

The leak was introduced by the lazy getDirectoryContents' that was
introduced in e6330988dd in order to scale to
millions of journal files -- if the lazy list was never fully consumed, the
directory handle did not get closed.

Instead, pull in openDirectory/readDirectory/closeDirectory code that I
already developed and submitted in a patch to the haskell directory library
earlier. Using this in journalDirty avoids the place that the lazy list
caused a problem. And using it in stageJournal eliminates the need for
getDirectoryContents'.

The getJournalFiles* functions are switched back to using the regular
strict getDirectoryContents. I'm not sure if those always consume the whole
list, so this avoids any leak. And the things that call those are things
like git annex unused, which also look at every file committed to the
git-annex branch, so would need more work to scale to insane numbers of
files anyway.
2014-07-09 23:36:53 -04:00
Joey Hess
33ad8beabb Support building with bloomfilter 2.0.0. 2014-07-07 12:24:12 -04:00
Joey Hess
c5c4fd76ec fix windows build 2014-07-05 11:18:59 -04:00
Joey Hess
c90e4e8778
work around getDirectoryContents not streaming lazily 2014-07-04 17:59:26 -04:00
Joey Hess
9fb8038166 Windows: Assistant now logs to daemon.log.
Yes, this means that git annex webapp on windows execs git-annex, which
execs itself to set env, and the execs itself again to redirect logs.

This is disgusting. This is Windows(TM).
2014-06-17 19:27:54 -04:00
Joey Hess
4fe2e53f5b finish fixing windows timezone madness
Rather than calculating the TSDelta once, and caching it, this now
reads the inode sential file's InodeCache file once, and then each time a
new InodeCache is generated, looks at the sentinal file to get the current
delta.

This way, if the time zone changes while git-annex is running, it will
adapt.

This adds some inneffiency, but only on Windows, and only 1 stat per new
file added. The worst innefficiency is that `git annex status` and
`git annex sync` will now (on Windows) stat the inode sentinal file once per
file in the repo.

It would be more efficient to use getCurrentTimeZone, rather than needing
to stat the sentinal file. This should be easy to do, once the time
package gets my bugfix patch.

This commit was sponsored by Jürgen Lüters.
2014-06-12 13:54:08 -04:00
Joey Hess
e4d7e2ebde fix for Windows file timestamp timezone madness
On Windows, changing the time zone causes the apparent mtime of files to
change. This confuses git-annex, which natually thinks this means the files
have actually been modified (since THAT'S WHAT A MTIME IS FOR, BILL <sheesh>).

Work around this stupidity, by using the inode sentinal file to detect if
the timezone has changed, and calculate a TSDelta, which will be applied
when generating InodeCaches.

This should add no overhead at all on unix. Indeed, I sped up a few
things slightly in the refactoring.

Seems to basically work! But it has a big known problem:
If the timezone changes while the assistant (or a long-running command)
runs, it won't notice, since it only checks the inode cache once, and
so will use the old delta for all new inode caches it generates for new
files it's added. Which will result in them seeming changed the next time
it runs.

This commit was sponsored by Vincent Demeester.
2014-06-12 13:42:21 -04:00
Joey Hess
db8982c45b minimal exports 2014-06-11 16:17:01 -04:00
Joey Hess
1c18056e15 deal with FAT on Linux timestamp issue
Deal with FAT's low resolution timestamps, which in combination with
Linux's caching of higher res timestamps while a FAT is mounted, caused
direct mode repositories on FAT to seem to have modified files after they
were unmounted and remounted.

This commit was sponsored by Fabrice Rossi.
2014-06-11 14:46:03 -04:00
Joey Hess
8ff9938d97 Fix build with wai 0.3.0.
This version of wai changed the type of Middleware, so I cannot seem
to liftIO inside it. So, got rid of a lot of not really needed
complexity to use System.Log.Logger's logging stuff, and just use
the standard wai stdout logger when debug logging is enabled.

Format may change some, and it logs http to stdout instead of stderr
now. Doesn't matter for the webapp since both go to the same log anyway.
2014-06-11 01:29:00 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
6eb5e6c135 Windows: Fix opening webapp when repository is in a directory with spaces in the path. 2014-06-10 18:37:33 -04:00
Joey Hess
7671d228af Avoid leaving behind .tmp files when failing in some cases, including importing files to a disk that is full. 2014-06-09 15:24:05 -04:00
Joey Hess
787d9e27ba Ignore setsid failures. 2014-06-09 14:44:18 -04:00
Joey Hess
a59ac68c22 qualify comment 2014-06-09 13:48:44 -04:00
Joey Hess
b7560e660e
fix build with old versions of warp 2014-06-04 14:37:08 -04:00
Joey Hess
0eb6576455 lazy bytestring fromStrict is not available in stable; use fromChunks instead 2014-05-29 20:25:01 -04:00
Joey Hess
450eecaa76 fix warning message
This should work even back in debian stable's warp
2014-05-29 13:49:45 -04:00
Joey Hess
b197ec8917 get rid of (completely safe) uses of Char8
Char8 often indicates an encoding bug. It didn't here, but I can avoid it
and not worry about it.
2014-05-27 20:26:10 -04:00
Joey Hess
a66c942645 refactor 2014-05-14 17:31:20 -04:00
Joey Hess
740de08461 fix windows build 2014-05-14 15:54:41 -04:00
Joey Hess
e391224516 setsid when running webapp in foreground too
This avoids ssh prompting for passwords on stdin, ever.

It may also change other behavior of other programs, as there is no
controlling terminal now. However, setsid was already done when running the
assistant in daemon mode, so any behavior changes should not be really new.
2014-05-14 14:26:28 -04:00
Joey Hess
2427832bed relicense general utility library code to BSD
Omitted a couple of files what have had significant contributions from
others.
2014-05-10 11:01:27 -03:00
Joey Hess
2aed2d8510 Merge branch 'master' into bootstrap3
Conflicts:
	debian/changelog
2014-05-02 15:32:49 -03:00
Joey Hess
72ec0ab736 hlint 2014-04-26 19:25:05 -04:00
Sören Brunk
54fe9af0bb create a local copy of Yesod.Form.Bootstrap3 2014-04-25 12:26:33 +02:00
Sören Brunk
00c1cd0db1 bootstrap3 forms 2014-04-21 19:47:05 +02:00
Joey Hess
515d251127
better approach for building distributionupdate
Use cabal include file
2014-04-21 10:57:01 -04:00
Joey Hess
a10d3731b8 fix distributionupdate build
Including avoiding needing cabal's defines for Utility.URI
2014-04-21 10:47:26 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
e53a85743e
adjust to not use cpp in modules used by configure 2014-04-14 13:37:12 -04:00
Joey Hess
2ff9ba9f74
add missing Network.URI Ord instance for Debian stable 2014-04-14 13:25:49 -04:00
Joey Hess
15917ec1a8 sync, assistant, remotedaemon: Use ssh connection caching for git pushes and pulls.
For sync, saves 1 ssh connection per remote. For remotedaemon, the same
ssh connection that is already open to run git-annex-shell notifychanges
is reused to pull from the remote.

Only potential problem is that this also enables connection caching
when the assistant syncs with a ssh remote. Including the sync it does
when a network connection has just come up. In that case, cached ssh
connections are likely to be stale, and so using them would hang.
Until I'm sure such problems have been dealt with, this commit needs to
stay on the remotecontrol branch, and not be merged to master.

This commit was sponsored by Alexandre Dupas.
2014-04-12 15:59:34 -04:00
Joey Hess
9fff243ff2
wrote test case; found bugs in date math; fixed and simplified using Data.Time.Calendar 2014-04-12 13:29:35 -04:00
Joey Hess
b815988d16
Improve handling on monthly/yearly scheduling.
Avoid back-to-back runs.
2014-04-12 12:58:32 -04:00
Joey Hess
d955cfe78a Improve handling on monthly/yearly scheduling.
Code was still buggy, it turns out (though the recursion checker caught
it). In the case of (Schedule (Monthly Nothing) AnyTime), where the last
run was on yyyy-12-31, it looped forever.

Also, the handling of (Schedule (Yearly Nothing) AnyTime) was wacky where
the last run was yyyy-12-31. It would suggest a window starting on the 3rd
for the next run (because 31 mod 28 is 3).

I think that originally I was wanted to avoid running on 01-01 if it had
just run on 12-31. But the code didn't accomplish this, and it's not
necessary anyway. This is supposed to calculate the next window meeting the
schedule, and for (Schedule (Monthly Nothing), the window starts at 01-01
and runs through 01-31. If that causes two back-to-back runs, well the next
one will not be until 02-01 at the earliest.

Also, back-to-back runs can be avoided, if desired, by using Divisible 2.
2014-04-11 21:42:43 -04:00
Joey Hess
e26dead493 guard against any remaining infinite recursion bugs; throw error if no candidate cn be found in next hundred years
Note that the exception thrown is not visible in the webapp currently
because it crashes one of Cronner's 2 worker threads, which is never
checked.
2014-04-11 18:08:46 -04:00
Joey Hess
9e16171c7f Last release didn't quite fix the high cpu issue in all cases, this should.
This is supposed to look for a day past the last day it ran, not a month
past.

Seems to work, at least in anarcat's test case.
2014-04-11 17:14:19 -04:00
Joey Hess
e0ca99b480
rename confusingly named "day" 2014-04-11 14:38:23 -04:00
Joey Hess
c678798a5c assistant: Fix high CPU usage triggered when a monthly fsck is scheduled, and the last time the job ran was a day of the month > 12. This caused a runaway loop. Thanks to Anarcat for his assistance, and to Maximiliano Curia for identifying the cause of this bug. 2014-04-11 14:34:09 -04:00
Joey Hess
a6df4698a3 Fix rsync progress parsing in locales that use comma in number display. Closes: #744148 2014-04-10 15:36:51 -04:00
Joey Hess
38862bf65d merge from propellor 2014-04-09 01:15:24 -04:00
Joey Hess
cbcb7f50d8 remotedaemon: avoid extraneous stdout output 2014-04-08 14:02:25 -04:00
Joey Hess
5af30678c7 factored out Utility.SimpleProtocol from the external special remote implementation 2014-04-05 13:29:28 -04:00
Joey Hess
611e060385 factor out library code (also used by propellor) 2014-04-03 21:25:59 -04:00
Joey Hess
b98db5d6f1 clean up warning in windows 2014-04-02 18:08:35 -04:00
Joey Hess
9b707f0116
windows fix try 2 2014-04-02 18:02:32 -04:00
Joey Hess
43ed9c7383
fix windows build 2014-04-02 17:21:36 -04:00
Joey Hess
523a98a952 merge from propellor 2014-04-01 17:53:55 -04:00
Joey Hess
bbd2efede3 refactor 2014-03-31 20:15:16 -04:00
Joey Hess
be76a0e22c avoid importing Common
it caused a loop when reusing this in propellor
2014-03-30 18:43:05 -04:00
Joey Hess
065248f3d2 Added required content configuration.
This includes checking when dropping files that any required content
configuration is satisfied. However, it does not yet include an active
check on the required content; the location log is trusted when checking
the required content expression.
2014-03-29 16:03:33 -04:00
Joey Hess
16387edd00 avoid exception when curl exits nonzero (due to eg, bad domain name) 2014-03-27 13:01:57 -04:00
Joey Hess
e426fac273 add desktop notifications
Motivation: Hook scripts for nautilus or other file managers
need to provide the user with feedback that a file is being downloaded.

This commit was sponsored by THM Schoemaker.
2014-03-22 14:12:19 -04:00
Joey Hess
b51f0e80ed
fix build with QuickCheck 2.7.1
For some reason this was working w/o a cast before, despite POSIXTime etc
being newtypes. It stopped working with the new QuickCheck:

Utility/QuickCheck.hs:31:33:
    No instance for (Integral POSIXTime)
      arising from a use of `arbitrarySizedIntegral'
    Possible fix: add an instance declaration for (Integral POSIXTime)
    In the first argument of `nonNegative', namely
      `arbitrarySizedIntegral'
    In the expression: nonNegative arbitrarySizedIntegral
    In an equation for `arbitrary':
        arbitrary = nonNegative arbitrarySizedIntegral
2014-03-20 11:08:40 -04:00
Joey Hess
1052eeface Windows: Fix some filename encoding bugs.
http://git-annex.branchable.com/bugs/Unicode_file_names_ignored_on_Windows/

Not a complete fix yet.
2014-03-19 15:57:56 -04:00
Joey Hess
b7b9afa9ad Better workaround for problem umasks when eg, setting up ssh keys. 2014-03-14 13:37:58 -04:00
Joey Hess
66b8b9c094 webapp: Use securemem for constant time auth token comparisons.
Debian stable does not have securemem, but neither does it have warp-tls,
so just disable use of securemem when not building with https support.
2014-03-12 21:41:20 -04:00
Joey Hess
06218428c7 quick fix to build with old warp-tls
Debian stable's warp-tls is too old to support the new https feature well,
so only use http with that old version.

Note that the webapp still depends on warp-tls, because the TLSSettings
type is used.
2014-03-12 12:19:48 -04:00
Joey Hess
5a37266080 use Monoid instance for Ordering 2014-03-09 14:08:45 -04:00
Joey Hess
a3fe8270ca annex.startupscan can be set to false to disable the assistant's startup scan. 2014-03-05 17:44:14 -04:00
Joey Hess
b9d6e70019
fix build w/o TDFA 2014-03-04 23:49:46 -04:00
Joey Hess
d41367cc08 remove buggy --listen=host:port support 2014-02-28 22:53:26 -04:00
Joey Hess
3c3744c9a9 use https when .git/annex/privkey.pem and .git/annex/certificate.pem exist (untested)
I have not managed to generate a key that is accepted by the old version of
warp-tls I have here.
2014-02-28 21:32:18 -04:00
Joey Hess
7ac37a7854 Probe for quvi version at run time.
Overhead: git annex addurl runs quvi --version once.
And more bloat to Annex state..
2014-02-28 14:54:02 -04:00
Joey Hess
3f6e4b8c7c fix all remaining -Wall warnings on Windows 2014-02-25 14:48:50 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
c69d6eb035 Make annex.web-options be used in several places that call curl. 2014-02-24 21:29:37 -04:00
Joey Hess
24f8136504 --metadata field=value can now use globs to match, and matches case insensatively, the same as git annex view field=value does.
Also refactored glob code into its own module.
2014-02-21 18:34:34 -04:00
Joey Hess
67fd06af76 add git annex view command
(And a vpop command, which is still a bit buggy.)

Still need to do vadd and vrm, though this also adds their documentation.

Currently not very happy with the view log data serialization. I had to
lose the TDFA regexps temporarily, so I can have Read/Show instances of
View. I expect the view log format will change in some incompatable way
later, probably adding last known refs for the parent branch to View
or something like that.

Anyway, it basically works, although it's a bit slow looking up the
metadata. The actual git branch construction is about as fast as it can be
using the current git plumbing.

This commit was sponsored by Peter Hogg.
2014-02-18 18:22:20 -04:00
Joey Hess
9633c67842 filter branches (incomplete)
Promosing work toward metadata driven filter branches. A few methods
to construct them are stubbed out; all the data types and pure code
seems good.

This commit was sponsored by Walter Somerville.
2014-02-16 17:39:54 -04:00
Joey Hess
9f883192af windows: Fix start for already running daemon 2014-02-13 17:38:56 -04:00
Joey Hess
c60f0b57d2 windows: Fix daemon pid file locking.
Well, as much as it can be fixed on windows. Not atomic; not entirely
guarded against the wrong process having the pid file locked.
2014-02-13 17:02:28 -04:00
Joey Hess
f11f7520b5 windows: Fix process termination code.
The ctrl-c hack used before didn't actually seem to work.

No haskell libraries expose TerminateProcess. I tried just calling it via
FFI, but got segfaults, probably to do with the wacky process handle not
being managed correctly. Moving it all into one C function worked.

This was hell. The EvilLinker hack was just final icing on the cake.
We all know what the cake was made of.
2014-02-13 15:53:10 -04:00
Joey Hess
1b79d18a40 data types and serialization for metadata
A very haskell commit! Just data types, instances to serialize the metadata
to a nice format, and QuickCheck tests.

This commit was sponsored by Andreas Leha.
2014-02-12 17:57:32 -04:00
Joey Hess
7b19c7d25b cleanup thanks to Utility.PID 2014-02-11 15:39:51 -04:00
Joey Hess
c390e896d1 fix windows build (and make --stop work on windows, incidentially)
The Utility.PID will clean up other code soon.
2014-02-11 15:25:59 -04:00
Joey Hess
b2fae4b78f
remove unimplemented windows stubs 2014-02-11 13:18:59 -04:00
Joey Hess
ae2bc90a1d eliminating absNormPath
git-annex has been using MissingH's `abdNormPath` forever, but that's
unmaintained and possibly buggy, and doesn't work on Windows. I've been
wanting to get rid of it for some time, and finally did today, writing a
`simplifyPath` that does the things git-annex needs and will work with all
the Windows filename craziness, and takes advantage of the more modern
System.FilePath to be quite a simple peice of code. A QuickCheck test found
no important divergences from absNormPath. A good first step to making
git-annex not depend on MissingH at all.

And it fixed some weird behaviors on Windows like
`git annex add ..\subdir\file` not working.

Note that absNormPathUnix has been left alone for now.
2014-02-07 17:15:18 -04:00
Joey Hess
3cbaa68be8 Android: Avoid crashing when unable to set file mode for ssh config file due to Android filesystem horribleness. 2014-02-07 14:57:23 -04:00
Joey Hess
dc3b7effd2 add regression test for symlink calculation
Note: Test reordered because running git-annex sync early broke the
environment for some other tests.
2014-02-06 20:57:58 -04:00
Joey Hess
f404379716 wiiindddoowws 2014-02-06 20:05:27 -04:00
Joey Hess
810b26e1d9 fix windows code again (argh) 2014-02-06 17:58:43 -04:00
Joey Hess
897d877472 work around absNormPath not working on Windows
When making git-annex links, we want unix-style paths in the link targets.
2014-02-06 17:17:35 -04:00
Joey Hess
28cabd9909 Revert "work around absNormPath not working on Windows"
This reverts commit 0fc3ad82c5.

That caused 2 other test cases to fail. Windows path slash issues are
horrible.
2014-02-06 16:59:39 -04:00
Joey Hess
0fc3ad82c5 work around absNormPath not working on Windows
Seems I punted on this while porting before. This hack relies on DOS not
using / in filenames, it's effectively an alternate path separatr in at
least current versions of windows..
2014-02-06 15:06:45 -04:00
Joey Hess
1572c460e8 avoid using openFile when withFile can be used
Potentially fixes some FD leak if an action on an opened file handle fails
for some reason. There have been some hard to reproduce reports of
git-annex leaking FDs, and this may solve them.
2014-02-03 10:19:06 -04:00
Joey Hess
fd1382f96f factor out utility function 2014-02-03 10:08:28 -04:00
Joey Hess
fd7e9431ff comment spelling 2014-02-03 10:01:49 -04:00
Joey Hess
e40a119e74 tell moveFile to overwrite existing files 2014-01-29 17:19:46 -04:00
Joey Hess
1ad3c1b2b3 fix windows build 2014-01-29 15:24:22 -04:00
Joey Hess
070ed4a766 change a few renameFile's to rename
AFAIK, none of these ever operate on directories, but nor do I want to
explicitly check if they're files and fail if not.
2014-01-29 15:21:02 -04:00
Joey Hess
1669e80e85 Windows: Avoid using unix-compat's rename, which refuses to rename directories.
Opened a bug about this: https://github.com/jystic/unix-compat/issues/10
2014-01-29 15:19:03 -04:00
Joey Hess
891c85cd88 use locking on Windows
This is all the easy cases, where there was already a separate lock file.
2014-01-28 14:42:03 -04:00
Joey Hess
8de4db664d Windows file locking (atrociously horrible) 2014-01-28 13:43:43 -04:00
Joey Hess
e0bd088f08 add webapp UI to manage unused files 2014-01-23 15:09:43 -04:00
Joey Hess
85aae97b63 add getDiskSize
Couldn't find anything that exposed this for Windows.
2014-01-22 22:19:52 -04:00
Joey Hess
207ac67aaa avoid needing a build-dep on hxt for Data.AssocList 2014-01-14 16:42:10 -04:00
Joey Hess
0cac4402ac Android: Avoid passing --clobber to busybox wget. 2014-01-13 14:52:49 -04:00
Joey Hess
7aec957c5b comment 2014-01-07 17:33:38 -04:00
Joey Hess
d8e8128b50 work around a strance change in gpg behavior; needs a trustdb to exist even when using --trust-model=always 2014-01-07 13:20:07 -04:00
Joey Hess
e5b4d447b6 assistant: Start a new git-annex transferkeys process after a network connection change
So that remotes that use a persistent network connection are restarted.

A remote might keep open a long duration network connection, and could
fail to deal well with losing the connection. This is particularly a
concern now that we have external special reotes. An external
special remote that is implemented naively might open the connection only
when PREPARE is sent, and if it loses connection, throw errors on each
request that is made.

(Note that the ssh connection caching should not have this problem; if the
long-duration ssh process loses connection, the named pipe is disconnected
and the next ssh attempt will reconnect. Also, XMPP already deals with
disconnection robustly in its own way.)

There's no way for git-annex to know if a lost network connection actually
affects a given remote, which might have a transfer in process. It does not
make sense to force kill the transferkeys process every time the NetWatcher
detects a change. (Especially because the NetWatcher sometimes polls 1
change per hour.)

In any case, the NetWatcher only detects connection to a network, not
disconnection. So if a transfer is in progress over the network, and the
network goes down, that will need to time out on its own.

An alternate approch that was considered is to use a separate transferkeys
process for each remote, and detect when a request fails, and assume that
means that process is in a failing state and restart it. The problem with
that approach is that if a resource is not available and a remote fails
every time, it degrades to starting a new transferkeys process for every
file transfer, which is too expensive.

Instead, this commit only handles the network reconnection case, and restarts
transferkeys only once the network has reconnected and another transfer needs
to be made. So, a transferkeys process will be reused for 1 hour, or until the
next network connection.

----

The NotificationBroadcaster was rewritten to use TMVars rather than MSampleVars,
to allow checking without blocking if a notification has been received.

----

This commit was sponsored by Tobias Brunner.
2014-01-06 16:03:39 -04:00
Joey Hess
fdcc04096c assistant: Ensure that .ssh/config and .ssh/authorized_keys are not group or world writable when writing to those files, as that can make ssh refuse to use them, if it allows another user to write to them. 2014-01-03 17:44:12 -04:00
Joey Hess
858eb26303 Avoid looping if long-running git cat-file or git hash-object crashes and keeps crashing when restarted. 2014-01-01 21:42:25 -04:00
Joey Hess
b6379bc101 fix 2013-12-30 14:05:07 -04:00
Joey Hess
0cb1a0a605 fix 2013-12-30 13:41:11 -04:00
Joey Hess
52a46585b9 fix build failure in windows code 2013-12-30 13:18:40 -04:00
Joey Hess
3b858ece45 typo 2013-12-29 22:40:23 -04:00
Joey Hess
d6d5b0a6d8 Android: Avoid trying to use Android's own ionice, which does not allow specifying a command to run. Fixes transferring files to/from android and probably a few other things. 2013-12-29 17:32:08 -04:00
Joey Hess
a96118ec55 work around windows insanity that is 97.61% more brain-damaged than normal 2013-12-29 16:25:12 -04:00
Joey Hess
7f6685c16e addurl, importfeed: Sanitize | and some other symbols and special characters. 2013-12-27 17:52:20 -04:00
Joey Hess
6c565ec905 external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!

The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)

Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo

Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:

* Need to lock the remote when using it to eg make a request, because
  in theory git-annex could have multiple threads that each try to use
  a remote at the same time. I don't think that git-annex ever does
  that currently, but better safe than sorry.

* Rather than starting up every external special remote program when
  git-annex starts, they are started only on demand, when first used.
  This will avoid slowdown, especially when running fast git-annex query
  commands. Once started, they keep running until git-annex stops, currently,
  which may not be ideal, but it's hard to know a better time to stop them.

* Bit of a chicken and egg problem with caching the cost of the remote,
  because setting annex-cost in the git config needs the remote to already
  be set up. Managed to finesse that.

This commit was sponsored by Lukas Anzinger.
2013-12-26 18:23:13 -04:00
Joey Hess
5897fb4a86 convert hacky shell linux mklibs code to haskell ; fixing symlink bug
The shell code was nasty, and buggy. New haskell code is much nicer,
and it's easy to do complicated calculations to properly convert possibly
absolute symlinks between libraries into relative links using it.
2013-12-24 13:13:17 -04:00
Joey Hess
3a6e0d1215 assistant: Set StrictHostKeyChecking yes when creating ssh remotes, and add it to the configuration for any ssh remotes previously created by the assistant. This avoids repeated prompts by ssh if the host key changes, instead syncing with such a remote will fail. Closes: #732602 2013-12-20 20:58:36 -04:00
Joey Hess
6e3cc0a228 need to stat symlink 2013-12-18 15:20:26 -04:00
Joey Hess
c99d6a8151 assistant: Fix OSX-specific bug that caused the startup scan to try to follow symlinks to other directories, and add their contents to the annex. 2013-12-18 15:05:29 -04:00
Joey Hess
a168b8de12 switch define used, not android specific 2013-12-18 02:06:15 -04:00
Joey Hess
2fd63f3cfa port transferkeys to windows; make stopping in progress transfers work too (probably)
transferkeys had used special FDs for communication, but that would be
quite annoying to do in Windows.

Instead, use stdin and stdout. But, to avoid commands like rsync stomping
on them and messing up the communications channel, they're duplicated to a
different handle; stdin is replaced with a null handle, and stdout is
replaced with a copy of stderr. This should all work in windows too.

Stopping in progress transfers may work on windows.. if the types unify
anyway. ;) May need some more porting.
2013-12-10 23:19:18 -04:00
Joey Hess
5b0cb642c9 warning 2013-12-10 01:35:27 -04:00
Joey Hess
89b5256f09 missing include 2013-12-10 01:21:39 -04:00
Joey Hess
063a183a61 windows disk free space checking 2013-12-10 01:18:04 -04:00
Joey Hess
0544818e59 squash warning 2013-12-09 17:24:32 -04:00
Joey Hess
472e0e97fe whups 2013-12-06 22:57:54 -04:00
Joey Hess
67df015767 didn't seem to help 2013-12-06 22:53:36 -04:00
Joey Hess
85936392ec fix 2013-12-06 22:38:01 -04:00
Joey Hess
b712e2cf57 try to use build-in yesod socket bind code on windows (mine seems to make t crash) 2013-12-06 22:35:11 -04:00
Joey Hess
00f3781f47 add withSocketsDo to hopefully appease the windows crash report gods 2013-12-06 22:18:58 -04:00
Joey Hess
2fc0330c3b weird DOS CMD shell is weird 2013-12-06 22:07:16 -04:00
Joey Hess
d5a50a0c8b Revert "try different syntax"
This reverts commit 891ca9a75d.
2013-12-06 21:48:35 -04:00
Joey Hess
891ca9a75d try different syntax 2013-12-06 21:37:06 -04:00
Joey Hess
7ac6090088 Revert "don't use writeFileProtected on windows"
This reverts commit c698d34b03.
2013-12-06 21:24:34 -04:00
Joey Hess
c698d34b03 don't use writeFileProtected on windows
This was preventing the webapp from working.
2013-12-06 21:11:17 -04:00
Joey Hess
4d5e0a90ed support for opening urls on windows 2013-12-06 18:18:05 -04:00
Joey Hess
1f2b42a853 oop 2013-12-06 13:59:22 -04:00
Joey Hess
ef1ea499ba oop 2013-12-06 13:56:06 -04:00
Joey Hess
5b5c33e06d pass COLLECT_GCC_OPTIONS 2013-12-06 13:53:58 -04:00
Joey Hess
ffd3e70046 same fix as android for windows getAddrInfo not working 2013-12-05 23:03:54 -04:00
Joey Hess
c077cee44a reorg 2013-12-04 23:09:54 -04:00
Joey Hess
abb0b3103f avoid blocked indefinitely on mvar on windows
by repeatedly reading lines and pausing since cannot do signal foo used on
unix
2013-12-04 17:25:41 -04:00
Joey Hess
0a63ed563f rsync special remote: Fix fallback mode for rsync remotes that use hashDirMixed. Closes: #731142 2013-12-02 12:53:39 -04:00
Joey Hess
0fd6078865 avoid repeatedly searching path to make batch command when running transferkeys 2013-12-01 15:37:51 -04:00
Joey Hess
7fc84717e8 avoid an unncessary use of the shell 2013-12-01 15:01:56 -04:00
Joey Hess
4882a611e5 assistant: Batch jobs are now run with ionice and nocache, when those commands are available. 2013-12-01 14:53:15 -04:00
Joey Hess
46ee2ea2e6 golf 2013-11-25 00:02:48 -04:00
Joey Hess
325f220058 typo 2013-11-24 23:51:25 -04:00
Joey Hess
e2f50f5110 Added support for quvi 0.9. Slightly suboptimal due to limitations in its interface compared with the old version. 2013-11-24 23:44:30 -04:00
Joey Hess
6d23786d96 Avoid misbehavior when addurl is used with quvi 0.9.
In 0.9, -v shows version, rather than controlling verbosity.

Still need to port to 0.9, this just avoids massively confusing addurl when
quvi prints its version and exits successfully, on urls that it cannot be
used with.
2013-11-22 14:12:44 -04:00
Joey Hess
a1f8621efc oops 2013-11-20 13:44:49 -04:00
Joey Hess
9c9a3c10ce add allowRead 2013-11-20 13:42:13 -04:00
Joey Hess
94251de91e add readFileStrictAnyEncoding 2013-11-20 13:41:13 -04:00
Joey Hess
b1ed98636b merge with git-repair 2013-11-19 17:08:57 -04:00
Joey Hess
310c549b5a Ensure execute bit is set on directories when core.sharedrepsitory is set. 2013-11-18 18:13:09 -04:00
Joey Hess
7ebdcc7535 use Win32-inotify
May mean the watcher works on Windows. Untested.
2013-11-12 17:29:29 -04:00
Joey Hess
b9b5e3370d build assistant and watcher on windows (doesn't work yet) 2013-11-12 17:22:34 -04:00
Joey Hess
750a3ca22f refactor 2013-11-12 02:54:19 -04:00