Commit graph

1129 commits

Author SHA1 Message Date
Joey Hess
c8fad345f2
add tryLockShared 2015-10-08 13:40:23 -04:00
Joey Hess
9461019e9a
open lock file ReadOnly when taking shared lock
It's only necessary to open a file for write when taking an exclusive lock.
2015-10-08 13:34:49 -04:00
Joey Hess
933fef6ae0 Merge branch 'winprocfix' 2015-10-04 15:46:25 -04:00
Joey Hess
06f1f03e7a Ported disk free space checking code to work on Solaris.
On Solaris, using f_bsize provided a value that is apparently much larger
than the real block size. The solaris docs for statvfs say
f_bsize is the "preferred" file system block size, and I guess the
filesystem prefers larger blocks, but uses smaller ones or something.
The docs also say that f_frsize is the "fundamental" block size.

Switched to using f_frsize on Linux and kFreeBSD too, since I guess
f_bsize could in theory vary the same way there too. Assuming that Solaris
is not violating the posix spec, I guess the linux man page for statvfs
is not as well written and I misunderstood it.
2015-10-02 16:31:15 -04:00
Joey Hess
cdbce512bd deal with more backward-compatible breaking renamings in conduit
This is the kind of annoying thing that makes me not want to use a library.
conduitManagerSettings was a perfectly fine name and could have been kept
forever.
2015-10-02 15:18:54 -04:00
Joey Hess
9e3ac97608 avoid deprecation warnings when built with http-client >= 0.4.18
Since I want git-annex to keep building on debian stable, I need to still
support the old http-client, which required explicit calls to
closeManager, or use of withManager to get Managers to close at appropriate
times. This is not needed in the new version, and so they added a
deprecation warning. IMHO much too early, because look at the mess I had to
go through to avoid that deprecation warning while supporting both
versions..
2015-10-01 13:48:56 -04:00
Joey Hess
69d37bd894 fix bug in back-compat ifdef 2015-09-23 13:09:08 -04:00
Joey Hess
017c00c581 redundant import 2015-09-22 12:31:54 -04:00
Joey Hess
1dcb86498e Improve ~/.ssh/config modification code to not add trailing spaces to lines it cannot parse.
"Host\n" is a valid line, and actually gets parsed ok, but this also holds
for other lines that it fails to parse for some reason.
2015-09-22 12:06:10 -04:00
Joey Hess
0ebde659bf assistant: When updating ~/.ssh/config, preserve any symlinks. 2015-09-21 12:39:13 -04:00
Joey Hess
f77a873a15 improve comment 2015-09-15 13:12:21 -04:00
Joey Hess
16947ef654 Fix bug in combination of preferred and required content settings. When one was set to the empty string and the other set to some expression, this bug caused all files to be wanted, instead of only files matching the expression.
Avoid: MAny `MOr` otherexpression
Which matches anything.
2015-09-15 12:50:14 -04:00
Joey Hess
ca33921bf2 I've been not documenting these import Preludes used to deal with the AMP transition 2015-09-15 11:32:47 -04:00
Simon Jakobi
b468890f3e Silence redundant import warning with base-4.8.* 2015-09-15 11:32:23 -04:00
Joey Hess
2d2e94798f merge hlint changes from propellor 2015-09-13 13:39:48 -04:00
Joey Hess
0390efae8c support gpg.program
When gpg.program is configured, it's used to get the command to run for
gpg. Useful on systems that have only a gpg2 command or want to use it
instead of the gpg command.
2015-09-09 18:06:49 -04:00
Joey Hess
19dbe2a611 webapp: Fix support for entering password when setting up a ssh remote. 2015-09-03 11:03:08 -07:00
Joey Hess
86e638567a Fix Windows build to work with ghc 7.10
It was failing at link time, some problem with terminatePID.
Re-implemented that to not use a C wrapper function, which cleared up the
problem. Removed old EvilLinker hack with must have been related to the
same problem.

Note that I have not tested this with older ghc's. In
f11f7520b5 I mention having tried this
approach before, and getting segfaults.. So, who knows. It seems to work
fine with ghc 7.10 at least.
2015-09-01 14:51:14 -07:00
Joey Hess
7be58b5e11 make sync --no-content be accepted
It's the default, but this is a step toward changing that default later..
2015-08-20 17:21:14 -04:00
Joey Hess
0f5d6c09ac importfeed --relaxed: Avoid hitting the urls of items in the feed. 2015-08-19 12:24:55 -04:00
Joey Hess
edd1ea54e4 package qualify imports
needed for "make fast" to work
2015-08-14 17:23:25 -04:00
Joey Hess
4665fc9e84 add debug logging of process exits
This is mostly to be able to see how long a command took to run. Also exit
code may be useful.

Unofrtunately, I can't put the command name in there, because it's not
available at this point, and it would be a much larger change to wrap the
ProcessHandle data type to add that. However, it's generally pretty obvious
which process exited from context.
2015-08-13 13:12:44 -04:00
Joey Hess
e953be11af avoid throwing exception when String is not encoded using the filesystem encoding
Since _encodeFilePath generates a String that doesn't use the filesystem
encoding, when this exception is caught, we know we already have such a
String, and can just return it as-is.
2015-08-12 10:57:48 -04:00
Joey Hess
4e4e11849a fix test suite fail in LANG=C
This was caused by 23e9d3bb77

an Arbitrary String is not necessarily encoded using the filesystem
encoding, and in a non-utf8 locale, encodeBS throws an exception on such a
string. All I could think to do is limit test data to ascii.

This shouldn't be a problem in practice, because the all Strings in
git-annex that are not generated by Arbitrary should be loaded in a way
that does apply the filesystem encoding.
2015-08-12 10:36:51 -04:00
Joey Hess
23e9d3bb77 Fix setting/setting/viewing metadata that contains unicode or other special characters, when in a non-unicode locale.
Oh boy, not again. So, another place that the filesystem encoding needs to
be applied. Yay.

In passing, I changed decodeBS so if a NUL is embedded in the input, the
resulting FilePath doesn't get truncated at that NUL. This was needed to
make prop_b64_roundtrips pass, and on reviewing the callers of decodeBS, I
didn't see any where this wouldn't make sense. When a FilePath is used to
operate on the filesystem, it'll get truncated at a NUL anyway, whereas if
a String is being used for something else, it might conceivably have a NUL
in it, and we wouldn't want it to get truncated when going through
decodeBS.
(NB: There may be a speed impact from this change.)
2015-08-11 18:40:59 -04:00
Joey Hess
0ec9bc2200 Added support for SHA3 hashed keys (in 8 varieties), when git-annex is built using the cryptonite library.
While cryptohash has SHA3 support, it has not been updated for the final
version of the spec. Note that cryptonite has not been ported to all arches
that cryptohash builds on yet.
2015-08-06 15:02:25 -04:00
Joey Hess
3cff287b26 proxy: Fix behavior when run in subdirectory of git repo.
This fixes a reversion introduced by relative path changes back last winter.

The root cause is simplifyPath "../foo" was incorrectly yielding "foo".

absPathFrom seems quite horrible. Probably most things that use it should
use </> instead.
2015-08-04 14:58:21 -04:00
Joey Hess
88e4fe6093 remove unused imports 2015-08-03 15:58:12 -04:00
Joey Hess
ea765ec022 windows build warning fixes 2015-08-03 15:54:29 -04:00
Joey Hess
6ca08f02a4 remove unused imports 2015-08-03 15:49:35 -04:00
Joey Hess
bf3e9945fc fix build warning when building with yesod 1.2 and newer yesod-core 2015-08-03 15:42:44 -04:00
Joey Hess
631557aa60 Revert "fix build warning when building with yesod 1.2"
This reverts commit 160b0ac824.
2015-08-03 15:40:04 -04:00
Joey Hess
160b0ac824 fix build warning when building with yesod 1.2 2015-08-03 13:13:57 -04:00
Joey Hess
8a547e82b1 addidential debugging 2015-08-03 11:27:53 -04:00
Joey Hess
d986d24494 analysis; forwarded 2015-08-03 11:27:27 -04:00
Joey Hess
730cc3feb5 wire tasty's option parser into the main program option parser
This makes bash completion work for git-annex test, and is
generally cleaner.
2015-07-13 13:20:10 -04:00
Joey Hess
b59b8be737 generalize parseDuration so it can be used in the ReadM monad 2015-07-08 16:08:26 -04:00
Joey Hess
4018e5f6f1 better method for running tasty's optparse as a subcommand 2015-07-08 00:39:19 -04:00
Joey Hess
625303226d import: Fix failure of cross-device import on Windows.
As well as import, 2 other places ran "mv" manually, so changed them to use
moveFile as well.
2015-07-07 14:48:23 -04:00
Joey Hess
78ef8912f8 avoid "Defined but not used" warning on android 2015-07-02 15:24:33 -04:00
Joey Hess
adba0595bd use bloom filter in second pass of sync --all --content
This is needed because when preferred content matches on files,
the second pass would otherwise want to drop all keys. Using a bloom filter
avoids this, and in the case of a false positive, a key will be left
undropped that preferred content would allow dropping. Chances of that
happening are a mere 1 in 1 million.
2015-06-16 18:50:13 -04:00
Joey Hess
a0a8127956 instance Hashable Key for bloomfilter 2015-06-16 18:37:41 -04:00
Joey Hess
a6c56fb459 improve url parsing more
Now can handle eg, "http://[::1]/download/cdrom-fontzip[foo]", where
the first [] need to stay unescaped, but the rest have to be escaped.
2015-06-14 13:54:24 -04:00
Joey Hess
829007d629 Improve url parsing to handle some urls containing illegal [] characters in their paths.
Ie, "https://archive.org/download/zoom-2/Zoom - Release 2 (1996)(Active Software)[!].iso"
2015-06-14 13:39:44 -04:00
Joey Hess
256b86b948 oh foo, I didn't mean to include this in the prev commit 2015-06-11 16:43:59 -04:00
Joey Hess
5c960601aa 4 ns optimisation of repeated calls to hasDifference on the same Differences
I want this as fast as possible, so it can be added to code paths without
slowing them down.

Avoid the set lookup, and rely on laziness,
drops runtime from 14.37 ns to 11.03 ns according to this criterion benchmark:

import Criterion.Main
import qualified Types.Difference as New
import qualified Types.DifferenceOld as Old

main :: IO ()
main = defaultMain
	[ bgroup "hasDifference"
		[ bench "new" $ whnf (New.hasDifference New.OneLevelObjectHash) new
		, bench "old" $ whnf (Old.hasDifference Old.OneLevelObjectHash) old
		]
	]
  where
	s = "fromList [ObjectHashLower, OneLevelObjectHash, OneLevelBranchHash]"
	new = New.readDifferences s
	old = Old.readDifferences s

A little bit of added boilerplate, but I suppose it's worth it to not
need to worry about set lookup overhead. Note that adding more differences
would slow down the old implementation; the new implementation will run
the same speed.
2015-06-11 16:34:35 -04:00
Joey Hess
b8f044660f two more breakages introduced when removing the Params constructor 2015-06-03 13:02:33 -04:00
Joey Hess
6688f52577 fix bug introduced in recent Params removal 2015-06-02 16:28:05 -04:00
Joey Hess
8be78783b0 Revert "When listing DBus services, also list activatable services."
This reverts commit ef0e3ac22e.

Sebastian thinks best to revert this:

It seems to me the reason I needed to look at activatable sockets
might actually be a networkd bug, and I was in error about patch 0001.
On my machines (without DHCP), networkd quits after configuring the
links. I thought this had to do with network activation, but that was
probably mistaken. This was obscured by my testing the change by doing
systemctl stop/start on networkd; now that I actually unplugged the
network cable, I noticed no DBus messages are triggered by this on
this machine. Your test case might have had a similar problem
(networkd quitting on idle). Might be related to [1].

On another machine (with DHCP) networkd remains active all the time,
and patch 0002 works there. You might want to revert 0001, though:
Suppose someone’s running no manager at all, so that polling would be
required. Because networkd is still listed as activable, we would
refrain from polling – by mistake, because networkd doesn’t seem to
actually go active if we listen on its bus, and it’s listed as
activable even when it’s not configured. Connectivity-related messages
will come in when stopping/starting the service, but not when
unplugging the cable.
2015-06-02 14:38:24 -04:00
Sebastian Reuße
ef0e3ac22e When listing DBus services, also list activatable services. 2015-06-02 12:51:13 -04:00
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
ade6ed2d71 AMP hack 2015-05-31 16:54:07 -04:00
Joey Hess
bf17af16b7 merge haddock formatting from propellor 2015-05-30 11:27:40 -04:00
Joey Hess
fa38fe0cf6 qualify imports to avoid conflict with regex-tdfa-rc 2015-05-30 02:03:09 -04:00
Joey Hess
8990d4cc68 fsck: When checksumming a file fails due to a hardware fault, the file is now moved to the bad directory, and the fsck proceeds. Before, the fsck immediately failed. 2015-05-27 16:40:03 -04:00
Joey Hess
ff629a1ec0 implment catchHardwareFault 2015-05-27 16:36:54 -04:00
Joey Hess
7cb16b9efb improve error message when checksum command exits nonzero 2015-05-27 15:43:50 -04:00
Joey Hess
ccf307307d merge haddock documentation from propellor 2015-05-27 12:39:04 -04:00
Joey Hess
6c3cea7699 need more polymorphism 2015-05-22 13:50:37 -04:00
Joey Hess
696ed9b3f9 fix windows build 2015-05-22 13:44:24 -04:00
Joey Hess
9de5cd2966 fix crash in stale transfer lockfile cleanup code
Need to differentiate between the lockfile not being locked, and it not
existing.
2015-05-19 23:35:24 -04:00
Joey Hess
846e7db245 unused imports 2015-05-19 15:05:30 -04:00
Joey Hess
b47c9fd587 honor core.sharedRepository settings in lockContent
The content file may not be owned by the user running git-annex, in which
case, setting the owner write bit was not enough to let lockContent
act on the file. However, with some core.sharedRepository configs, the file
should be writable by the user's group. So, the thing to do is to call
thawContent on it.
2015-05-19 14:53:19 -04:00
Joey Hess
1312e721ed convert lockContent to use new LockPools
Also cleaned up the code, avoiding creating a lock file if we're going to
open it for create later anyway.

And, if there's an exception while preparing to lock the file, but not at
the point of actually taking the lock, throw an exception, instead of
silently not locking and pretending to succeed.

And, on Windows, always use lock file, even if the repo somehow got into
indirect mode (maybe with cygwin git..)
2015-05-19 14:12:23 -04:00
Joey Hess
ecb0d5c087 use lock pools throughout git-annex
The one exception is in Utility.Daemon. As long as a process only
daemonizes once, which seems reasonable, and as long as it avoids calling
checkDaemon once it's already running as a daemon, the fcntl locking
gotchas won't be a problem there.

Annex.LockFile has it's own separate lock pool layer, which has been
renamed to LockCache. This is a persistent cache of locks that persist
until closed.

This is not quite done; lockContent stil needs to be converted.
2015-05-19 14:09:52 -04:00
Joey Hess
6915b71c57 lock pools to work around non-concurrency/composition safety of POSIX fcntl 2015-05-18 15:57:17 -04:00
Joey Hess
e9172263e5 comment typos 2015-05-17 14:22:14 -04:00
Joey Hess
8c2dd7d8ee Fix an unlikely race that could result in two transfers of the same key running at once.
As discussed in bug report.
2015-05-12 19:39:28 -04:00
Joey Hess
03667a162a couple of AMP warnings I missed before 2015-05-10 16:51:03 -04:00
Joey Hess
5c7cdbae46 more {-# OPTIONS_GHC -fno-warn-tabs #-} ... Forcing people who have what is merely a difference of opinion to you to do this is a bit of an asshole move. Just saying. 2015-05-10 16:38:49 -04:00
Joey Hess
ed4fe02896 disable horrible tab warning, needed in every file that Setup.hs pulls in
This is certianly a cabal bug for not passing the build options in the
cabal file when building Setup.hs.

And, why oh why did ghc enable this warning by default? So unhappy with
this choice.
2015-05-10 16:31:50 -04:00
Joey Hess
ec267aa1ea rejigger imports for clean build with ghc 7.10's AMP changes
The explict import Prelude after import Control.Applicative is a trick
to avoid a warning.
2015-05-10 16:20:30 -04:00
Joey Hess
6466dbc950 FlexibleContexts needed by ghc 7.10 2015-05-10 15:37:55 -04:00
Joey Hess
fcb8394e5f switch to Control.Monad.ST to avoid build warning in ghc 7.10 2015-05-10 14:42:16 -04:00
Joey Hess
f395ca7a41 Improve quvi 0.4 output parsing to handle cases wher there is no known filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used. 2015-05-08 13:39:00 -04:00
Joey Hess
d9bfc78444 avoid using relative path from temp dir to dest file
That failed on OSX. The temp dir was
/var/folders/fb/pnwjj52n7fg0r9mnvpsfll180000gr/T/downloadurl
and the relative path
../../../../../../Volumes/Visitors/joeyh/git-annex/r/.git/...
Didn't work. I have no clue why, how did OSX manage to break this?

But, the relative path is longer most of the time anyway, so let's
just use the absolute path.
2015-05-07 18:47:24 -04:00
Magnus Therning
8ab56a5cf2 Replace dataenc with sandi.
The library dataenc is unmaintained and has been superseded by sandi.

Signed-off-by: Magnus Therning <magnus@therning.org>
2015-05-07 18:07:05 -04:00
Joey Hess
cf786f42a4 Support checking ftp urls for file presence. 2015-05-05 14:05:02 -04:00
Joey Hess
0b18228516 Work around wget bug #784348 which could cause it to clobber git-annex symlinks when downloading from ftp. 2015-05-05 13:53:06 -04:00
Joey Hess
3d8cf16eec add more generic interface 2015-05-05 13:51:31 -04:00
Joey Hess
38403092ae revert reversion merged from propellor 2015-04-29 14:15:08 -04:00
Joey Hess
d8ffa498ef avoid dupicate bracket import errror
Ambiguous occurrence `bracket'
    It could refer to either `Control.Exception.bracket',
                             imported from `Control.Exception' at Utility/FileMode.hs:14:27-33
                             (and originally defined in `Control.Exception.Base')
                          or `Utility.Exception.bracket',
                             imported from `Utility.Exception' at Utility/FileMode.hs:22:1-24
                             (and originally defined in `Control.Monad.Catch')
2015-04-29 13:43:11 -04:00
Joey Hess
d40dea58e3 merge generalization of writeFileProtected from propellor 2015-04-28 14:58:29 -04:00
Joey Hess
eb8ef44133 Dropped support for older versions of yesod and warp than the ones in Debian Jessie.
466 lines of compat cruft deleted!
2015-04-22 16:19:11 -04:00
Joey Hess
5f73bc3a12 Removed dependency on haskell SHA library, instead using cryptohash >= 0.11.0. 2015-04-19 11:05:32 -04:00
Joey Hess
be3895be45 refactor 2015-04-19 10:57:14 -04:00
Joey Hess
addc82dab7 removed all uses of undefined from code base
It's a code smell, can lead to hard to diagnose error messages.
2015-04-19 00:38:29 -04:00
Joey Hess
8489057e8d fsck --from remote: When bad content is found in the remote, and the local repo does not have a copy of the content, preserve the bad content in .git/annex/bad/ to avoid further data loss. 2015-04-18 14:13:07 -04:00
Joey Hess
cd31b69ff6 don't test with null paths 2015-04-14 15:15:29 -04:00
Joey Hess
f84ccaa4e8 fix relPathDirToFileAbs on windows with different drive letters
Since we started using this for git repos, when a remote was on another
drive, it resulted in a bogus relative path to it being used by git-annex,
which didn't work.
2015-04-14 14:16:44 -04:00
Joey Hess
2be4834822 info: Added --bytes option. 2015-04-12 14:08:40 -04:00
Joey Hess
c65e71e6a5 cleanup 2015-04-09 12:57:30 -04:00
Joey Hess
42281f12d6 bring back --quiet filtering of stdout and stderr, with deadlock fixed
I don't quite understand the cause of the deadlock. It only occurred
when git-annex-shell transferinfo was being spawned over ssh to feed
download transfer progress back. And if I removed this line from
feedprogressback, the deadlock didn't occur:
	bytes <- readSV v

The problem was not a leaked FD, as far as I could see. So what was it?
I don't know.

Anyway, this is a nice clean implementation, that avoids the deadlock.
Just fork off the async threads to handle filtering the stdout and stderr,
and let them clean up their handles whenever they decide to exit.

I've verified that the handles do get promptly closed, although a little
later than I would expect. Presumably that "little later" is what
was making waiting on the threads deadlock.

Despite the late exit, the last line of stdout and stderr appears where
I'd want it to, so I guess this is ok..
2015-04-06 20:20:52 -04:00
Joey Hess
0a89d55269 Fixes a bug in the last release that caused rsync and possibly other commands to hang at the end of a file transfer.
Stderr reader blocks waiting for all stderr, and so blocks the process ever
exiting.

I tried several ways to get around this, but no success yet. For now,
disable the stderr reader entirely.
2015-04-06 17:12:38 -04:00
Joey Hess
30aa902174 relay external special remote stderr through progress suppression machinery (eep!)
It sounds worse than it is. ;)

Some external special remotes may run commands that display progress on
stderr. If git-annex is run with --quiet, this should filter out such
displays while letting the errors through.
2015-04-04 14:54:03 -04:00
Joey Hess
2343f99c85 well along the way to fully quiet --quiet
Came up with a generic way to filter out progress messages while keeping
errors, for commands that use stderr for both.

--json mode will disable command outputs too.
2015-04-04 14:34:03 -04:00
Joey Hess
b2ad3403c6 make downloadQuiet quiet again
This was broken in commit c64ede23cd
2015-04-03 20:38:20 -04:00
Joey Hess
45e97191c3 fix stderr handling 2015-04-03 19:48:56 -04:00
Joey Hess
20fb91a7ad WIP on making --quiet silence progress, and infra for concurrent progress bars 2015-04-03 16:48:30 -04:00
Joey Hess
1c91024978 rename bothHandles -> ioHandles 2015-04-03 15:35:18 -04:00
Joey Hess
294991dacb Significantly sped up processing of large numbers of directories passed to a single git-annex command. (try 2)
New approach is to do it the expensive way for the first 100 paths
on the command line, but then assume the user doesn't care about order too
much and fall back to the cheap way that does not preserve order.
2015-04-02 01:44:32 -04:00
Joey Hess
f79502d377 Revert "Significantly sped up processing of large numbers of directories passed to a single git-annex command."
This reverts commit 705112903e.

Whoops, git ls-files does not always output in the input ordering.
That's why all this work is needed. Urk.
2015-04-02 01:24:37 -04:00
Joey Hess
bf563d5df0 comment 2015-04-02 00:34:06 -04:00
Joey Hess
ee4af54005 minor opt 2015-04-02 00:33:08 -04:00
Joey Hess
7b32e7acb5 make segmentXargs preserve order 2015-04-02 00:31:36 -04:00
Joey Hess
705112903e Significantly sped up processing of large numbers of directories passed to a single git-annex command. 2015-04-02 00:19:49 -04:00
Joey Hess
6dd13d2684 micro-opt 2015-04-01 23:14:59 -04:00
Joey Hess
b8f0b7309f Work around curl bug when asked to download an empty url to a file.
In this situation, curl -o exits successfully without creating the output
file.

There was already a workaround for curl file:/// but I did not realize this
also affected regular url downloads.

To fix it, pre-create the destination file before starting curl.
Since we cannot always know the size of an url before trying to download
it, let's always do this.

Note that since curl is told -C -, we have to consider if this
makes curl try to do a ranged download, which might fail on some servers
where a regular download would have succeeded. My testing indicates
this isn't a problem; since the file is empty, curl seems to not try to
do a ranged download.

Original report: https://github.com/datalad/datalad/issues/79
Curl bug report: https://github.com/bagder/curl/issues/183
2015-03-27 10:22:36 -04:00
Joey Hess
7cb2f91f5b update comment for segmentXargs 2015-03-26 13:48:12 -04:00
Joey Hess
9b93278e8a metadata: Fix encoding problem that led to mojibake when storing metadata strings that contained both unicode characters and a space (or '!') character.
The fix is to stop using w82s, which does not properly reconstitute unicode
strings. Instrad, use utf8 bytestring to get the [Word8] to base64. This
passes unicode through perfectly, including any invalid filesystem encoded
characters.

Note that toB64 / fromB64 are also used for creds and cipher
embedding. It would be unfortunate if this change broke those uses.

For cipher embedding, note that ciphers can contain arbitrary bytes (should
really be using ByteString.Char8 there). Testing indicated it's not safe to
use the new fromB64 there; I think that characters were incorrectly
combined.

For credpair embedding, the username or password could contain unicode.
Before, that unicode would fail to round-trip through the b64.
So, I guess this is not going to break any embedded creds that worked
before.

This bug may have affected some creds before, and if so,
this change will not fix old ones, but should fix new ones at least.
2015-03-04 12:54:30 -04:00
Joey Hess
6726ce8116 fix intentional typo left in after debugging session, broke quvi support (last release is ok) 2015-02-19 14:31:04 -04:00
Joey Hess
a787cead35 bittorrent: Fix mojibake introduced in parsing arai2c progress output.
hGetSomeString reads one byte at a time, so unicode bytes are not composed.
The problem comes when outputting that to the console with hPut; that
tried to apply the handle's encoding, and so we get mojibake.

Instead, use ByteStrings, and only convert it to a string for parsing, not
for display.

Note that there are a couple of other things that use hGetSomeString,
which I've left as-is for now.
2015-02-10 12:34:34 -04:00
Joey Hess
d3b61d69b1 addurl: Avoid crash if quvi is not installed, when git-annex was built with process-1.2
createProcess has been changed to throw an exception if the program is
not in path.
2015-02-10 11:59:59 -04:00
Joey Hess
3e78b83875 Windows: Fix bug in dropping an annexed file, which caused a symlink to be staged that contained backslashes. 2015-02-09 15:37:26 -04:00
Joey Hess
8eb01bc894 Added MD5 and MD5E backends. 2015-02-04 13:47:54 -04:00
Joey Hess
95c1593098 Remove support for building without cryptohash.
This will prevent backporting to wheezy, but it's time to simplify the
code.
2015-02-04 13:41:26 -04:00
Joey Hess
e8c376e0ad import Data.Default in Common 2015-01-28 16:11:28 -04:00
Joey Hess
587f6a919b addurl: When a Content-Disposition header suggests a filename to use, addurl will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this) 2015-01-22 14:52:52 -04:00
Joey Hess
91f1b2bdcf excess indent 2015-01-22 13:47:06 -04:00
Joey Hess
d29bf2f5b2 merge change from propellor
exporting Unit allows custom data units
2015-01-21 23:13:50 -04:00
Joey Hess
09a66f702d Revert "remove absNormPathUnix, using my absPathFrom replacement"
This reverts commit a7f05c007b.

Consider: relPathDirToFile (absPathFrom "/tmp/repo/xxx" "y/bar") "/tmp/repo/.git/annex/objects/xxx"

This needs to always yield "../../../.git/annex/objects/xxx" but on
Windows, it is "..\\..\\/tmp/repo/.git/annex/objects/xxx"
2015-01-21 13:54:47 -04:00
Joey Hess
a7f05c007b remove absNormPathUnix, using my absPathFrom replacement 2015-01-21 13:37:09 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
068aaf943b on second thought, InodeCache should use getFileSize
This is necessary for interop between inode caches created on unix and
windows. Which is more important than supporting inodecaches for large keys
with the wrong size, which are broken anyway.

There should be no slowdown from this change, except on Windows.
2015-01-20 19:35:50 -04:00
Joey Hess
7a14edce5a build fix 2015-01-20 17:27:42 -04:00
Joey Hess
27c24b7407 fix upFrom of "." (and upFrom of "/" on Windows) 2015-01-20 17:23:23 -04:00
Joey Hess
4f657aa14e add getFileSize, which can get the real size of a large file on Windows
Avoid using fileSize which maxes out at just 2 gb on Windows.
Instead, use hFileSize, which doesn't have a bounded size.
Fixes support for files > 2 gb on Windows.

Note that the InodeCache code only needs to compare a file size,
so it doesn't matter it the file size wraps. So it has been
left as-is. This was necessary both to avoid invalidating existing inode
caches, and because the code passed FileStatus around and would have become
more expensive if it called getFileSize.

This commit was sponsored by Christian Dietrich.
2015-01-20 17:09:24 -04:00
Joey Hess
f4de021a54 convert parentDir to be based on takeDirectory, but fixed for trailing / 2015-01-09 14:26:52 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
c70bbdfe9d fix build failure 2015-01-09 11:58:38 -04:00
Joey Hess
858d776352 Merge branch 'master' into relativepaths
Conflicts:
	Locations.hs
	debian/changelog
2015-01-06 19:00:01 -04:00
Joey Hess
9d75709bf7 build fix 2015-01-06 18:59:13 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
cd865c3b8f Switch to using relative paths to the git repository.
This allows the git repository to be moved while git-annex is running in
it, with fewer problems.

On Windows, this avoids some of the problems with the absurdly small
MAX_PATH of 260 bytes. In particular, git-annex repositories should
work in deeper/longer directory structures than before. See
http://git-annex.branchable.com/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/

There are several possible ways this change could break git-annex:

1. If it changes its working directory while it's running, that would
   be Bad News. Good news everyone! git-annex never does so. It would also
   break thread safety, so all such things were stomped out long ago.

2. parentDir "." -> "" which is not a valid path. I had to fix one
   instace of this, and I should probably wipe all calls to parentDir out
   of the git-annex code base; it was never a good idea.

3. Things like relPathDirToFile require absolute input paths,
   and code assumes that the git repo path is absolute and passes it to it
   as-is. In the case of relPathDirToFile, I converted it to not make
   this assumption.

Currently, the test suite has 16 failures.
2015-01-06 16:19:41 -04:00
Joey Hess
5db31e2386 Fix build with process 1.2.1.0. 2015-01-04 13:07:57 -04:00
Joey Hess
e013183280 OSX: Switched away from deprecated statfs64 interface.
Getting rid of build warning

warning: 'statfs64' is deprecated: first deprecated in OS X 10.6
[-Wdeprecated-declarations]

10.6 is much older than the oldest git-annex OSX port, so won't break
anything.
2014-12-31 12:20:27 -04:00
Joey Hess
23051899e6 Windows: Got the rsync special remote working.
More aggressive rsync params fixup for windows. Param may contain a url, or
a file path, so check if it looks like a local file path and if so, fix it
up.

On windows only, rsyncUrlIsPath will treat c:foo as a path, rather than as
a rsyncurl starting with a host "c".
2014-12-30 15:05:59 -04:00
Joey Hess
5b30ba63cd Windows: Fix local rsync filepath munging (fixes 26 test suite failures). 2014-12-30 13:57:49 -04:00
Joey Hess
cdc69d7477 final warning fix 2014-12-29 17:45:36 -04:00
Joey Hess
ef82f0bcef fix 2014-12-29 17:44:58 -04:00
Joey Hess
7c511f900d fix more warnings on BSD/OSX 2014-12-29 17:43:26 -04:00
Joey Hess
66cdc4d612 one more warning 2014-12-29 17:35:19 -04:00
Joey Hess
c9a3e80d32 fixed all remaining build warnings on Windows 2014-12-29 17:30:20 -04:00
Joey Hess
67828b1c86 fix a OSX C compiler warning about null pointer 2014-12-29 17:07:42 -04:00
Joey Hess
415b2c53ed fix OSX build warning 2014-12-29 16:38:52 -04:00
Joey Hess
02f150b33d Work around statfs() overflow on some XFS systems.
statfs(".", 0xffa8ad50)                 = -1 EOVERFLOW (Value too large for defined data type)

Ref <20141222221621.GO7251@onerussian.com>
2014-12-23 11:52:23 -04:00
Joey Hess
1c88b59bd0 refactor 2014-12-17 13:21:55 -04:00
Joey Hess
c64ede23cd Use wget -q --show-progress for less verbose wget output, when built with wget 1.16. 2014-12-16 14:04:40 -04:00
Joey Hess
67c05daf5e sanitize filepaths provided by checkUrl 2014-12-11 20:08:49 -04:00
Joey Hess
0da402e0b9 more general readProcess' merged from propellor 2014-11-19 00:59:45 -04:00
Joey Hess
63fa5c0782 Work around apparent bug in lsof 4.88's -F output format. 2014-11-12 18:00:40 -04:00
Joey Hess
c3390f4c98 generalize monads 2014-11-12 14:59:24 -04:00
Joey Hess
5d946fe3a9 switch from hGetSome to hGet
This should be essentially no-op change for hGetContentsMetered, since it
always gets the entire contents. So the only difference is that each chunk
of the lazy bytestring will always be the full chunk size. So, I'm pretty
sure this is safe. Also, the only current users of hGetContentsMetered are
reading files, so the stream won't block for long in the middle.

The improvement is that hGetUntilMetered will always get some multiple of
the defaultChunkSize. This will allow the S3 multipart code to pick a fixed
size and know that hGetUntilMetered will really get that size.

(cherry picked from commit bd09046291)
2014-11-03 22:11:47 -04:00
Joey Hess
0602b26314 hGetUntilMetered 2014-11-03 18:37:05 -04:00
Joey Hess
0f6aaf8012 Windows: Fix crash when user.name is not set in git config. 2014-10-31 16:14:12 -04:00
Joey Hess
86b326782a getUserEntryForID does not work on windows
Throw an error instead, mentioning the relevant env vars.
2014-10-31 15:46:44 -04:00
Joey Hess
613f396efd more accurate version bounds 2014-10-24 12:11:27 -04:00
Joey Hess
ce9076370e deal with yesod's second gratuitous rename of the same function in a year 2014-10-23 20:26:46 -04:00
Joey Hess
431c1eafe5 typo 2014-10-16 15:39:19 -04:00
Joey Hess
b6be403d3a typos 2014-10-16 15:37:12 -04:00
Joey Hess
1e59df083d Use haskell setenv library to clean up several ugly workarounds for inability to manipulate the environment on windows.
Didn't know that this library existed!

This includes making git-annex not re-exec itself on start on windows, and
making the test suite on Windows run tests without forking.
2014-10-15 20:33:52 -04:00
Joey Hess
9c8abb761f revert reversion
This reverts commit 7bfc4a5442.

Android build is going to have consistent versions again.
2014-10-15 11:49:13 -04:00
Joey Hess
7bfc4a5442 unbreak android build
This reverts commit dd667844b6
and commit e6eff0e951.

Those commits were fine, except the android autobuilder currently has a bit
of a mess of yesod versions and broke. Better to wait on this.
2014-10-14 14:38:38 -04:00
Joey Hess
e6eff0e951 more accurate bounds on yesod boostrap3 ifdefs 2014-10-13 15:33:51 -04:00
Joey Hess
dd667844b6 avoid using Assistant.WebApp.Bootstrap3 when building with current yesod
Only use that when building with ancient yesod, which does not include it.

This also let me remove ifdefs in the file to support building with the new
version of yesod.
2014-10-09 15:19:24 -04:00
Joey Hess
9fd95d9025 indent with tabs not spaces
Found these with:
git grep "^  " $(find -type  f -name \*.hs) |grep -v ':  where'

Unfortunately there is some inline hamlet that cannot use tabs for
indentation.

Also, Assistant/WebApp/Bootstrap3.hs is a copy of a module and so I'm
leaving it as-is.
2014-10-09 15:09:26 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
2abeae7582 Windows: Avoid crashing trying to list gpg secret keys, for gcrypt which is not yet supported on Windows. 2014-09-16 13:40:44 -04:00
Joey Hess
7482166180 Rather than crashing when there's a problem with the requested bloomfilter capacity/accuracy, fall back to a reasonable default bloom filter size. 2014-09-12 12:26:12 -04:00
Joey Hess
6eb5c3f479 Do not preserve permissions and acls when copying files from one local git repository to another. Timestamps are still preserved as long as cp --preserve=timestamps is supported.
This avoids cp -a overriding the default mode acls that the user might have
set in a git repository.

With GNU cp, this behavior change should not be a breaking change, because
git-anex also uses rsync sometimes in the same situation, and has only ever
preserved timestamps when using rsync.

Systems without GNU cp will no longer use cp -a, but instead just cp.
So, timestamps will no longer be preserved. Preserving timestamps when
copying between repos is not guaranteed anyway.

Closes: #729757
2014-08-26 17:10:25 -07:00
Joey Hess
2b234634f6 fix imports for windows 2014-08-23 16:27:24 -07:00
Joey Hess
59eae904b1 final scary locking refactoring (for now)
Note that while before checkTransfer this called getLock with WriteLock,
getLockStatus's use of ReadLock will also notice any exclusive locks.
Since transfer info files are only locked exclusively, never shared,
there is no behavior change.

Also, fixes checkLocked to actually return Just False when the file
exists, but is not locked.
2014-08-20 19:30:40 -04:00
Joey Hess
1994771215 more lock file refactoring
Also fixes a test suite failures introduced in recent commits, where
inAnnexSafe failed in indirect mode, since it tried to open the lock file
ReadWrite. This is why the new checkLocked opens it ReadOnly.

This commit was sponsored by Chad Horohoe.
2014-08-20 18:58:14 -04:00
Joey Hess
ec7dd0446a more lock file refactoring 2014-08-20 17:03:04 -04:00
Joey Hess
d279180266 reorganize and refactor lock code
Added a convenience Utility.LockFile that is not a windows/posix
portability shim, but still manages to cut down on the boilerplate around
locking.

This commit was sponsored by Johan Herland.
2014-08-20 16:45:58 -04:00
Joey Hess
092041fab0 Ensure that all lock fds are close-on-exec, fixing various problems with them being inherited by child processes such as git commands.
(With the exception of daemon pid locking.)

This fixes at part of #758630. I reproduced the assistant locking eg, a
removable drive's annex journal lock file and forking a long-running
git-cat-file process that inherited that lock.

This did not affect Windows.

Considered doing a portable Utility.LockFile layer, but git-annex uses
posix locks in several special ways that have no direct Windows equivilant,
and it seems like it would mostly be a complication.

This commit was sponsored by Protonet.
2014-08-20 11:37:02 -04:00
Joey Hess
8b15af309a add compat cruft for old versions of http-types and http-conduit 2014-08-17 15:39:46 -04:00
Joey Hess
6ab0737a75 work around default Accept-Encoding in http-client 2014-08-15 18:02:17 -04:00
Joey Hess
e0227dfedf memoize construction of the Request -> Request function to apply the UrlOptions 2014-08-15 17:47:21 -04:00
Joey Hess
dd619c7166 Switched from the old haskell HTTP library to http-conduit.
The hoary old HTTP library was only used when checking if an url exists,
when curl was not available. It had many problems, including not supporting
https at all.

Now, this is done using http-conduit for all urls that it supports. Falls
back to curl for any url that http-conduit doesn't like (probably ftp etc,
but could also be an url that its parser chokes on for whatever reason).

This adds a new dependency on http-conduit, but webdav support already
indirectly depended on that, and the s3-aws branch also uses it.

This opens up the possibility of using http-conduit for large file
downloads, but for now I've left it using wget/curl.

This commit was sponsored by Paul Tötterman.
2014-08-15 17:37:42 -04:00
Joey Hess
9193d6e6bd fix build on windows with unix-compat-0.4.1.3
FileID type changed, needs Arbitrary instance.

On the plus side, getFileStatus on Windows now actually gets file id's,
not always 0, so direct mode is safer there now.
2014-08-11 17:25:31 -04:00
Joey Hess
fb66ee8c85 qualify catch and try
got a build failure on android due to there being a Prelude.catch
2014-08-10 19:40:21 -04:00
Joey Hess
c27d9f909b fix windows build 2014-08-10 15:30:55 -04:00
Joey Hess
c784ef4586 unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.

Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.

Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.

However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.

Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-07 22:03:29 -04:00
Joey Hess
8e3d62dd5d generalized using the extensions package 2014-08-07 21:01:07 -04:00
Joey Hess
6c450aad1d move ugly rsync zombie workaround
This reaping of any processes came to cause me problems when redoing the
rsync special remote -- a gpg process that was running gets waited on and
the place that then checks its return code fails.

I cannot reproduce any zombies when using the rsync special remote.
But I still can when using a normal git remote, accessed over ssh.
There is 1 zombie per file downloaded without this horrible hack enabled.

So, move the hack to only be used in that case.
2014-08-03 16:53:29 -04:00
Joey Hess
9720ee9e56 testremote: New command to test uploads/downloads to a remote.
This only performs some basic tests so far; no testing of chunking or
resuming. Also, the existing encryption type of the remote is used; it
would be good later to derive an encrypted and a non-encrypted version of
the remote and test them both.

This commit was sponsored by Joseph Liu.
2014-08-01 15:10:01 -04:00
Joey Hess
1d263e1e7e lift types from IO to Annex
Some remotes like External need to run store and retrieve actions in Annex,
not IO. In order to do that lift, I had to dive pretty deep into the
utilities, making Utility.Gpg and Utility.Tmp be partly converted to using
MonadIO, and Control.Monad.Catch for exception handling.

There should be no behavior changes in this commit.

This commit was sponsored by Michael Barabanov.
2014-07-29 16:28:44 -04:00
Joey Hess
9d4a766cd7 resume interrupted chunked downloads
Leverage the new chunked remotes to automatically resume downloads.
Sort of like rsync, although of course not as efficient since this
needs to start at a chunk boundry.

But, unlike rsync, this method will work for S3, WebDAV, external
special remotes, etc, etc. Only directory special remotes so far,
but many more soon!

This implementation will also properly handle starting a download
from one remote, interrupting, and resuming from another one, and so on.

(Resuming interrupted chunked uploads is similarly doable, although
slightly more expensive.)

This commit was sponsored by Thomas Djärv.
2014-07-27 18:56:32 -04:00
Joey Hess
ab4cce4114 core implementation of new style chunking
Not yet used by any special remotes, but should not be too hard to add it
to most of them.

storeChunks is the hairy bit! It's loosely based on
Remote.Directory.storeLegacyChunked. The object is read in using a lazy
bytestring, which is streamed though, creating chunks as needed, without
ever buffering more than 1 chunk in memory.

Getting the progress meter update to work right was also fun, since
progress meter values are absolute. Finessed by constructing an offset
meter.

This commit was sponsored by Richard Collins.
2014-07-25 16:20:32 -04:00
Joey Hess
cf384cddfb increase sleep 2014-07-16 16:16:25 -04:00
Joey Hess
aacd5c6994 Windows: fix crash after 10 minutes
getLine in waitForTermination doesn't work when stdin is closed..
Just loop forever, there was no reason to getLine here I think.
2014-07-16 15:42:12 -04:00
Joey Hess
339a3b8ab0 Windows: Fix locking issue that prevented the webapp starting (since 5.20140707).
Reversion introduced in 9fb8038166.
The locking code was wrong; the webapp re-ran itself, saw pid was locked,
and so didn't start!
2014-07-14 15:52:00 -04:00
Joey Hess
e36439d4fe deal with process-1.2.0.0 throwing an exception when program DNE (may only fix the tip of the iceberg)
Configure crashed on systems with that process and without eg, sha256sum.

The rest of the code in configure looks to work ok, since it uses sh -c to
probe for commands, and sh is always in path so it works.

Dunno about all the rest of git-annex. Not a huge amount of external
program use, other than git, so perhaps this won't be a large pain.

Note that boolSystem can throw an exception now if the program doesn't
exist. Could easily be changed back to False.
2014-07-13 22:33:28 -04:00
Joey Hess
e5b88713a1 refactor 2014-07-10 00:16:53 -04:00
Joey Hess
d9d76cf98b Fix minor FD leak in journal code.
Minor because normally only 1 FD is leaked per git-annex run. However,
the test suite leaks a few hundred FDs, and this broke it on the Debian
autobuilders, which seem to have a tigher than usual ulimit.

The leak was introduced by the lazy getDirectoryContents' that was
introduced in e6330988dd in order to scale to
millions of journal files -- if the lazy list was never fully consumed, the
directory handle did not get closed.

Instead, pull in openDirectory/readDirectory/closeDirectory code that I
already developed and submitted in a patch to the haskell directory library
earlier. Using this in journalDirty avoids the place that the lazy list
caused a problem. And using it in stageJournal eliminates the need for
getDirectoryContents'.

The getJournalFiles* functions are switched back to using the regular
strict getDirectoryContents. I'm not sure if those always consume the whole
list, so this avoids any leak. And the things that call those are things
like git annex unused, which also look at every file committed to the
git-annex branch, so would need more work to scale to insane numbers of
files anyway.
2014-07-09 23:36:53 -04:00
Joey Hess
33ad8beabb Support building with bloomfilter 2.0.0. 2014-07-07 12:24:12 -04:00
Joey Hess
c5c4fd76ec fix windows build 2014-07-05 11:18:59 -04:00
Joey Hess
c90e4e8778
work around getDirectoryContents not streaming lazily 2014-07-04 17:59:26 -04:00
Joey Hess
9fb8038166 Windows: Assistant now logs to daemon.log.
Yes, this means that git annex webapp on windows execs git-annex, which
execs itself to set env, and the execs itself again to redirect logs.

This is disgusting. This is Windows(TM).
2014-06-17 19:27:54 -04:00
Joey Hess
4fe2e53f5b finish fixing windows timezone madness
Rather than calculating the TSDelta once, and caching it, this now
reads the inode sential file's InodeCache file once, and then each time a
new InodeCache is generated, looks at the sentinal file to get the current
delta.

This way, if the time zone changes while git-annex is running, it will
adapt.

This adds some inneffiency, but only on Windows, and only 1 stat per new
file added. The worst innefficiency is that `git annex status` and
`git annex sync` will now (on Windows) stat the inode sentinal file once per
file in the repo.

It would be more efficient to use getCurrentTimeZone, rather than needing
to stat the sentinal file. This should be easy to do, once the time
package gets my bugfix patch.

This commit was sponsored by Jürgen Lüters.
2014-06-12 13:54:08 -04:00
Joey Hess
e4d7e2ebde fix for Windows file timestamp timezone madness
On Windows, changing the time zone causes the apparent mtime of files to
change. This confuses git-annex, which natually thinks this means the files
have actually been modified (since THAT'S WHAT A MTIME IS FOR, BILL <sheesh>).

Work around this stupidity, by using the inode sentinal file to detect if
the timezone has changed, and calculate a TSDelta, which will be applied
when generating InodeCaches.

This should add no overhead at all on unix. Indeed, I sped up a few
things slightly in the refactoring.

Seems to basically work! But it has a big known problem:
If the timezone changes while the assistant (or a long-running command)
runs, it won't notice, since it only checks the inode cache once, and
so will use the old delta for all new inode caches it generates for new
files it's added. Which will result in them seeming changed the next time
it runs.

This commit was sponsored by Vincent Demeester.
2014-06-12 13:42:21 -04:00
Joey Hess
db8982c45b minimal exports 2014-06-11 16:17:01 -04:00
Joey Hess
1c18056e15 deal with FAT on Linux timestamp issue
Deal with FAT's low resolution timestamps, which in combination with
Linux's caching of higher res timestamps while a FAT is mounted, caused
direct mode repositories on FAT to seem to have modified files after they
were unmounted and remounted.

This commit was sponsored by Fabrice Rossi.
2014-06-11 14:46:03 -04:00
Joey Hess
8ff9938d97 Fix build with wai 0.3.0.
This version of wai changed the type of Middleware, so I cannot seem
to liftIO inside it. So, got rid of a lot of not really needed
complexity to use System.Log.Logger's logging stuff, and just use
the standard wai stdout logger when debug logging is enabled.

Format may change some, and it logs http to stdout instead of stderr
now. Doesn't matter for the webapp since both go to the same log anyway.
2014-06-11 01:29:00 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
6eb5e6c135 Windows: Fix opening webapp when repository is in a directory with spaces in the path. 2014-06-10 18:37:33 -04:00
Joey Hess
7671d228af Avoid leaving behind .tmp files when failing in some cases, including importing files to a disk that is full. 2014-06-09 15:24:05 -04:00
Joey Hess
787d9e27ba Ignore setsid failures. 2014-06-09 14:44:18 -04:00
Joey Hess
a59ac68c22 qualify comment 2014-06-09 13:48:44 -04:00
Joey Hess
b7560e660e
fix build with old versions of warp 2014-06-04 14:37:08 -04:00
Joey Hess
0eb6576455 lazy bytestring fromStrict is not available in stable; use fromChunks instead 2014-05-29 20:25:01 -04:00
Joey Hess
450eecaa76 fix warning message
This should work even back in debian stable's warp
2014-05-29 13:49:45 -04:00
Joey Hess
b197ec8917 get rid of (completely safe) uses of Char8
Char8 often indicates an encoding bug. It didn't here, but I can avoid it
and not worry about it.
2014-05-27 20:26:10 -04:00
Joey Hess
a66c942645 refactor 2014-05-14 17:31:20 -04:00
Joey Hess
740de08461 fix windows build 2014-05-14 15:54:41 -04:00
Joey Hess
e391224516 setsid when running webapp in foreground too
This avoids ssh prompting for passwords on stdin, ever.

It may also change other behavior of other programs, as there is no
controlling terminal now. However, setsid was already done when running the
assistant in daemon mode, so any behavior changes should not be really new.
2014-05-14 14:26:28 -04:00
Joey Hess
2427832bed relicense general utility library code to BSD
Omitted a couple of files what have had significant contributions from
others.
2014-05-10 11:01:27 -03:00
Joey Hess
2aed2d8510 Merge branch 'master' into bootstrap3
Conflicts:
	debian/changelog
2014-05-02 15:32:49 -03:00
Joey Hess
72ec0ab736 hlint 2014-04-26 19:25:05 -04:00
Sören Brunk
54fe9af0bb create a local copy of Yesod.Form.Bootstrap3 2014-04-25 12:26:33 +02:00
Sören Brunk
00c1cd0db1 bootstrap3 forms 2014-04-21 19:47:05 +02:00
Joey Hess
515d251127
better approach for building distributionupdate
Use cabal include file
2014-04-21 10:57:01 -04:00
Joey Hess
a10d3731b8 fix distributionupdate build
Including avoiding needing cabal's defines for Utility.URI
2014-04-21 10:47:26 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
e53a85743e
adjust to not use cpp in modules used by configure 2014-04-14 13:37:12 -04:00
Joey Hess
2ff9ba9f74
add missing Network.URI Ord instance for Debian stable 2014-04-14 13:25:49 -04:00
Joey Hess
15917ec1a8 sync, assistant, remotedaemon: Use ssh connection caching for git pushes and pulls.
For sync, saves 1 ssh connection per remote. For remotedaemon, the same
ssh connection that is already open to run git-annex-shell notifychanges
is reused to pull from the remote.

Only potential problem is that this also enables connection caching
when the assistant syncs with a ssh remote. Including the sync it does
when a network connection has just come up. In that case, cached ssh
connections are likely to be stale, and so using them would hang.
Until I'm sure such problems have been dealt with, this commit needs to
stay on the remotecontrol branch, and not be merged to master.

This commit was sponsored by Alexandre Dupas.
2014-04-12 15:59:34 -04:00
Joey Hess
9fff243ff2
wrote test case; found bugs in date math; fixed and simplified using Data.Time.Calendar 2014-04-12 13:29:35 -04:00
Joey Hess
b815988d16
Improve handling on monthly/yearly scheduling.
Avoid back-to-back runs.
2014-04-12 12:58:32 -04:00
Joey Hess
d955cfe78a Improve handling on monthly/yearly scheduling.
Code was still buggy, it turns out (though the recursion checker caught
it). In the case of (Schedule (Monthly Nothing) AnyTime), where the last
run was on yyyy-12-31, it looped forever.

Also, the handling of (Schedule (Yearly Nothing) AnyTime) was wacky where
the last run was yyyy-12-31. It would suggest a window starting on the 3rd
for the next run (because 31 mod 28 is 3).

I think that originally I was wanted to avoid running on 01-01 if it had
just run on 12-31. But the code didn't accomplish this, and it's not
necessary anyway. This is supposed to calculate the next window meeting the
schedule, and for (Schedule (Monthly Nothing), the window starts at 01-01
and runs through 01-31. If that causes two back-to-back runs, well the next
one will not be until 02-01 at the earliest.

Also, back-to-back runs can be avoided, if desired, by using Divisible 2.
2014-04-11 21:42:43 -04:00
Joey Hess
e26dead493 guard against any remaining infinite recursion bugs; throw error if no candidate cn be found in next hundred years
Note that the exception thrown is not visible in the webapp currently
because it crashes one of Cronner's 2 worker threads, which is never
checked.
2014-04-11 18:08:46 -04:00
Joey Hess
9e16171c7f Last release didn't quite fix the high cpu issue in all cases, this should.
This is supposed to look for a day past the last day it ran, not a month
past.

Seems to work, at least in anarcat's test case.
2014-04-11 17:14:19 -04:00
Joey Hess
e0ca99b480
rename confusingly named "day" 2014-04-11 14:38:23 -04:00
Joey Hess
c678798a5c assistant: Fix high CPU usage triggered when a monthly fsck is scheduled, and the last time the job ran was a day of the month > 12. This caused a runaway loop. Thanks to Anarcat for his assistance, and to Maximiliano Curia for identifying the cause of this bug. 2014-04-11 14:34:09 -04:00
Joey Hess
a6df4698a3 Fix rsync progress parsing in locales that use comma in number display. Closes: #744148 2014-04-10 15:36:51 -04:00
Joey Hess
38862bf65d merge from propellor 2014-04-09 01:15:24 -04:00
Joey Hess
cbcb7f50d8 remotedaemon: avoid extraneous stdout output 2014-04-08 14:02:25 -04:00
Joey Hess
5af30678c7 factored out Utility.SimpleProtocol from the external special remote implementation 2014-04-05 13:29:28 -04:00
Joey Hess
611e060385 factor out library code (also used by propellor) 2014-04-03 21:25:59 -04:00
Joey Hess
b98db5d6f1 clean up warning in windows 2014-04-02 18:08:35 -04:00
Joey Hess
9b707f0116
windows fix try 2 2014-04-02 18:02:32 -04:00
Joey Hess
43ed9c7383
fix windows build 2014-04-02 17:21:36 -04:00
Joey Hess
523a98a952 merge from propellor 2014-04-01 17:53:55 -04:00
Joey Hess
bbd2efede3 refactor 2014-03-31 20:15:16 -04:00
Joey Hess
be76a0e22c avoid importing Common
it caused a loop when reusing this in propellor
2014-03-30 18:43:05 -04:00
Joey Hess
065248f3d2 Added required content configuration.
This includes checking when dropping files that any required content
configuration is satisfied. However, it does not yet include an active
check on the required content; the location log is trusted when checking
the required content expression.
2014-03-29 16:03:33 -04:00
Joey Hess
16387edd00 avoid exception when curl exits nonzero (due to eg, bad domain name) 2014-03-27 13:01:57 -04:00
Joey Hess
e426fac273 add desktop notifications
Motivation: Hook scripts for nautilus or other file managers
need to provide the user with feedback that a file is being downloaded.

This commit was sponsored by THM Schoemaker.
2014-03-22 14:12:19 -04:00
Joey Hess
b51f0e80ed
fix build with QuickCheck 2.7.1
For some reason this was working w/o a cast before, despite POSIXTime etc
being newtypes. It stopped working with the new QuickCheck:

Utility/QuickCheck.hs:31:33:
    No instance for (Integral POSIXTime)
      arising from a use of `arbitrarySizedIntegral'
    Possible fix: add an instance declaration for (Integral POSIXTime)
    In the first argument of `nonNegative', namely
      `arbitrarySizedIntegral'
    In the expression: nonNegative arbitrarySizedIntegral
    In an equation for `arbitrary':
        arbitrary = nonNegative arbitrarySizedIntegral
2014-03-20 11:08:40 -04:00