Commit graph

1659 commits

Author SHA1 Message Date
Joey Hess
d84eab8a8a Revert "S3, Glacier, WebDAV: Fix bug that prevented accessing the creds when the repository was configured with encryption=shared embedcreds=yes."
This reverts commit fbdeeeed5f.

I can find no basis for that commit and think that I made it in error.
setRemoteCredPair always encrypts using the cipher from remoteCipher,
even when the cipher is shared.
2014-09-18 15:21:47 -04:00
Joey Hess
44e7d6e1fe add: In direct mode, adding an annex symlink will check it into git, as was already done in indirect mode. 2014-09-18 14:24:47 -04:00
Joey Hess
9964584c34 WebDav: Fix enableremote crash when the remote already exists. (Bug introduced in version 5.20140817.) 2014-09-17 13:04:55 -04:00
Joey Hess
2abeae7582 Windows: Avoid crashing trying to list gpg secret keys, for gcrypt which is not yet supported on Windows. 2014-09-16 13:40:44 -04:00
Joey Hess
09e20c3827 assistant: Detect when repository has been deleted or moved, and automatically shut down the assistant. Closes: #761261 2014-09-15 14:45:37 -04:00
Joey Hess
05e7518c3e prep 2014-09-15 10:47:10 -04:00
Joey Hess
70b2160d32 Fix build with optparse-applicative 0.10. Closes: #761484
This also works with 0.9, and probably 0.8.
2014-09-14 12:30:37 -04:00
Joey Hess
7482166180 Rather than crashing when there's a problem with the requested bloomfilter capacity/accuracy, fall back to a reasonable default bloom filter size. 2014-09-12 12:26:12 -04:00
Joey Hess
9711d529c8 WORM backend: Switched to include the relative path to the file inside the repository, rather than just the file's base name. Note that if you're relying on such things to keep files separate with WORM, you should really be using a better backend. 2014-09-11 14:50:18 -04:00
Joey Hess
4c429ad7ee sync: Ensure that pending changes to git-annex branch are committed when in direct mode. (Fixing a very minor reversion.) 2014-09-11 14:35:28 -04:00
Joey Hess
0598412e5c Fix transfer lock file FD leak that could occur when two separate git-annex processes were both working to perform the same set of transfers. 2014-09-11 13:53:26 -04:00
Joey Hess
9ed63d1545 Promote file not found warning message to an error. 2014-09-11 13:36:28 -04:00
Joey Hess
a97c9e43b7 The annex-rsync-transport configuration is now also used when checking if a key is present on a rsync remote, and when dropping a key from the remote. 2014-09-11 13:21:35 -04:00
Joey Hess
11f111bf1a Fix parsing of ipv6 address in git remote address when it was not formatted as an url. 2014-09-10 14:17:02 -04:00
Joey Hess
b874f84086 New annex.hardlink setting. Closes: #758593
* New annex.hardlink setting. Closes: #758593
* init: Automatically detect when a repository was cloned with --shared,
  and set annex.hardlink=true, as well as marking the repository as
  untrusted.

Had to reorganize Logs.Trust a bit to avoid a cycle between it and
Annex.Init.
2014-09-05 13:44:09 -04:00
Joey Hess
0881f6d45c webapp: Fixed visual glitch in xmpp pairing.
It seems that all other uses of <div .col-sm-9> occur outside of
<div .content-box>. This one occurred inside it, when xmpp pairing.

This was introduced in the bootstrap 3 conversion.
2014-09-04 16:18:20 -04:00
Joey Hess
62e6ad8bee prep release 2014-08-31 12:30:29 -07:00
Joey Hess
6eb5c3f479 Do not preserve permissions and acls when copying files from one local git repository to another. Timestamps are still preserved as long as cp --preserve=timestamps is supported.
This avoids cp -a overriding the default mode acls that the user might have
set in a git repository.

With GNU cp, this behavior change should not be a breaking change, because
git-anex also uses rsync sometimes in the same situation, and has only ever
preserved timestamps when using rsync.

Systems without GNU cp will no longer use cp -a, but instead just cp.
So, timestamps will no longer be preserved. Preserving timestamps when
copying between repos is not guaranteed anyway.

Closes: #729757
2014-08-26 17:10:25 -07:00
Joey Hess
756a6277ff Fix stub git-annex test support when built without tasty. 2014-08-23 18:46:08 -07:00
Joey Hess
4405650828 Fix handing of autocorrection when running outside a git repository.
Old behavior was to take the first fuzzy match. Now, it checks the globa
git config, and runs the normal fuzzy handling, including failing to run a
semi-random command by default.
2014-08-23 16:51:33 -07:00
Joey Hess
96dc423e39 When accessing a local remote, shut down git-cat-file processes afterwards, to ensure that remotes on removable media can be unmounted. Closes: #758630
This does mean that eg, copying multiple files to a local remote will
become slightly slower, since it now restarts git-cat-file after each copy.
Should not be significant slowdown.

The reason git-cat-file is run on the remote at all is to update its
location log. In order to add an item to it, it needs to get the current
content of the log. Finding a way to avoid needing to do that would be a
good path to avoiding this slowdown if it does become a problem somehow.

This commit was sponsored by Evan Deaubl.
2014-08-20 12:07:57 -04:00
Joey Hess
092041fab0 Ensure that all lock fds are close-on-exec, fixing various problems with them being inherited by child processes such as git commands.
(With the exception of daemon pid locking.)

This fixes at part of #758630. I reproduced the assistant locking eg, a
removable drive's annex journal lock file and forking a long-running
git-cat-file process that inherited that lock.

This did not affect Windows.

Considered doing a portable Utility.LockFile layer, but git-annex uses
posix locks in several special ways that have no direct Windows equivilant,
and it seems like it would mostly be a complication.

This commit was sponsored by Protonet.
2014-08-20 11:37:02 -04:00
Joey Hess
37293dc28f Make --help work when not in a git repository. Closes: #758592
Note that this means getopt parsing is done even when not in a git
repository, even though currently cmdnorepo is not passed the results of
it. I'd like to move to cmdnorepo not doing its own ad-hoc option parsing,
so this is really a good thing. (But as long as eg, getOptionFlag needs an
Annex monad, it cannot be used in cmdnorepo handling.)

There is a potential for problems if any cmdnorepo branch of a command
handles options that are not in its regular getopt, but that would be a bug
anyway.
2014-08-19 12:55:15 -04:00
Joey Hess
f273882567 prep release 2014-08-17 10:31:21 -04:00
Joey Hess
dd619c7166 Switched from the old haskell HTTP library to http-conduit.
The hoary old HTTP library was only used when checking if an url exists,
when curl was not available. It had many problems, including not supporting
https at all.

Now, this is done using http-conduit for all urls that it supports. Falls
back to curl for any url that http-conduit doesn't like (probably ftp etc,
but could also be an url that its parser chokes on for whatever reason).

This adds a new dependency on http-conduit, but webdav support already
indirectly depended on that, and the s3-aws branch also uses it.

This opens up the possibility of using http-conduit for large file
downloads, but for now I've left it using wget/curl.

This commit was sponsored by Paul Tötterman.
2014-08-15 17:37:42 -04:00
Joey Hess
852185c242 git-annex-shell sendkey: Don't fail if a remote asks for a key to be sent that already has a transfer lock file indicating it's being sent to that remote. The remote may have moved between networks, or reconnected. 2014-08-15 14:17:05 -04:00
Joey Hess
bb6cec3461 direct: Avoid leaving file content in misctemp if interrupted. 2014-08-15 13:38:05 -04:00
Joey Hess
fbdeeeed5f S3, Glacier, WebDAV: Fix bug that prevented accessing the creds when the repository was configured with encryption=shared embedcreds=yes.
Since encryption=shared, the encryption key is stored in the git repo, so
there is no point at all in encrypting the creds, also stored in the git
repo with that key. So `initremote` doesn't. The creds are simply stored
base-64 encoded.

However, it then tried to always decrypt creds when encryption was used..
2014-08-12 15:35:29 -04:00
Joey Hess
f0df660570 WORM backend: When adding a file in a subdirectory, avoid including the subdirectory in the key name. 2014-08-12 14:38:53 -04:00
Joey Hess
d8be828734 direct: Fix ugly warning messages.
replaceFileOr was broken and ran the rollback action always.
Luckily, for replaceFile, the rollback action was safe to run, since it
just nuked a temp file that had already been moved into place.

However, when `git annex direct` used replaeFileOr, its rollback printed a
scary message:

  /home/joey/tmp/rrrr/.git/annex/misctmp/tmp32268: rename: does not exist (No such file or directory)

There was actually no bad result though.
2014-08-12 13:00:08 -04:00
Joey Hess
7285896996 make windows depend on new enough unix-compat to get inode numbers 2014-08-12 12:30:40 -04:00
Joey Hess
2fd9518f72 unlock: Better error handling; continue past files that are not available or cannot be unlocked due to disk space, and try all specified files. 2014-08-09 11:09:54 -04:00
Joey Hess
c3f8512475 WebDAV: Avoid buffering whole file in memory when downloading.
httpBodyRetriever will later also be used by S3

This commit was sponsored by Ethan Aubin.
2014-08-08 13:40:55 -04:00
Joey Hess
871b6cb886 DAV version turns out to be 1.0. 2014-08-08 12:50:51 -04:00
Joey Hess
2dd8dab314 WebDAV: Avoid buffering whole file in memory when uploading.
The httpStorer will later also be used by S3.

This commit was sponsored by Torbjørn Thorsen.
2014-08-07 19:32:23 -04:00
Joey Hess
0b1b85d9ea use DAV monad
This speeds up the webdav special remote somewhat, since it often now
groups actions together in a single http connection when eg, storing a
file.

Legacy chunks are still supported, but have not been sped up.

This depends on a as-yet unreleased version of DAV.

This commit was sponsored by Thomas Hochstein.
2014-08-07 17:32:57 -04:00
Joey Hess
aacb0b2823 convert WebDAV to new special remote interface, adding new-style chunking support
Reusing http connection when operating on chunks is not done yet,
I had to submit some patches to DAV to support that. However, this is no
slower than old-style chunking was.

Note that it's a fileRetriever and a fileStorer, despite DAV using
bytestrings that would allow streaming. As a result, upload/download of
encrypted files is made a bit more expensive, since it spools them to temp
files. This was needed to get the progress meters to work.

There are probably ways to avoid that.. But it turns out that the current
DAV interface buffers the whole file content in memory, and I have
sent in a patch to DAV to improve its interfaces. Using the new interfaces,
it's certainly going to need to be a fileStorer, in order to read the file
size from the file (getting the size of a bytestring would destroy
laziness). It should be possible to use the new interface to make it be a
byteRetriever, so I'll change that when I get to it.

This commit was sponsored by Andreas Olsson.
2014-08-06 16:57:06 -04:00
Joey Hess
d12becfdde fix removal from local gcrypt repo that had files stored using rsync
When files are stored using rsync, they have their write bit removed;
so does the directory they're put in. The local repo code did not turn
these bits back on, so failed to remove.
2014-08-03 20:21:46 -04:00
Joey Hess
b35f7983ff convert gcrypt to new regime, including chunking
Some reorg of Remote.Rsync code to export the things gcrypt needs.
2014-08-03 17:31:10 -04:00
Joey Hess
e1e5853c94 rsync: support chunking
Chunking does not speed up rsync at all, so it's only useful for
interop with the directory special remote.
2014-08-03 15:07:43 -04:00
Joey Hess
b261df735d convert bup to new ChunkedEncryptable API (but do not support chunking)
bup already splits files and does rolling deltas, so there is no reason to
use chunking here.

The new API made it easier to add progress support for storeKey, so that's
done. Unfortunately, bup-split still outputs its own progress with -q,
so a little ugly, but not too bad.

Made dropping remove the branch for an object, for two reasons:

1. The new API calls removeKey to roll back a storeKey when the content
   changed unexpectedly.
2. So that testremote will be happy.

Also, fixed a bug that caused a crash when removing the branch for an
object in rollback.
2014-08-02 18:48:49 -04:00
Joey Hess
0c7c39840d Merge branch 'master' into newchunks 2014-08-02 17:25:50 -04:00
Joey Hess
7f5cd868d7 hook: use ChunkedEncryptable 2014-08-02 17:25:16 -04:00
Joey Hess
32e4368377 S3: support chunking
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
2014-08-02 15:51:58 -04:00
Joey Hess
5aa2286e7b Merge branch 'newchunks'
I am happy enough with this to make it live!
2014-08-01 18:00:47 -04:00
Joey Hess
9720ee9e56 testremote: New command to test uploads/downloads to a remote.
This only performs some basic tests so far; no testing of chunking or
resuming. Also, the existing encryption type of the remote is used; it
would be good later to derive an encrypted and a non-encrypted version of
the remote and test them both.

This commit was sponsored by Joseph Liu.
2014-08-01 15:10:01 -04:00
Joey Hess
c03e1c5648 add new section for testing commands 2014-08-01 12:49:26 -04:00
Joey Hess
76d894f2e5 Display exception message when a transfer fails due to an exception.
For example, I had a copy to a remote that was failing for an unknown
reason. This let me see the exception was createDirectory: permission
denied; the underlying problem being a permissions issue.
2014-07-30 15:57:19 -04:00
Joey Hess
b5ac627fee WebDAV: Dropped support for DAV before 0.6.1.
0.6.1 is in testing, and stable does not have DAV at all, so I can dispense
with this compatability code
2014-07-30 11:20:35 -04:00
Joey Hess
c0dc134cde support chunking for all external special remotes!
Removing code and at the same time adding great features, including
upload/download resuming.

This commit was sponsored by Romain Lenglet.
2014-07-29 18:50:20 -04:00