Commit graph

315 commits

Author SHA1 Message Date
Joey Hess
96613e85a9 build fix 2013-02-15 13:48:25 -04:00
Joey Hess
9e69fca5bb optimise sending to encrypted rsync
With an encrypted rsync remote, the encrpyted file can be renamed, rather
than being copied, in crippled filesystem mode. This gets back to just as
fast as non-crippled mode for this very common case.
2013-02-15 13:42:41 -04:00
Joey Hess
92b4a63a06 rsync special remote support for crippled filesystem mode
Cannot make a hard link, have to copy.

I did find a way to make it work without setting up a tree, just using
--include and --exclude. But it needs the same hash directories to be used
on both sides, which is normally not the case. Still, I hope one day I will
convert non-bare repos to use the same hash dirs as everything else, and
then this will get more efficient.
2013-02-15 13:33:36 -04:00
Joey Hess
47477b2807 crippled filesystem support, probing and initial support
git annex init probes for crippled filesystems, and sets direct mode, as
well as `annex.crippledfilesystem`.

Avoid manipulating permissions of files on crippled filesystems.
That would likely cause an exception to be thrown.

Very basic support in Command.Add for cripped filesystems; avoids the lock
down entirely since doing it needs both permissions and hard links.
Will make this better soon.
2013-02-14 14:15:26 -04:00
Joey Hess
18a6935e42 safe recv-key in direct mode
Checks the key's size and checksum. This is sorta expensive, but it avoids
needing to add another round-trip to the protocol.
2013-01-11 16:03:45 -04:00
Joey Hess
fec55e742f check for direct mode file change when copying from a local git remote
Only missing direct mode transfer check now is git-annex shell recvkey.
2013-01-10 11:56:06 -04:00
Joey Hess
a6a5ed8121 check for direct mode file change when copying to a local git remote 2013-01-10 11:45:44 -04:00
Joey Hess
1bc49b7158 Special remotes now all rollback storage of keys that get modified during the transfer, which can happen in direct mode. 2013-01-09 18:42:29 -04:00
Joey Hess
909f67443f Fix transferring files to special remotes in direct mode. 2013-01-06 14:29:01 -04:00
Joey Hess
24c6eae1b5 show errors 2013-01-02 13:50:16 -04:00
Joey Hess
4008590c68 type based git config handling for remotes
Still a couple of places that use git config ad-hoc, but this is most of it
done.
2013-01-01 13:58:14 -04:00
Joey Hess
ffdd08fd2e Merge branch 'master' into desymlink 2012-12-13 00:46:10 -04:00
Joey Hess
0d50a6105b whitespace fixes 2012-12-13 00:45:27 -04:00
Joey Hess
e7b8cb0063 direct mode committing 2012-12-12 19:20:38 -04:00
Joey Hess
b4c6da9cbd Got object sending working in direct mode.
However, I don't yet have a reliable way to deal with files being modified
while they're being transferred. I have code that detects it on the sending
side, but the receiver is still free to move the wrong content into its
annex, and record that it has the content. So that's not acceptable, and
I'll need to work on it some more.

However, at this point I can use a direct mode repository as a remote and
transfer files from and to it.
2012-12-08 17:03:39 -04:00
Joey Hess
5460414486 webdav: Avoid trying to set props, avoiding incompatability with livedrive.com. Needs DAV version 0.3. 2012-12-01 17:12:41 -04:00
Joey Hess
757f041cd8 instrument webdav test 2012-12-01 14:32:50 -04:00
Joey Hess
0b6c889012 webapp: S3 and Glacier forms now have a select list of all currently-supported AWS regions. 2012-12-01 14:11:37 -04:00
Joey Hess
020a25abe1 avoid unnecessary Maybe 2012-11-30 00:55:59 -04:00
Joey Hess
ea5d7292e6 dropping from web 2012-11-29 17:01:07 -04:00
Joey Hess
3b35cde0e8 assistant: Retrival from glacier now handled. 2012-11-29 15:23:33 -04:00
Joey Hess
8dd1d9aaf9 webapp: Defaults to sharing box.com account info with friends, allowing one-click enabling of the repository. 2012-11-28 13:31:49 -04:00
Joey Hess
5ff666ec99 rsync: Fix bug introduced in last release that broke encrypted rsync special remotes. 2012-11-27 16:29:31 -04:00
Joey Hess
356120652f remove redundant showOutput calls.
The meter code does that too.
2012-11-25 16:13:06 -04:00
Joey Hess
fb19d56476 progress bars for glacier downloads 2012-11-25 13:49:22 -04:00
Joey Hess
606c210378 progress bars for glacier uploads 2012-11-25 13:27:20 -04:00
Joey Hess
da6c738dad adjust glacier remote cost to 1000
Higher than any other remote, this is mostly due to the long retrieval
time, so it'd make sense to get a file from nearly any other remote.
(Unless it's behind a very slow connection.)
2012-11-22 16:59:10 -04:00
Joey Hess
f53496830a pass --quiet to checkpresent 2012-11-21 19:35:28 -04:00
Joey Hess
a5111a6d85 Amazon Glacier special remote; 100% working 2012-11-20 16:43:58 -04:00
Joey Hess
9221e62d87 Allow controlling whether login credentials for S3 and webdav are committed to the repository, by setting embedcreds=yes|no when running initremote. 2012-11-19 17:32:58 -04:00
Joey Hess
f7a7ec4ebf new storage regime implemented for webdav 2012-11-19 14:08:39 -04:00
Joey Hess
7b71685a93 Bugfix: directory special remote could loop forever storing a key when a too small chunksize was configured.
Ensure that each file has something written to it, even if the bytestring
chunk size is greater than the configured chunksize.

This means we may write a bit larger than the configured value, but only
when the configured value is very small; ie, < 8 kb.
2012-11-19 13:30:58 -04:00
Joey Hess
5f977cc725 directory special remote: Made more efficient and robust.
Files are now written to a tmp directory in the remote, and once all
chunks are written, etc, it's moved into the final place atomically.

For now, checkpresent still checks every single chunk of a file, because
the old method could leave partially transferred files with some chunks
present and others not.
2012-11-19 13:18:23 -04:00
Joey Hess
d3dfeeb3d9 remove annex/ from key locations used for webdav 2012-11-18 23:59:39 -04:00
Joey Hess
7df1e71fe3 S3: Added progress display for uploading and downloading. 2012-11-18 22:49:07 -04:00
Joey Hess
b0e08ae457 S3: upload progress display 2012-11-18 22:20:43 -04:00
Joey Hess
e2b7fc1ebd refactor 2012-11-18 21:50:16 -04:00
Joey Hess
afa2f9c967 upload progress bars for webdav! 2012-11-18 20:30:05 -04:00
Joey Hess
c8751be151 simplify 2012-11-18 18:27:53 -04:00
Joey Hess
81379bb29c better streaming while encrypting/decrypting
Both the directory and webdav special remotes used to have to buffer
the whole file contents before it could be decrypted, as they read
from chunks. Now the chunks are streamed through gpg with no buffering.
2012-11-18 15:27:44 -04:00
Joey Hess
3607c92222 fix warning 2012-11-18 14:06:54 -04:00
Joey Hess
8a6941a216 fix build with xml-conduit newer than in debian
The Element data type changed to use a map of attributes. Rather than
ifdef, I'm avoiding directly using that data type.
2012-11-18 13:46:38 -04:00
Joey Hess
7addb89dc1 webapp: support box.com 2012-11-17 15:30:11 -04:00
Joey Hess
1fe76b57d6 webdav now checks presence of and receives chunked content
Note that receiving encrypted chunked content currently involves buffering.
(So does doing so with the directory special remote.)
2012-11-16 23:16:18 -04:00
Joey Hess
0b3126a30b back to standard directory layout for webdav remotes
This allows deleting all chunks for a file with a single http command,
so it's a win after all.

However, does not look in the mixed case hash directories, which were
in the past used by the directory, etc remotes.
2012-11-16 18:14:07 -04:00
Joey Hess
a1869ad662 webdav now supports sending chunked content
Not yet getting it though.
2012-11-16 17:58:58 -04:00
Joey Hess
92d5d81c2c generic chunked content helper
However, directory still uses its optimzed chunked file writer, as it uses
less memory than the generic one in the helper.
2012-11-16 17:58:08 -04:00
Joey Hess
0f782bd028 encrypted webdav working 2012-11-16 13:57:32 -04:00
Joey Hess
bb28c6114a drop webdav compatability with the directory special remote etc
The benefit of using a compatable directory structure does not outweigh the
cost in complexity of handling the multiple locations content can be stored
in directory special remotes. And this also allows doing away with the parent
directories, which can't be made unwritable in DAV, so have no benefit
there. This will save 2 http calls per file store.

But, kept the directory hashing, just in case.
2012-11-16 00:42:33 -04:00
Joey Hess
a4b86c63d6 webdav is fully working in non-enctypted mode 2012-11-16 00:09:22 -04:00