Commit graph

517 commits

Author SHA1 Message Date
Joey Hess
e4cb50db33 Merge branch 'master' into newchunks 2014-07-26 12:02:48 -04:00
Joey Hess
005aded3e0 Fix cost calculation for non-encrypted remotes.
Encyptable types of remotes that were not actually encrypted still had
the encryptedRemoteCostAdj applied to their configured cost, which was a
bug.
2014-07-25 17:29:59 -04:00
Joey Hess
9e8a4a0950 support new style chunking in directory special remote
Only when storing non-encrypted so far, not retrieving or checking if a key
is present or removing.

This commit was sponsored by Renaud Casenave-Péré.
2014-07-25 16:21:01 -04:00
Joey Hess
ab4cce4114 core implementation of new style chunking
Not yet used by any special remotes, but should not be too hard to add it
to most of them.

storeChunks is the hairy bit! It's loosely based on
Remote.Directory.storeLegacyChunked. The object is read in using a lazy
bytestring, which is streamed though, creating chunks as needed, without
ever buffering more than 1 chunk in memory.

Getting the progress meter update to work right was also fun, since
progress meter values are absolute. Finessed by constructing an offset
meter.

This commit was sponsored by Richard Collins.
2014-07-25 16:20:32 -04:00
Joey Hess
ceea04e77f move meteredWriteFileChunks out of legacy 2014-07-24 16:42:35 -04:00
Joey Hess
e2c44bf656 implement chunk logs
Slightly tricky as they are not normal UUIDBased logs, but are instead maps
from (uuid, chunksize) to chunkcount.

This commit was sponsored by Frank Thomas.
2014-07-24 16:23:36 -04:00
Joey Hess
bbdb2c04d5 improve chunk data types 2014-07-24 15:08:07 -04:00
Joey Hess
9e2d49d441 prepare for new style chunking
Moved old legacy chunking code, and cleaned up the directory and webdav
remotes use of it, so when no chunking is configured, that code is not
used.

The config for new style chunking will be chunk=1M instead of chunksize=1M.

There should be no behavior changes from this commit.

This commit was sponsored by Andreas Laas.
2014-07-24 14:49:22 -04:00
Joey Hess
ec5ed2af9d Set gcrypt-publish-participants when setting up a gcrypt repository, to avoid unncessary passphrase prompts.
This is a security/usability tradeoff. To avoid exposing the gpg key ids
who can decrypt the repository, users can unset
gcrypt-publish-participants.

The gcrypt-publish-participants option is available in my fork of
git-remote-gcrypt.

This commit was sponsored by Christopher Kernahan.
2014-07-15 17:33:14 -04:00
Joey Hess
cdf61071bc optimise handling of unavailable repos
The exception handling resulted in git config --list being run twice
for unavailable repos. This dials it back down to running it only once.
2014-07-15 14:45:27 -04:00
Joey Hess
bd514eb65a catch exception when repo is really not available 2014-07-15 14:39:31 -04:00
Joey Hess
522a0922b8 sync: Fix git sync with local git remotes even when they don't have an annex.uuid set.
Catch an exception when ensureInitialized is run in a non-initted
repository. In this case, just read the git config, so that the Git.Repo
object is not LocalUnknown, which is what is used to represent remotes
on eg, drives that are not connected.

The assistant already got this right, and like with the assistant, this
causes an implicit git-annex init of the local remote on the second sync,
once the git-annex branch has been pushed to it.
See this comment for more analysis:
http://git-annex.branchable.com/todo/Recovering_from_a_bad_sync/#comment-64e469a2c1969829ee149cbb41b1c138

This commit was sponsored by jscit.
2014-07-15 14:27:43 -04:00
Joey Hess
604740b720 S3: Deal with AWS ACL configurations that do not allow creating or checking the location of a bucket, but only reading and writing content to it. 2014-07-11 15:21:43 -04:00
Joey Hess
26ee27915a refactor locking 2014-07-10 00:32:23 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
2f84659d51 fix build with old versions of bytestring 2014-06-06 14:04:35 -04:00
Joey Hess
0c2a14e4aa fix dodgy use of Char8
I don't know if this was a bug, but I don't know if it was not a bug
either.

See also,
http://git-annex.branchable.com/bugs/Truncated_file_transferred_via_S3/
where the file is not truncated, but mangled..
2014-05-27 20:31:25 -04:00
Joey Hess
c07343e4f7 initremote/enableremote: Basic support for using with regular git remotes
initremote stores the location of an already existing git remote, and
enableremote setups up a remote using its stored location.
2014-05-22 13:42:17 -04:00
Joey Hess
c34b5e09f8 factor out getRemoteGitConfig 2014-05-16 16:08:20 -04:00
Fraser Tweedale
4eb72392b4 execute remote.<name>.annex-shell on remote, if set
It is useful to be able to specify an alternative git-annex-shell
program to execute on the remote, e.g., to run a version not on the
PATH.  Use remote.<name>.annex-shell if specified, instead of the
default "git-annex-shell" i.e., first so-named executable on the
PATH.
2014-05-16 15:46:43 -04:00
Joey Hess
0b899fa2f1 show a much longer message when annex-ignore is automatically set, to help the user fix their problem 2014-05-16 12:58:50 -04:00
Joey Hess
b1cddea7e4 remove odd character that snuck in somehow and broke build 2014-05-15 16:36:19 -04:00
Robie Basak
4184566627 ddar special remote 2014-05-15 16:32:44 -04:00
Joey Hess
f00cb21037 Bring back rsync -p, but only when git-annex is running on a non-crippled file system. This is a better approach to fix #700282 while not unncessarily losing file permissions on non-crippled systems. 2014-04-17 14:31:42 -04:00
Joey Hess
5af30678c7 factored out Utility.SimpleProtocol from the external special remote implementation 2014-04-05 13:29:28 -04:00
Joey Hess
3b8d5f03bb Fix glacier repo creation bug
Version 5.20140227 broke creation of glacier repositories, not including
the datacenter and vault in their configuration. This bug is fixed, but
glacier repositories set up with the broken version of git-annex need to
have the datacenter and vault set in order to be usable. This can be done
using git annex enableremote to add the missing settings. For details, see
http://git-annex.branchable.com/bugs/problems_with_glacier/
2014-03-27 14:30:36 -04:00
Alberto Berti
0f7c2dd39b Fix thaoe remote to work with latest tahoe (v. 1.10.0) 2014-03-26 00:31:02 +01:00
Joey Hess
e426fac273 add desktop notifications
Motivation: Hook scripts for nautilus or other file managers
need to provide the user with feedback that a file is being downloaded.

This commit was sponsored by THM Schoemaker.
2014-03-22 14:12:19 -04:00
Joey Hess
40b599eff2 rsync special remote: Fix slashes when used on Windows. 2014-03-18 13:02:10 -04:00
Joey Hess
b63276309e clean up cleanup action enumeration 2014-03-13 19:06:26 -04:00
Joey Hess
4d06037fdd Fix zombie leak and general inneficiency when copying files to a local git repo.
Benchmarking this with 1000 small files being copied, the time reduced from
15.98s to 14.64s -- an 8% improvement in the non-data-transfer overhead of
git-annex copy.
2014-03-06 17:13:27 -04:00
Joey Hess
aa377ed567 webdav: When built with a new enough haskell DAV (0.6), disable the http response timeout, which was only 5 seconds. 2014-03-05 13:51:54 -04:00
Joey Hess
1f98d6fb00 glacier: Pass --region to glacier checkpresent.
I suppose this is not necessary when it has a local cache, so I didn't
notice it was missing.
2014-03-04 23:22:24 -04:00
Joey Hess
a1432bce2f Put non-object tmp files in .git/annex/misctmp, leaving .git/annex/tmp for only partially transferred objects.
This allows eg, putting .git/annex/tmp on a ram disk, if the disk IO
of temp object files is too annoying (and if you don't want to keep
partially transferred objects across reboots).

.git/annex/misctmp must be on the same filesystem as the git work tree,
since files are moved to there in a way that will not work cross-device,
as well as symlinked into there.

I first wanted to put the tmp objects in .git/annex/objects/tmp, but
that would pose transition problems on upgrade when partially transferred
objects existed.

git annex info does not currently show the size of .git/annex/misctemp,
since it should stay small. It would also be ok to make something clean it
out, periodically.
2014-02-26 16:52:56 -04:00
Joey Hess
2aeb0750f9 more DAV url fixes for windows 2014-02-25 16:16:14 -04:00
Joey Hess
b1931d1cc1 add protocol-level debugging for dav 2014-02-25 15:58:44 -04:00
Joey Hess
2b66aaa763 Windows webdav: Fix DOS path separator bug.
Use posix </> etc for urls.
2014-02-25 15:26:33 -04:00
Joey Hess
360ecb9f35 fix bare repo optimisation on Windows 2014-02-25 13:47:09 -04:00
Joey Hess
06142f4943
fix #740010 properly 2014-02-25 01:55:01 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
c69d6eb035 Make annex.web-options be used in several places that call curl. 2014-02-24 21:29:37 -04:00
Joey Hess
d5a2b498f6 webdav: When built with DAV 0.6.0, use the new DAV monad to avoid locking files, which is not needed by git-annex's use of webdav, and does not work on Box.com. 2014-02-24 18:21:51 -04:00
Joey Hess
45e7040142 webapp: Fix creation of box.com, S3, and Glacier repositories, broken in 5.20140221. 2014-02-24 15:29:17 -04:00
Joey Hess
ded4ab5704 Fix handling of rsync remote urls containing a username, including rsync.net.
This breakage seems to have been caused way back in a1eded86,
but I am pretty sure rsync.net support has not been entirely
broken since last April. AFAICS, the generated .ssh/config
has not changed since then -- it has never included a Username setting
line. So, I am puzzled at when this reversion was introduced.

Note that the breakage only affected checkpresent and remove. Upload and
download use the ssh connection caching, which includes a -l username.
2014-02-21 13:20:57 -04:00
Joey Hess
7d288d83c9 glacier: Do not try to run glacier value create when an existing glacier remote is enabled. 2014-02-20 15:56:26 -04:00
Joey Hess
4e0be2792b remove Read instance for Ref
Removed instance, got it all to build using fromRef. (With a few things
that really need to show something using a ref for debugging stubbed out.)

Then added back Read instance, and made Logs.View use it for serialization.
This changes the view log format.
2014-02-19 01:19:57 -04:00
Joey Hess
7b19c7d25b cleanup thanks to Utility.PID 2014-02-11 15:39:51 -04:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
e885080d06 Add progress display for transfers to/from external special remotes. 2014-02-10 21:33:22 -04:00
Joey Hess
08afe3a1f6 fix failing test case on Windows
ensure file being modified is all read before it's opened for write
2014-02-03 10:20:18 -04:00