Commit graph

738 commits

Author SHA1 Message Date
Joey Hess
800e9f2658 tahoe: Use ~/.tahoe-git-annex/ rather than ~/.tahoe/git-annex/ to avoid old versions of tahoe create-client choking. 2015-06-09 15:29:16 -04:00
Joey Hess
f2486b21dd show S3 urls for public repos in whereis
Note that it's possible for a S3 bucket to be configured to allow public
access, but for git-annex to not know that it is. I chose to not show the
url unless public=yes.
2015-06-05 16:52:38 -04:00
Joey Hess
5f0f063a7a S3: Publically accessible buckets can be used without creds. 2015-06-05 16:23:35 -04:00
Joey Hess
4acd28bf21 public=yes config to send AclPublicRead
In my tests, this has to be set when uploading a file to the bucket
and then the file can be accessed using the bucketname.s3.amazonaws.com
url.

Setting it when creating the bucket didn't seem to make the whole bucket
public, or allow accessing files stored in it. But I have gone ahead and
also sent it when creating the bucket just in case that is needed in some
case.
2015-06-05 14:38:01 -04:00
Joey Hess
334fd6d598 groundwork for readonly access
Split S3Info out of S3Handle and added some stubs
2015-06-05 13:12:45 -04:00
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
77c43a388e fromkey, registerurl: Allow urls to be specified instead of keys, and generate URL keys.
This is especially useful because the caller doesn't need to generate valid
url keys, which involves some escaping of characters, and may involve
taking a md5sum of the url if it's too long.
2015-05-22 22:41:36 -04:00
Joey Hess
ecb0d5c087 use lock pools throughout git-annex
The one exception is in Utility.Daemon. As long as a process only
daemonizes once, which seems reasonable, and as long as it avoids calling
checkDaemon once it's already running as a daemon, the fcntl locking
gotchas won't be a problem there.

Annex.LockFile has it's own separate lock pool layer, which has been
renamed to LockCache. This is a persistent cache of locks that persist
until closed.

This is not quite done; lockContent stil needs to be converted.
2015-05-19 14:09:52 -04:00
Joey Hess
61ccf95004 Avoid accumulating transfer failure log files unless the assistant is being used.
Only the assistant uses these, and only the assistant cleans them up, so
make only git annex transferkeys write them,

There is one behavior change from this. If glacier is being used, and a
manual git annex get --from glacier fails because the file isn't available
yet, the assistant will no longer later see that failed transfer file and
retry the get. Hope no-one depended on that old behavior.
2015-05-12 15:53:38 -04:00
Joey Hess
a812d598ef Take space that will be used by running downloads into account when checking annex.diskreserve. 2015-05-12 15:20:22 -04:00
Joey Hess
e27b97d364 Merge branch 'master' into concurrentprogress
Conflicts:
	Command/Fsck.hs
	Messages.hs
	Remote/Directory.hs
	Remote/Git.hs
	Remote/Helper/Special.hs
	Types/Remote.hs
	debian/changelog
	git-annex.cabal
2015-05-12 13:23:22 -04:00
Joey Hess
9f14f51d63 generalied elem/notElem in ghc 7.10 require some additional type signatures when using OverloadedStrings 2015-05-10 15:41:41 -04:00
Joey Hess
4aba1c74bd remaining dataenc to sandi conversions
I've tested all the dataenc to sandi conversions except Assistant.XMPP,
and all have unchanged behavior, including behavior on large unicode code
points.
2015-05-07 18:07:13 -04:00
Joey Hess
ee78958798 S3: Fix incompatability with bucket names used by hS3; the aws library cannot handle upper-case bucket names. git-annex now converts them to lower case automatically.
For example, it failed to get files from a bucket named S3.

Also fixes `git annex initremote UPPERCASE type=S3`, which failed with the
new aws library, with a signing error message.
2015-04-27 18:00:58 -04:00
Joey Hess
cfbeb1e7b7 Fix bogus failure of fsck --fast. 2015-04-27 17:40:21 -04:00
Joey Hess
22a4e92df7 S3: git annex enableremote will not create a bucket name, which failed since the bucket already exists. 2015-04-23 14:16:53 -04:00
Joey Hess
b3eccec68c S3: git annex info will show additional information about a S3 remote (endpoint, port, storage class) 2015-04-23 14:12:25 -04:00
Joey Hess
ae9bbf25a0 convert all log prorities, not just debug
In particular, error should go to stderr
2015-04-21 15:59:30 -04:00
Joey Hess
3b3aaf0d56 S3: Enable debug logging when annex.debug or --debug is set.
To debug a bug report, but generally useful.
2015-04-21 15:55:42 -04:00
Joey Hess
addc82dab7 removed all uses of undefined from code base
It's a code smell, can lead to hard to diagnose error messages.
2015-04-19 00:38:29 -04:00
Joey Hess
0def1f0b53 Fix fsck --from a git remote in a local directory, and from a directory special remote. This was a reversion caused by the relative path changes in 5.20150113.
The directory special remote was not affected in its normal configuration,
since annex-directory is an absolute path normally. But it could fail
when a relative path was used.

The git remote was affected even when an absolute path to it was used in
.git/config, since git-annex now converts all such paths to relative.
2015-04-18 13:36:12 -04:00
Joey Hess
a2902cdaaf add filename to progress bar, and display ok/failed at end
This needed plumbing an AssociatedFile through retrieveKeyFileCheap.
2015-04-14 16:35:10 -04:00
Joey Hess
dc4de7faf7 add missing progress bar 2015-04-14 16:00:20 -04:00
Joey Hess
86a2f9dc4d Merge branch 'master' into concurrentprogress
Conflicts:
	debian/changelog
2015-04-14 15:35:15 -04:00
Joey Hess
efd6a8e3aa bittorrent: Fix handling of magnet links. 2015-04-14 12:57:01 -04:00
Joey Hess
75b6b5cbc7 only display built-in meters in parallel mode 2015-04-10 15:20:23 -04:00
Joey Hess
f8e700ed06 use built-in progress meters for git when in parallel mode 2015-04-10 15:15:21 -04:00
Joey Hess
30aa902174 relay external special remote stderr through progress suppression machinery (eep!)
It sounds worse than it is. ;)

Some external special remotes may run commands that display progress on
stderr. If git-annex is run with --quiet, this should filter out such
displays while letting the errors through.
2015-04-04 14:54:03 -04:00
Joey Hess
2343f99c85 well along the way to fully quiet --quiet
Came up with a generic way to filter out progress messages while keeping
errors, for commands that use stderr for both.

--json mode will disable command outputs too.
2015-04-04 14:34:03 -04:00
Joey Hess
e87f3b40eb propigate outer output state into inner state when running onLocal
Otherwise, progress displays would not be suppressed here when running with
--quiet. Interesting wrinkle!
2015-04-03 20:08:38 -04:00
Joey Hess
20fb91a7ad WIP on making --quiet silence progress, and infra for concurrent progress bars 2015-04-03 16:48:30 -04:00
Joey Hess
1c91024978 rename bothHandles -> ioHandles 2015-04-03 15:35:18 -04:00
Joey Hess
f0195b2a43 Fix GETURLS in external special remote protocol to strip downloader prefix from logged url info before checking for the specified prefix.
This doesn't change what GETURLS returns, but only whether it matches
any prefix that the external special remote asked for.
2015-03-27 18:49:03 -04:00
Joey Hess
707293ba7e remotedaemon: Fixed support for notifications of changes to gcrypt remotes, which was never tested and didn't quite work before. 2015-03-16 15:28:29 -04:00
Joey Hess
6045406deb Added SETURIPRESENT and SETURIMISSING to external special remote protocol
Useful for things like ipfs that don't use regular urls.

An external special remote can add a regular url to a key, and then
git-annex get will download it from the web. But for ipfs, we want to
instead tell git-annex that the uri uses OtherDownloader. Before this
change, the external special remote protocol lacked a way to do that.
2015-03-05 13:50:15 -04:00
Joey Hess
9b93278e8a metadata: Fix encoding problem that led to mojibake when storing metadata strings that contained both unicode characters and a space (or '!') character.
The fix is to stop using w82s, which does not properly reconstitute unicode
strings. Instrad, use utf8 bytestring to get the [Word8] to base64. This
passes unicode through perfectly, including any invalid filesystem encoded
characters.

Note that toB64 / fromB64 are also used for creds and cipher
embedding. It would be unfortunate if this change broke those uses.

For cipher embedding, note that ciphers can contain arbitrary bytes (should
really be using ByteString.Char8 there). Testing indicated it's not safe to
use the new fromB64 there; I think that characters were incorrectly
combined.

For credpair embedding, the username or password could contain unicode.
Before, that unicode would fail to round-trip through the b64.
So, I guess this is not going to break any embedded creds that worked
before.

This bug may have affected some creds before, and if so,
this change will not fix old ones, but should fix new ones at least.
2015-03-04 12:54:30 -04:00
Joey Hess
450ee53ab6 When re-execing git-annex, use current program location, rather than ~/.config/git-annex/program, when possible.
Most of the time, there will be no discreprancy between programPath and
readProgramFile.

But, the programFile might have been written by an old version of git-annex
that is still installed, while a newer one is currently running. In this
case, we want to run the same one that's currently running.

This is especially important for things like the GIT_SSH=git-annex used for
ssh connection caching.

The only code that still uses readProgramFile directly is the upgrade code,
which needs to know where the standalone git-annex was installed, in order to
upgrade it.
2015-02-28 17:23:13 -04:00
Joey Hess
5be7ba7ee5 The ssh-options git config is now used by gcrypt, rsync, and ddar special remotes that use ssh as a transport. 2015-02-12 15:44:10 -04:00
Joey Hess
52e40970c8 avoid unncessary IO 2015-02-12 15:33:44 -04:00
Joey Hess
a22eaaae27 comment 2015-02-09 14:16:42 -04:00
Joey Hess
69a9c98e71 glacier: Detect when the glacier command in PATH is the wrong one, from boto, rather than from glacier-cli, and refuse to use it, since the boto program fails to fail when passed parameters it does not understand. 2015-02-06 14:39:27 -04:00
Joey Hess
1af8107fec windows build fix 2015-01-29 13:46:57 -04:00
Joey Hess
e0187d5d12 test suite found a problem with today's work
". def" did not do what I thought it would, at all.
2015-01-28 18:05:08 -04:00
Joey Hess
009bd050c1 implement annex.tune.objecthashlower
Split out Annex.DirHashes which never really belonged in Locations.
2015-01-28 16:52:08 -04:00
Joey Hess
e8c376e0ad import Data.Default in Common 2015-01-28 16:11:28 -04:00
Joey Hess
0fd5f257d0 groundwork for parameterizing hash depth 2015-01-28 15:55:17 -04:00
Joey Hess
32fac4b71b remove unnecessary use of MissingH 2015-01-21 13:36:48 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
4f657aa14e add getFileSize, which can get the real size of a large file on Windows
Avoid using fileSize which maxes out at just 2 gb on Windows.
Instead, use hFileSize, which doesn't have a bounded size.
Fixes support for files > 2 gb on Windows.

Note that the InodeCache code only needs to compare a file size,
so it doesn't matter it the file size wraps. So it has been
left as-is. This was necessary both to avoid invalidating existing inode
caches, and because the code passed FileStatus around and would have become
more expensive if it called getFileSize.

This commit was sponsored by Christian Dietrich.
2015-01-20 17:09:24 -04:00
Joey Hess
534c29deae implemented old Richih wishlist about remote/uuid info
* info: Can now display info about a given uuid.
  * Added to remote/uuid info: Count of the number of keys present
    on the remote, and their size. This is rather expensive to calculate,
    so comes last and --fast will disable it.
  * Git remote info now includes the date of the last sync with the remote.
2015-01-13 18:13:14 -04:00