Commit graph

689 commits

Author SHA1 Message Date
Joey Hess
534c29deae implemented old Richih wishlist about remote/uuid info
* info: Can now display info about a given uuid.
  * Added to remote/uuid info: Count of the number of keys present
    on the remote, and their size. This is rather expensive to calculate,
    so comes last and --fast will disable it.
  * Git remote info now includes the date of the last sync with the remote.
2015-01-13 18:13:14 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
6b3d0cb11a bittorrent: Fix locking problem when using addurl file://
Fixes:
/home/joey/tmp/xxx/.git/annex/misctmp/torrent18347: openFile: resource busy (file is locked)
2014-12-30 13:07:20 -04:00
Joey Hess
c9a3e80d32 fixed all remaining build warnings on Windows 2014-12-29 17:30:20 -04:00
Joey Hess
27fb7e514d Fix build with -f-S3. 2014-12-19 16:53:25 -04:00
Joey Hess
ef12386924 When possible, build with the haskell torrent library for parsing torrent files. 2014-12-18 14:26:10 -04:00
Joey Hess
e2214f6ac8 remove default untrusted hack for bittorrent
This is better handled by checkPresent always failing.
2014-12-17 15:38:00 -04:00
Joey Hess
e08fa65131 note about http://hackage.haskell.org/package/torrent 2014-12-17 15:34:38 -04:00
Joey Hess
6ca54c521d make checkkey always fail for torrents
See comment.
2014-12-17 14:54:54 -04:00
Joey Hess
2192c54877 more robust fallback when a file is available from multiple torrents and some torrent files cannot be downloaded 2014-12-17 14:38:04 -04:00
Joey Hess
bf9df3fc7e fix fencepost error and aria resume after partial download of multi-file torrent 2014-12-17 14:21:48 -04:00
Joey Hess
3a7d0be120 remove excess directory 2014-12-17 14:17:19 -04:00
Joey Hess
d5cbbe1b9a fix torrentUrlNum when there is no #n 2014-12-17 14:07:05 -04:00
Joey Hess
7e422269a6 move dummy uuids to Annex.UUID 2014-12-17 13:57:52 -04:00
Joey Hess
af05ac3ec2 add aria2 progress parsing 2014-12-17 13:40:04 -04:00
Joey Hess
a7690de016 Added bittorrent special remote
addurl behavior change: When downloading an url ending in .torrent,
it will download files from bittorrent, instead of the old behavior
of adding the torrent file to the repository.

Added Recommends on aria2 and bittornado | bittorrent.

This commit was sponsored by Asbjørn Sloth Tønnesen.
2014-12-16 23:22:46 -04:00
Joey Hess
65bce2c80d reformat 2014-12-16 15:26:13 -04:00
Joey Hess
67c05daf5e sanitize filepaths provided by checkUrl 2014-12-11 20:08:49 -04:00
Joey Hess
8a17bcb0be simplify external special remote implementation 2014-12-11 17:44:27 -04:00
Joey Hess
bce7e0dd96 use subdir for addurl when it creates multiple files
The --file parameter specifies the subdir in this mode.
2014-12-11 16:09:56 -04:00
Joey Hess
2cd84fcc8b Expand checkurl to support recommended filename, and multi-file-urls
This commit was sponsored by an anonymous bitcoiner.
2014-12-11 15:33:42 -04:00
Joey Hess
7ae16bb6f7 Revert "let url claims optionally include a suggested filename"
This reverts commit 85df9c30e9.

Putting filename in the claim was a bad idea.
2014-12-11 14:09:57 -04:00
Joey Hess
85df9c30e9 let url claims optionally include a suggested filename 2014-12-11 12:47:57 -04:00
Joey Hess
aafb121068 unmangled mangled urls from the log before passing to external special remote 2014-12-08 19:27:40 -04:00
Joey Hess
30bf112185 Urls can now be claimed by remotes. This will allow creating, for example, a external special remote that handles magnet: and *.torrent urls. 2014-12-08 19:15:07 -04:00
Joey Hess
ee27298b91 implement CLAIMURL for external special remote 2014-12-08 13:57:13 -04:00
Joey Hess
cb6e16947d add stub claimUrl 2014-12-08 13:40:15 -04:00
Joey Hess
8093008ef4 External special remote protocol now includes commands for setting and getting the urls associated with a key. 2014-12-08 13:32:46 -04:00
Joey Hess
911ba8d972 Merge branch 's3-aws' 2014-12-03 14:10:52 -04:00
Joey Hess
55fa1789dd Don't show "(gpg)" when decrypting the remote encryption cipher, since this could be taken to read that's the only time git-annex runs gpg, which is not the case. 2014-12-02 13:50:45 -04:00
Joey Hess
0a891fcfc5 support S3 front-end used by globalways.net
This threw an unusual exception w/o an error message when probing to see if
the bucket exists yet. So rather than relying on tryS3, catch all
exceptions.

This does mean that it might get an exception for some transient network
error, think this means the bucket DNE yet, and try to create it, and then
fail when it already exists.
2014-11-05 12:42:12 -04:00
Joey Hess
93feefae05 Revert "work around minimum part size problem"
This reverts commit a42022d8ff.

I misunderstood the cause of the problem.
2014-11-04 16:21:55 -04:00
Joey Hess
a42022d8ff work around minimum part size problem
When uploading the last part of a file, which was 640229 bytes, S3 rejected
that part: "Your proposed upload is smaller than the minimum allowed size"

I don't know what the minimum is, but the fix is just to include the last
part into the previous part. Since this can result in a part that's
double-sized, use half-sized parts normally.
2014-11-04 16:06:13 -04:00
Joey Hess
ad2125e24a fix a couple type errors and the progress bar 2014-11-04 15:39:48 -04:00
Joey Hess
fccdd61eec fix memory leak
Unfortunately, I don't fully understand why it was leaking using the old
method of a lazy bytestring. I just know that it was leaking, despite
neither hGetUntilMetered nor byteStringPopper seeming to leak by
themselves.

The new method avoids the lazy bytestring, and simply reads chunks from the
handle and streams them out to the http socket.
2014-11-04 15:22:08 -04:00
Joey Hess
29871e320c combine 2 checks 2014-11-04 14:47:18 -04:00
Joey Hess
0f78f197eb casts; now fully working.. but still leaking
Still seems to buffer the whole partsize in memory, but I'm pretty sure my
code is not what's doing it. See https://github.com/aristidb/aws/issues/142
2014-11-03 21:12:15 -04:00
Joey Hess
f0551578d6 this should avoid leaking memory 2014-11-03 20:49:30 -04:00
Joey Hess
4230b56b79 logic error 2014-11-03 20:15:33 -04:00
Joey Hess
62de9a39bf WIP 3 2014-11-03 20:04:42 -04:00
Joey Hess
d16382e99f WIP 2 2014-11-03 19:50:33 -04:00
Joey Hess
5360417436 WIP try sending using RequestBodyStreamChunked
May not work; if it does this is gonna be the simplest way to get good
memory size and progress reporting.
2014-11-03 19:18:46 -04:00
Joey Hess
8f61bfad51 link to memory leak bug 2014-11-03 17:55:05 -04:00
Joey Hess
711b18a6eb improve info display for multipart 2014-11-03 17:24:53 -04:00
Joey Hess
2c53f331bd fix build 2014-11-03 17:23:46 -04:00
Joey Hess
6a965cf8d7 adjust version check
I assume 0.10.6 will have the fix for the bug I reported, which got fixed
in master already..
2014-11-03 16:23:00 -04:00
Joey Hess
5c3d9d6caa show multipart configuration in git annex info s3remote 2014-11-03 16:07:41 -04:00
Joey Hess
a3ec6ed73b Merge branch 'master' into s3-aws-multipart 2014-11-03 16:05:03 -04:00
Joey Hess
8faeb25076 finish multipart support using unreleased update to aws lib to yield etags
Untested and not even compiled yet.

Testing should include checks that file content streams through without
buffering in memory.

Note that CL.consume causes all the etags to be buffered in memory.
This is probably nearly unavoidable, since a request has to be constructed
that contains the list of etags in its body. (While it might be possible to
stream generation of the body, that would entail making a http request that
dribbles out parts of the body as the multipart uploads complete, which is
not likely to work well..

To limit this being a problem, it's best for partsize to be set to some
suitably large value, like 1gb. Then a full terabyte file will need only
1024 etags to be stored, which will probably use around 1 mb of memory.
2014-11-03 16:04:55 -04:00