Commit graph

21035 commits

Author SHA1 Message Date
Joey Hess
d3778e631b remove write bit when storing to local gcrypt repo
Same as is done by rsync, and for regular git repos.
2014-08-03 20:25:44 -04:00
Joey Hess
d12becfdde fix removal from local gcrypt repo that had files stored using rsync
When files are stored using rsync, they have their write bit removed;
so does the directory they're put in. The local repo code did not turn
these bits back on, so failed to remove.
2014-08-03 20:21:46 -04:00
Joey Hess
8601f8f571 when not using rsync (for local gcrypt repo), display own progress meter 2014-08-03 20:19:04 -04:00
Joey Hess
1cd2273035 finally properly fixed ssh zombie leak
The leak was caused by the thread that sshd'd to send transferinfo
not waiting on its ssh. Doh.
2014-08-03 20:14:20 -04:00
Joey Hess
8587cd40ed Merge branch 'master' into newchunks 2014-08-03 19:48:50 -04:00
Joey Hess
93f20541f5 testremote --fast 2014-08-03 18:08:34 -04:00
Joey Hess
b35f7983ff convert gcrypt to new regime, including chunking
Some reorg of Remote.Rsync code to export the things gcrypt needs.
2014-08-03 17:31:10 -04:00
Joey Hess
f5f961215b finish making rsync support chunking
This breaks gcrypt, which relies on some internals of the rsync remote.
To fix next..
2014-08-03 16:54:57 -04:00
Joey Hess
6c450aad1d move ugly rsync zombie workaround
This reaping of any processes came to cause me problems when redoing the
rsync special remote -- a gpg process that was running gets waited on and
the place that then checks its return code fails.

I cannot reproduce any zombies when using the rsync special remote.
But I still can when using a normal git remote, accessed over ssh.
There is 1 zombie per file downloaded without this horrible hack enabled.

So, move the hack to only be used in that case.
2014-08-03 16:53:29 -04:00
Joey Hess
b3fe23b552 remove redundant progress meter display code
specialRemote handles all meter display, so this is redundant.
2014-08-03 16:18:40 -04:00
Joey Hess
4b16989e98 roll ChunkedEncryptable into Special and improve interface
Allow disabling progress displays, for eg, rsync.
2014-08-03 15:40:01 -04:00
Joey Hess
e1e5853c94 rsync: support chunking
Chunking does not speed up rsync at all, so it's only useful for
interop with the directory special remote.
2014-08-03 15:07:43 -04:00
Joey Hess
c653e80829 Merge branch 'master' into newchunks 2014-08-03 15:04:10 -04:00
http://joeyh.name/
2d997d4d49 Added a comment 2014-08-03 18:59:58 +00:00
Joey Hess
c648548e1f formatting 2014-08-03 14:56:40 -04:00
http://joeyh.name/
63c00daa08 Added a comment 2014-08-03 18:40:26 +00:00
http://joeyh.name/
ac166a898e Added a comment 2014-08-03 18:27:32 +00:00
http://joeyh.name/
699a0a3bf8 Added a comment 2014-08-03 18:22:58 +00:00
zardoz
ef0404f5d4 2014-08-03 15:25:47 +00:00
Joey Hess
00f92a7e59 whitespace 2014-08-03 01:21:38 -04:00
Joey Hess
d05b7b9182 better byteRetriever
Make the byteRetriever be passed the callback that consumes the bytestring.

This way, there's no worries about the lazy bytestring not all being read
when the resource that's creating it is closed.

Which in turn lets bup, ddar, and S3 each switch from using an unncessary
fileRetriver to a byteRetriever. So, more efficient on chunks and encrypted
files.

The only remaining fileRetrievers are hook and external, which really do
retrieve to files.
2014-08-03 01:12:24 -04:00
https://www.google.com/accounts/o8/id?id=AItOawmkuFJVGp6WVvJtIV5JYb8IqN8mRvSGQdI
116f70555e Added a comment: Would you accept a patch? 2014-08-03 01:18:54 +00:00
Joey Hess
f0adec1288 Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-02 19:18:54 -04:00
Joey Hess
42655caacc correction 2014-08-02 19:16:35 -04:00
http://joeyh.name/
51f774f8c8 Added a comment 2014-08-02 23:13:41 +00:00
Joey Hess
b5eb02bf77 Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-02 19:10:55 -04:00
http://joeyh.name/
2ca8ddca0f Added a comment 2014-08-02 23:08:44 +00:00
Joey Hess
3f8605870e devblog 2014-08-02 19:05:16 -04:00
Joey Hess
19b71cfb8f convert ddar to new ChunkedEncryptable API (but do not support chunking)
Since ddar de-deuplicates, I assume there is no benefit from chunking.

This has not been tested!
2014-08-02 18:58:48 -04:00
divB
8fbdcd7d9f 2014-08-02 22:57:48 +00:00
Joey Hess
b261df735d convert bup to new ChunkedEncryptable API (but do not support chunking)
bup already splits files and does rolling deltas, so there is no reason to
use chunking here.

The new API made it easier to add progress support for storeKey, so that's
done. Unfortunately, bup-split still outputs its own progress with -q,
so a little ugly, but not too bad.

Made dropping remove the branch for an object, for two reasons:

1. The new API calls removeKey to roll back a storeKey when the content
   changed unexpectedly.
2. So that testremote will be happy.

Also, fixed a bug that caused a crash when removing the branch for an
object in rollback.
2014-08-02 18:48:49 -04:00
Joey Hess
0c7c39840d Merge branch 'master' into newchunks 2014-08-02 17:25:50 -04:00
Joey Hess
7f5cd868d7 hook: use ChunkedEncryptable 2014-08-02 17:25:16 -04:00
Joey Hess
d4dc1bce75 document that encryption + bup = inneficient 2014-08-02 17:04:40 -04:00
Joey Hess
0eb1f057c4 convert glacier to new ChunkedEncryptable API (but do not support chunking)
Chunking would complicate the assistant's code that checks when a pending
retrieval of a key from glacier is done. It would perhaps be nice to
support it to allow resuming, but not right now.

Converting to the new API still simplifies the code.
2014-08-02 16:59:07 -04:00
divB
668e67719a 2014-08-02 20:29:06 +00:00
divB
9973dc85e6 2014-08-02 20:24:15 +00:00
divB
a8b0311d8a 2014-08-02 20:23:10 +00:00
divB
84b5d1241c 2014-08-02 20:17:05 +00:00
divB
7d015c8eef 2014-08-02 20:12:42 +00:00
divB
49db204da4 2014-08-02 20:10:12 +00:00
Joey Hess
32e4368377 S3: support chunking
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
2014-08-02 15:51:58 -04:00
Joey Hess
c3750901d8 specialize Preparer a bit, so resourcePrepare can be added
The forall a. in Preparer made resourcePrepare not seem to be usable, so
I specialized a to Bool. Which works for both Preparer Storer and
Preparer Retriever, but wouldn't let the Preparer be used for hasKey
as it currently stands.
2014-08-02 15:34:09 -04:00
zardoz
5e2dd2ea86 Added a comment 2014-08-02 14:29:26 +00:00
zardoz
670f4d43e3 2014-08-02 12:21:38 +00:00
https://id.koumbit.net/anarcat
25ac3a4c4a some progress. maybe. 2014-08-02 02:46:07 +00:00
Joey Hess
0f7b8de91b update roadmap 2014-08-01 18:22:13 -04:00
Joey Hess
dabf0bc501 typography 2014-08-01 18:18:52 -04:00
http://joeyh.name/
195c403ae3 removed 2014-08-01 22:16:29 +00:00
Joey Hess
b978adb503 document testremote 2014-08-01 18:10:02 -04:00