mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
21009 commits 13 branches 442 tags 103 MiB
Commit graph

15199 commits

Author SHA1 Message Date
gernot
b07b1d8ec8 2014-08-05 20:14:25 +00:00
https://www.google.com/accounts/o8/id?id=AItOawkBTVYS5lTecuenAB01eHgfUxE20vWVpU4
beecf61fe0 How to remove a directory with git-annex added? 2014-08-05 18:07:56 +00:00
https://www.google.com/accounts/o8/id?id=AItOawkBTVYS5lTecuenAB01eHgfUxE20vWVpU4
d0779c815a How to remove a directory with git-annex added? 2014-08-05 18:04:38 +00:00
jg123h12jh3y12g3y
353cf1eaca 2014-08-05 14:46:51 +00:00
annexuser123
f58930f26c Added a comment 2014-08-04 18:00:00 +00:00
zardoz
13827c8a97 Added a comment 2014-08-04 08:05:37 +00:00
http://svario.it/gioele
f12a00f159 Added a comment: Multipart S3 support files > 5 GB 2014-08-04 06:00:45 +00:00
annexuser123
9f3845669f 2014-08-04 01:32:26 +00:00
annexuser123
dd312f26f3 2014-08-04 01:29:09 +00:00
annexuser123
03f97a5f53 2014-08-04 01:04:31 +00:00
Joey Hess
3b577e4358 Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-03 20:36:10 -04:00
Joey Hess
af620198fc devblog 2014-08-03 20:32:26 -04:00
Joey Hess
8587cd40ed Merge branch 'master' into newchunks 2014-08-03 19:48:50 -04:00
Joey Hess
93f20541f5 testremote --fast 2014-08-03 18:08:34 -04:00
Joey Hess
b35f7983ff convert gcrypt to new regime, including chunking 
Some reorg of Remote.Rsync code to export the things gcrypt needs.
 2014-08-03 17:31:10 -04:00
Joey Hess
e1e5853c94 rsync: support chunking 
Chunking does not speed up rsync at all, so it's only useful for
interop with the directory special remote.
 2014-08-03 15:07:43 -04:00
Joey Hess
c653e80829 Merge branch 'master' into newchunks 2014-08-03 15:04:10 -04:00
http://joeyh.name/
2d997d4d49 Added a comment 2014-08-03 18:59:58 +00:00
Joey Hess
c648548e1f formatting 2014-08-03 14:56:40 -04:00
http://joeyh.name/
63c00daa08 Added a comment 2014-08-03 18:40:26 +00:00
http://joeyh.name/
ac166a898e Added a comment 2014-08-03 18:27:32 +00:00
http://joeyh.name/
699a0a3bf8 Added a comment 2014-08-03 18:22:58 +00:00
zardoz
ef0404f5d4 2014-08-03 15:25:47 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmkuFJVGp6WVvJtIV5JYb8IqN8mRvSGQdI
116f70555e Added a comment: Would you accept a patch? 2014-08-03 01:18:54 +00:00
Joey Hess
f0adec1288 Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-02 19:18:54 -04:00
Joey Hess
42655caacc correction 2014-08-02 19:16:35 -04:00
http://joeyh.name/
51f774f8c8 Added a comment 2014-08-02 23:13:41 +00:00
Joey Hess
b5eb02bf77 Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-02 19:10:55 -04:00
http://joeyh.name/
2ca8ddca0f Added a comment 2014-08-02 23:08:44 +00:00
Joey Hess
3f8605870e devblog 2014-08-02 19:05:16 -04:00
divB
8fbdcd7d9f 2014-08-02 22:57:48 +00:00
Joey Hess
b261df735d convert bup to new ChunkedEncryptable API (but do not support chunking) 
bup already splits files and does rolling deltas, so there is no reason to
use chunking here.

The new API made it easier to add progress support for storeKey, so that's
done. Unfortunately, bup-split still outputs its own progress with -q,
so a little ugly, but not too bad.

Made dropping remove the branch for an object, for two reasons:

1. The new API calls removeKey to roll back a storeKey when the content
   changed unexpectedly.
2. So that testremote will be happy.

Also, fixed a bug that caused a crash when removing the branch for an
object in rollback.
 2014-08-02 18:48:49 -04:00
Joey Hess
0c7c39840d Merge branch 'master' into newchunks 2014-08-02 17:25:50 -04:00
Joey Hess
7f5cd868d7 hook: use ChunkedEncryptable 2014-08-02 17:25:16 -04:00
Joey Hess
d4dc1bce75 document that encryption + bup = inneficient 2014-08-02 17:04:40 -04:00
divB
668e67719a 2014-08-02 20:29:06 +00:00
divB
9973dc85e6 2014-08-02 20:24:15 +00:00
divB
a8b0311d8a 2014-08-02 20:23:10 +00:00
divB
84b5d1241c 2014-08-02 20:17:05 +00:00
divB
7d015c8eef 2014-08-02 20:12:42 +00:00
divB
49db204da4 2014-08-02 20:10:12 +00:00
Joey Hess
32e4368377 S3: support chunking 
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
 2014-08-02 15:51:58 -04:00
zardoz
5e2dd2ea86 Added a comment 2014-08-02 14:29:26 +00:00
zardoz
670f4d43e3 2014-08-02 12:21:38 +00:00
https://id.koumbit.net/anarcat
25ac3a4c4a some progress. maybe. 2014-08-02 02:46:07 +00:00
Joey Hess
0f7b8de91b update roadmap 2014-08-01 18:22:13 -04:00
Joey Hess
dabf0bc501 typography 2014-08-01 18:18:52 -04:00
http://joeyh.name/
195c403ae3 removed 2014-08-01 22:16:29 +00:00
Joey Hess
b978adb503 document testremote 2014-08-01 18:10:02 -04:00
Joey Hess
5aa2286e7b Merge branch 'newchunks' 
I am happy enough with this to make it live!
 2014-08-01 18:00:47 -04:00
Joey Hess
370c01811f Merge branch 'master' of ssh://git-annex.branchable.com 2014-08-01 17:56:57 -04:00
Joey Hess
85ec4130bc devblog 2014-08-01 17:56:31 -04:00
Joey Hess
1ee24a0366 testremote now tests with and without encryption 2014-08-01 17:52:40 -04:00
Joey Hess
50a1cac24f fix example external remote script to write files atomically 2014-08-01 17:52:09 -04:00
Joey Hess
20d7295386 improve testremote command, adding chunk size testing 
And also a --size parameter to configure the basic object size.
 2014-08-01 16:50:24 -04:00
Joey Hess
9720ee9e56 testremote: New command to test uploads/downloads to a remote. 
This only performs some basic tests so far; no testing of chunking or
resuming. Also, the existing encryption type of the remote is used; it
would be good later to derive an encrypted and a non-encrypted version of
the remote and test them both.

This commit was sponsored by Joseph Liu.
 2014-08-01 15:10:01 -04:00
gernot
0b078b2d37 2014-08-01 17:57:44 +00:00
Joey Hess
c03e1c5648 add new section for testing commands 2014-08-01 12:49:26 -04:00
https://www.google.com/accounts/o8/id?id=AItOawmAINLSovhWM_4_KrbngOcxduIbBuKv8ZA
0d5fe783e4 Added a comment 2014-08-01 09:28:21 +00:00
Xyem
4e9e700548 Added a comment 2014-08-01 09:05:45 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmAINLSovhWM_4_KrbngOcxduIbBuKv8ZA
532231a748 2014-08-01 08:56:23 +00:00
zardoz
b100938c37 Added a comment 2014-07-31 11:43:16 +00:00
zardoz
044dc0252d Add tip to make Firefox behave as expected when opening annexed HTML documents. 2014-07-31 10:48:11 +00:00
https://www.google.com/accounts/o8/id?id=AItOawm8BAEUyzYhORZmMuocRTk4M-3IumDm5VU
e1713220bc Added a comment: Bug still valid 2014-07-31 08:35:30 +00:00
http://olivier.mehani.name/
b9ddecac7a typo 2014-07-31 05:09:33 +00:00
https://www.google.com/accounts/o8/id?id=AItOawn-TDneVW-8kwb1fyTRAJfH3l1xs2VSEmk
a5eeefedd2 Added a comment 2014-07-30 20:37:27 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis
76d546adac Added a comment 2014-07-30 20:33:44 +00:00
Joey Hess
5848793d6b chunk size advice 2014-07-30 12:06:59 -04:00
Joey Hess
97f60ac140 Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-30 11:58:17 -04:00
http://joeyh.name/
64db37a974 Added a comment: interesting idea 2014-07-30 15:03:46 +00:00
http://joeyh.name/
1defbdb10c Added a comment 2014-07-30 14:55:16 +00:00
Joey Hess
83c58f17e0 mention possibility of parallel chunk upload/download 2014-07-30 10:43:21 -04:00
Joey Hess
58f8b8f250 devblog 2014-07-29 20:44:26 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlYGpCG2u-0fVbpQhIDMX_ZfE64E-K2EMI
585d7a84bf 2014-07-29 20:11:30 +00:00
Xyem
bb0ddec05d 2014-07-29 15:38:40 +00:00
http://svario.it/gioele
ff9131c93a new wish: add repository name to commit messages 2014-07-29 15:29:30 +00:00
http://svario.it/gioele
53518beb1c new bug: whereis does not work in direct mode 2014-07-29 15:17:09 +00:00
http://svario.it/gioele
2695dfbab5 Added a comment 2014-07-29 14:25:19 +00:00
https://id.koumbit.net/anarcat
bc64ec4880 i hope this is not too silly 2014-07-28 23:02:39 +00:00
https://id.koumbit.net/anarcat
61c31be21e found some files in misctmp 2014-07-28 22:56:14 +00:00
https://id.koumbit.net/anarcat
d6b8d63e16 2014-07-28 22:50:39 +00:00
https://id.koumbit.net/anarcat
9cb01f3a6e 2014-07-28 22:47:54 +00:00
https://id.koumbit.net/anarcat
3a5da63dc3 back from the dead, or not 2014-07-28 22:40:43 +00:00
Joey Hess
68189cc519 Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-28 17:26:29 -04:00
Joey Hess
e184e6c8ca devbog 2014-07-28 17:25:22 -04:00
Joey Hess
b39506d1cf expand to rolling hash based design 2014-07-28 17:11:37 -04:00
https://www.google.com/accounts/o8/id?id=AItOawkftzaCvV7EDKVDfJhsQZ3E1Vn-0db516w
100b44f46b Added a comment: One snag 2014-07-28 19:37:04 +00:00
Joey Hess
2474cf0032 make explicit the implicit requirement that CHECKPRESENT not say a key is present until it's all done being stored 2014-07-28 14:37:22 -04:00
gernot
955f2ab4c1 2014-07-28 17:16:24 +00:00
Joey Hess
406692c5be Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-28 13:05:19 -04:00
Joey Hess
a33dafae5a Merge branch 'master' of ssh://git-annex.branchable.com into newchunks 2014-07-28 13:03:43 -04:00
Joey Hess
e47182920c chunk log format should be extensible to allow for eg, logging when rolling hash chunks are used 2014-07-28 13:00:46 -04:00
divB
6249c5f30b 2014-07-27 23:16:19 +00:00
divB
b886ed1e25 2014-07-27 23:15:25 +00:00
Joey Hess
6c46a92040 devblog 2014-07-27 19:12:45 -04:00
Joey Hess
9d4a766cd7 resume interrupted chunked downloads 
Leverage the new chunked remotes to automatically resume downloads.
Sort of like rsync, although of course not as efficient since this
needs to start at a chunk boundry.

But, unlike rsync, this method will work for S3, WebDAV, external
special remotes, etc, etc. Only directory special remotes so far,
but many more soon!

This implementation will also properly handle starting a download
from one remote, interrupting, and resuming from another one, and so on.

(Resuming interrupted chunked uploads is similarly doable, although
slightly more expensive.)

This commit was sponsored by Thomas Djärv.
 2014-07-27 18:56:32 -04:00
Joey Hess
85d17a698d Merge branch 'master' into newchunks 
Conflicts:
	doc/design/assistant/chunks.mdwn
 2014-07-27 12:24:03 -04:00
Joey Hess
729d38a763 update 2014-07-27 12:23:28 -04:00
Joey Hess
7db60269eb update does for chunking 2014-07-26 23:39:51 -04:00
Joey Hess
fc10959f68 Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-26 20:52:27 -04:00
Joey Hess
5cd1025816 devblog 2014-07-26 20:51:58 -04:00
Joey Hess
275e284dda doc update for new chunking 2014-07-26 20:21:49 -04:00
https://www.google.com/accounts/o8/id?id=AItOawmURXBzaYE1gmVc-X9eLAyDat_6rHPl670
45a5276e23 added output of ls -lb in git directory to show that the file is not added to the annex 2014-07-26 16:50:32 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmURXBzaYE1gmVc-X9eLAyDat_6rHPl670
7bdac3a96b 2014-07-26 16:39:41 +00:00
Joey Hess
e4cb50db33 Merge branch 'master' into newchunks 2014-07-26 12:02:48 -04:00
https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8
712465c80e Added a comment 2014-07-26 14:57:53 +00:00
Joey Hess
655bdfd5bd Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-25 20:57:52 -04:00
Joey Hess
8a46a89adc devblog 2014-07-25 20:56:28 -04:00
Joey Hess
8f93982df6 use same hash directories for chunked key as are used for its parent 
This avoids a proliferation of hash directories when using new-style
chunking, and should improve performance since chunks are accessed
in sequence and so should have a common locality.

Of course, when a chunked key is encrypted, its hash directories have no
relation to the parent key.

This commit was sponsored by Christian Kellermann.
 2014-07-25 16:09:23 -04:00
Joey Hess
1755c5de40 thought about chunk key hashing 2014-07-25 15:12:51 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlVUq_c3-lrQBculOEUu3yjvdavE7JbvEI
b43a6bc7e5 Added a comment 2014-07-25 10:02:53 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnG_weCiH6IleQfVmRPmFJflHQMHdAcDkU
54f3ff878c add a note about direct mode 2014-07-25 07:57:21 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnG_weCiH6IleQfVmRPmFJflHQMHdAcDkU
5820a8d3cd file a bug 2014-07-25 07:42:52 +00:00
Joey Hess
d52c2b37ad Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-24 16:49:45 -04:00
Joey Hess
3de535cdac devblog 2014-07-24 16:48:07 -04:00
Joey Hess
e2c44bf656 implement chunk logs 
Slightly tricky as they are not normal UUIDBased logs, but are instead maps
from (uuid, chunksize) to chunkcount.

This commit was sponsored by Frank Thomas.
 2014-07-24 16:23:36 -04:00
Joey Hess
d751591ac8 add chunk metadata to Key 
Added new fields for chunk number, and chunk size. These will not appear
in normal keys ever, but will be used for chunked data stored on special
remotes.

This commit was sponsored by Jouni K Seppanen.
 2014-07-24 13:36:23 -04:00
Joey Hess
4bbc629cb0 document new chunk logfiles 2014-07-24 13:28:54 -04:00
Joey Hess
9e472c9c59 clarify field order reqirement 2014-07-24 12:54:29 -04:00
Joey Hess
937197842e update 2014-07-24 12:41:34 -04:00
Xyem
d0ea0a0479 2014-07-24 12:31:31 +00:00
Joey Hess
ca1d80d708 chunk then encrypt 2014-07-23 22:38:14 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlASRUX9jg2piT1lge4KkBU_ZGty6tvM5c
a42d2a6e4b added gpg instructions 2014-07-24 00:48:33 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmp_51DcIT2sqrOhtg7LgCj2Pyaa5ujxTI
46fec9c1ce 2014-07-23 23:38:55 +00:00
markusk
6df94fb489 Added a comment 2014-07-23 23:18:37 +00:00
http://joeyh.name/
7eb483156d Added a comment 2014-07-23 22:10:37 +00:00
Joey Hess
d82d9b4f30 Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-23 17:56:31 -04:00
Joey Hess
a354bf488b link 2014-07-23 17:56:26 -04:00
Joey Hess
09d807100b devblog 2014-07-23 17:55:44 -04:00
Joey Hess
f15c1fdc8f minor 2014-07-23 17:55:28 -04:00
markusk
0ae8bef4dd 2014-07-23 21:45:18 +00:00
Joey Hess
20627e9fab 4 designs for better chunking 
Having a hard time finding a way to totally obscure file sizes, but
otherwise happy with design #4.

This commit was sponsored by Michael Alan Dorman.
 2014-07-23 17:41:48 -04:00
markusk
ce7a8d895f 2014-07-23 21:01:22 +00:00
Joey Hess
5262e6a372 recommend users not change the chunksize 
Suppose A is 10 mb, and B is 20 mb, and the upload speed is the same. If B
starts first, when A will overwrite the file it is uploading for the 1st chunk.
Then A uploads the second chunk, and once A is done, B finishes the 1st chunk
and uploads its second. We now have 1(from A), 2(from B), so data loss.
 2014-07-23 16:46:15 -04:00
markusk
19a0c16232 New bug using webapp with --listen=IP:Port 2014-07-23 20:29:40 +00:00
http://joeyh.name/
c8b2e701ef Added a comment 2014-07-23 15:14:22 +00:00
Joey Hess
95590f878d Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-23 11:06:05 -04:00
http://joeyh.name/
48bb63b034 Added a comment 2014-07-23 15:05:36 +00:00
Joey Hess
28b355e98c close; not a git-annex bug 2014-07-23 11:04:29 -04:00
http://joeyh.name/
b427f72260 Added a comment 2014-07-23 15:02:04 +00:00
Joey Hess
c675e83f86 Merge branch 'master' of ssh://git-annex.branchable.com 2014-07-23 10:43:45 -04:00
Joey Hess
70d932a6d7 update per request 2014-07-23 10:43:09 -04:00
https://www.google.com/accounts/o8/id?id=AItOawkt8DjG40gowj_ETchFLDS_Z216tr7p1xw
d2894d7e48 Added a comment: ok some progress 2014-07-23 14:27:51 +00:00
https://www.google.com/accounts/o8/id?id=AItOawkt8DjG40gowj_ETchFLDS_Z216tr7p1xw
dcc43ef57a 2014-07-23 10:36:34 +00:00
divB
e47a651ce9 Added a comment 2014-07-23 05:20:47 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnD-eBOHeC-94xRxJ10VZHT8t_4FbOLiPM
b040fd1420 2014-07-23 00:19:10 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmMLeU-zCzx2mc5pL2XT8a1UNkQwHAHjg8
98fda2fc5f Added a comment 2014-07-22 02:10:14 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmMLeU-zCzx2mc5pL2XT8a1UNkQwHAHjg8
37797784f3 Added a comment 2014-07-22 01:47:29 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmMLeU-zCzx2mc5pL2XT8a1UNkQwHAHjg8
005325b924 Added a comment 2014-07-22 00:29:20 +00:00
divB
1c32e11369 Added a comment 2014-07-22 00:24:47 +00:00