mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
24681 commits 13 branches 442 tags 103 MiB
Commit graph

312 commits

Author SHA1 Message Date
Joey Hess
6fcca2f13e WIP converting S3 special remote from hS3 to aws library 
Currently, initremote works, but not the other operations. They should be
fairly easy to add from this base.

Also, https://github.com/aristidb/aws/issues/119 blocks internet archive
support.

Note that since http-conduit is used, this also adds https support to S3.
Although git-annex encrypts everything anyway, so that may not be extremely
useful. It is not enabled by default, because existing S3 special remotes
have port=80 in their config. Setting port=443 will enable it.

This commit was sponsored by Daniel Brockman.
 2014-08-08 19:00:53 -04:00
Joey Hess
aacb0b2823 convert WebDAV to new special remote interface, adding new-style chunking support 
Reusing http connection when operating on chunks is not done yet,
I had to submit some patches to DAV to support that. However, this is no
slower than old-style chunking was.

Note that it's a fileRetriever and a fileStorer, despite DAV using
bytestrings that would allow streaming. As a result, upload/download of
encrypted files is made a bit more expensive, since it spools them to temp
files. This was needed to get the progress meters to work.

There are probably ways to avoid that.. But it turns out that the current
DAV interface buffers the whole file content in memory, and I have
sent in a patch to DAV to improve its interfaces. Using the new interfaces,
it's certainly going to need to be a fileStorer, in order to read the file
size from the file (getting the size of a bytestring would destroy
laziness). It should be possible to use the new interface to make it be a
byteRetriever, so I'll change that when I get to it.

This commit was sponsored by Andreas Olsson.
 2014-08-06 16:57:06 -04:00
Joey Hess
b35f7983ff convert gcrypt to new regime, including chunking 
Some reorg of Remote.Rsync code to export the things gcrypt needs.
 2014-08-03 17:31:10 -04:00
Joey Hess
e1e5853c94 rsync: support chunking 
Chunking does not speed up rsync at all, so it's only useful for
interop with the directory special remote.
 2014-08-03 15:07:43 -04:00
Joey Hess
c653e80829 Merge branch 'master' into newchunks 2014-08-03 15:04:10 -04:00
Joey Hess
7f5cd868d7 hook: use ChunkedEncryptable 2014-08-02 17:25:16 -04:00
Joey Hess
d4dc1bce75 document that encryption + bup = inneficient 2014-08-02 17:04:40 -04:00
Joey Hess
32e4368377 S3: support chunking 
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
 2014-08-02 15:51:58 -04:00
http://joeyh.name/
195c403ae3 removed 2014-08-01 22:16:29 +00:00
Joey Hess
50a1cac24f fix example external remote script to write files atomically 2014-08-01 17:52:09 -04:00
Joey Hess
7db60269eb update does for chunking 2014-07-26 23:39:51 -04:00
Joey Hess
275e284dda doc update for new chunking 2014-07-26 20:21:49 -04:00
Joey Hess
5262e6a372 recommend users not change the chunksize 
Suppose A is 10 mb, and B is 20 mb, and the upload speed is the same. If B
starts first, when A will overwrite the file it is uploading for the 1st chunk.
Then A uploads the second chunk, and once A is done, B finishes the 1st chunk
and uploads its second. We now have 1(from A), 2(from B), so data loss.
 2014-07-23 16:46:15 -04:00
Joey Hess
ec5ed2af9d Set gcrypt-publish-participants when setting up a gcrypt repository, to avoid unncessary passphrase prompts. 
This is a security/usability tradeoff. To avoid exposing the gpg key ids
who can decrypt the repository, users can unset
gcrypt-publish-participants.

The gcrypt-publish-participants option is available in my fork of
git-remote-gcrypt.

This commit was sponsored by Christopher Kernahan.
 2014-07-15 17:33:14 -04:00
https://www.google.com/accounts/o8/id?id=AItOawndSPd52H_L7hcYsJjLQ4kR1LopprHy7Xw
3700fb00af 2014-07-10 22:37:59 +00:00
http://joeyh.name/
c50872cd6b Added a comment: that is an ancient version of git-annex, Shane 2014-07-02 17:11:24 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlhaXbLZQ5uQ6fM2kV0n8bj6IwZjx2CLeM
6bca9849cf Added a comment: Annex can't find Glacier remote 2014-06-26 05:50:38 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8
9f9499bcff removed 2014-05-21 04:07:59 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8
74c04de41b Added a comment: Box.com and unknown CA? 2014-05-21 04:07:36 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8
461555e33e Added a comment: Box.com and unknown CA? 2014-05-21 04:07:30 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw
aee63eaa84 Changing to box.com's current webdav server adress. 2014-05-18 05:29:06 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw
40c9aa1506 Added a comment 2014-05-17 20:47:01 +00:00
http://joeyh.name/
36a04427db Added a comment 2014-05-16 19:11:02 +00:00
http://joeyh.name/
185413e77b Added a comment 2014-05-16 18:19:22 +00:00
Robie Basak
4184566627 ddar special remote 2014-05-15 16:32:44 -04:00
https://www.google.com/accounts/o8/id?id=AItOawnq-RfkVpFN15SWvQ2lpSGAi0XpNQuLxKM
68a62b337d Added a comment: re: Change Password 2014-05-04 19:09:56 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnmF_9CAtfqdZkC4e-_dCX-rK5bqh4RWkw
e232f59375 Added a comment: Change Password 2014-05-02 12:31:02 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo
d2b72686c8 Added a comment: Enhancing special remotes as backup spaces 2014-04-22 21:08:49 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
cdc3874c56 removed 2014-04-07 11:32:46 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
1705215388 Added a comment: I can't add special remotes 2014-04-07 11:29:28 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
53f94bbcfc removed 2014-04-07 11:27:57 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
bdea0964e2 removed 2014-04-07 11:26:18 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
3db1ca5d57 Added a comment: I can't add special remotes 2014-04-07 11:25:46 +00:00
https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw
eefaa6eaab Added a comment: I can't add special remotes 2014-04-07 11:25:34 +00:00
Joey Hess
2940fa1ba2 respond; improve docs 2014-03-26 13:11:49 -04:00
Joey Hess
18c98c8cfa improve directory special remote docs 2014-02-02 15:26:33 -04:00
https://www.google.com/accounts/o8/id?id=AItOawm-19QERTsjJSB-m6dWvOa1w9SOxyhFqDM
a0cacffebf 2014-01-29 17:35:17 +00:00
Joey Hess
aafef31ecd format 2014-01-08 16:33:23 -04:00
Joey Hess
85272d8a98 Added tahoe special remote. 
Known problems:

1. Tries to tahoe start when daemon is already running.

2. If multiple tahoe remotes are set up on the same computer,
   they will have the same node.url configured by default,
   and this confuses tahoe commands.

This commit was sponsored by LeastAuthority.com
 2014-01-08 16:14:41 -04:00
Joey Hess
054e4f17e2 implement PREPARE-FAILURE for Tobias 2013-12-29 13:39:25 -04:00
https://www.google.com/accounts/o8/id?id=AItOawm0yDI6AsAuXPDHIDXm5H9EGDXkuwb1efA
bd1bdcb619 Added a comment: XMPP not working in Raspberry PI build 2013-12-27 23:16:38 +00:00
Joey Hess
445b7b41b9 add credential storage support for external special remotes & update example 2013-12-27 16:01:43 -04:00
Joey Hess
6de3bc2ab6 make directory absolute (mostly as an example of using SETCONFIG) 2013-12-27 14:30:00 -04:00
Joey Hess
dc7681b2bb emphasize link to protocol 2013-12-27 14:15:35 -04:00
Joey Hess
9b944e71e1 fix link 2013-12-27 14:15:01 -04:00
Joey Hess
5972ee50fa external special remotes working 2013-12-27 14:13:52 -04:00
Joey Hess
4d94e40ddd de-verbose cp 2013-12-27 14:06:56 -04:00
Joey Hess
e9cb88ca38 another bug in shell code. haskell has ruined me 2013-12-27 14:06:33 -04:00
Joey Hess
53b0228077 fix 3 more bugs in these measly 150 lines of shell code. sheesh 2013-12-27 14:04:51 -04:00
Joey Hess
5d8ff64dc1 make --debug show transcript of special remote protocol messages 2013-12-27 03:10:00 -04:00
Joey Hess
526a7bb2b4 improve comments 2013-12-27 02:56:52 -04:00
Joey Hess
5b7c38c90a improve code, seems to work well now 2013-12-27 02:49:00 -04:00
Joey Hess
5d00fa523c fix bug (no type safe protocol stuff here, alas..) 2013-12-27 02:12:33 -04:00
Joey Hess
6d504b57e7 make some requests optional, simplify and future-proof protocol more 2013-12-27 02:11:06 -04:00
Joey Hess
cfcc1c13a0 fix format of sample program 2013-12-26 18:36:57 -04:00
Joey Hess
409a85b264 update 2013-12-26 18:20:32 -04:00
Joey Hess
0de9135bc0 external special remote documentation and example script 2013-12-26 18:15:18 -04:00
https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8
736ce5e669 Google Apps just needs good SRV records 2013-11-27 18:18:12 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8
279416939b Added a comment: Google Apps & SRV 2013-11-27 18:13:18 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk
dc18398f7d Added a comment: Expected cost for a repository 2013-11-24 22:32:16 +00:00
http://joeyh.name/
4ad39330a3 Added a comment 2013-11-24 15:58:31 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg
f3d12e7449 Added a comment: Testing a special remote 2013-11-24 08:24:38 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlu-fdXIt_RF9ggvg4zP0yBbtjWQwHAMS4
56470fe678 Added a comment: Same problem, no XMPP showing up in daemon.log 2013-11-21 21:13:17 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY
ef57802c03 Added a comment: I know this thread is old but... 2013-11-18 00:27:51 +00:00
Joey Hess
8b5de8588d point at my fork of gcrypt, which is currently needed 2013-11-05 21:28:56 -04:00
https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U
2d32f6986f Added a comment 2013-10-22 14:30:58 +00:00
RaspberryPie
2dd2dc01b6 Added a comment 2013-09-26 03:46:18 +00:00
http://joeyh.name/
76b7f10cfc Added a comment 2013-09-25 18:13:24 +00:00
RaspberryPie
c867d3fa1a Added a comment: Nope 2013-09-24 22:05:56 +00:00
Joey Hess
4c954661a1 git-annex-shell: Added support for operating inside gcrypt repositories. 
* Note that the layout of gcrypt repositories has changed, and
  if you created one you must manually upgrade it.
  See http://git-annex.branchable.com/upgrades/gcrypt/
 2013-09-24 17:25:47 -04:00
http://joeyh.name/
475270f401 Added a comment 2013-09-19 21:07:35 +00:00
RaspberryPie
bcb94a308a Added a comment: Missing prerequisites for XMPP syncing? 2013-09-17 06:53:59 +00:00
Joey Hess
b0356e56c1 add gcrypt tip 2013-09-08 15:48:41 -04:00
Joey Hess
3e079cdcd1 gcrypt: now supports rsync 
Use rsync for gcrypt remotes that are not local to the disk.
(Note that I have punted on supporting http transport for now, it doesn't
seem likely to be very useful.)

This was mostly quite easy, it just uses the rsync special remote to handle
the transfers. The git repository url is converted to a RsyncOptions
structure, which required parsing it separately, since the rsync special
remote only supports rsync urls, which use a different format.

Note that annexed objects are now stored at the top of the gcrypt repo,
rather than inside annex/objects. This simplified the rsync suport,
since it doesn't have to arrange to create that directory. And git-annex
is not going to be run directly within gcrypt repos -- or if in some
strance scenario it was, it would make sense for it to not see the
encrypted objects.

This commit was sponsored by Sheila Miguez
 2013-09-08 14:54:28 -04:00
Joey Hess
c56f71ab71 update 2013-09-07 19:25:13 -04:00
Joey Hess
0439dd1b5c page for gcrypt 2013-09-07 19:08:34 -04:00
Joey Hess
47d0219ac8 update docs for changed initremote encryption syntax 2013-09-04 23:46:50 -04:00
http://joeyh.name/
ef19c88162 Added a comment 2013-08-26 18:36:03 +00:00
http://cstork.org/
08489ec560 Added a comment 2013-08-25 20:59:42 +00:00
http://joeyh.name/
2883ff9220 Added a comment 2013-08-23 17:39:56 +00:00
https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio
4eec2bc738 Added a comment: different s3 storage URLs 2013-08-23 08:59:33 +00:00
http://olivier.berger.myopenid.com/
f4e7da8f5c Added a comment: Added use cases to "special remotes" 2013-08-22 20:23:13 +00:00
http://joeyh.name/
7dd949ec90 Added a comment 2013-08-22 18:02:00 +00:00
http://olivier.berger.myopenid.com/
a248224c4f Added a comment: rsync remote is basically more intended for backups ? 2013-08-17 17:40:48 +00:00
http://joeyh.name/
a14c8cec75 Added a comment 2013-08-17 08:59:11 +00:00
http://olivier.berger.myopenid.com/
ff8b633c7f Added a comment: Which URL prefix are supported ? 2013-08-17 08:44:07 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8
244b14d14d 2013-08-15 17:28:59 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8
1b4da07d67 2013-08-15 16:02:59 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8
16d775d4f4 Removing: Facebook requires no TLS, so extremely unsafe. LoL requires SSL. 2013-08-15 14:41:20 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8
a4469b9157 2013-08-15 14:03:51 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
f5bfd549f4 order 2013-08-15 11:41:59 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
a8db8f4398 2013-08-15 11:41:37 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
0f56380995 2013-08-15 08:40:12 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
885d0d0059 2013-08-15 08:28:02 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
5acda0b0d6 2013-08-15 08:06:20 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
2a440fe3cf 2013-08-15 08:01:12 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
a2498d50bc 2013-08-15 08:00:32 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA
4e6ee9c7f8 2013-08-15 07:59:41 +00:00
Joey Hess
c55799c957 add note that this is not what's wanted if two trees with files visible is the desired use case 2013-08-11 14:21:19 +02:00
http://joeyh.name/
c88bbc47ce Added a comment 2013-08-01 23:51:48 +00:00