Commit graph

43 commits

Author SHA1 Message Date
Joey Hess
0f7143d226
support VURL backend
Not yet implemented is recording hashes on download from web and
verifying hashes.

addurl --verifiable option added with -V short option because I
expect a lot of people will want to use this.

It seems likely that --verifiable will become the default eventually,
and possibly rather soon. While old git-annex versions don't support
VURL, that doesn't prevent using them with keys that use VURL. Of
course, they won't verify the content on transfer, and fsck will warn
that it doesn't know about VURL. So there's not much problem with
starting to use VURL even when interoperating with old versions.

Sponsored-by: Joshua Antonishen on Patreon
2024-02-29 13:48:51 -04:00
Joey Hess
b813b132da
fix typo 2022-12-26 12:36:56 -04:00
Joey Hess
dcc2957d9c
improve documentation about backends
I noticed that, using just the man pages, there is no real description
of what backends are, or what ones are available. Except for some
examples.

Added a git-annex-backends man page, that is just a stub, but at least
describes what they basically are, and tells how to find the supported
ons, and links to the backends web page.

Sponsored-by: Brett Eisenberg on Patreon
2022-09-26 15:59:10 -04:00
Joey Hess
fc61915230
use GIT keys for export of non-annexed files
This solves the problem that import of such files gets confused and
converts them back to annexed files.

The import code already used GIT keys internally when it determined a
file should not be annexed. So now when it sees a GIT key that export
used, it already does the right thing.

This also means that even older version of git-annex can import and will
do the right thing, once a fixed version has exported. Still, there may
be other complications around upgrades; still need to think it all
through.

Moved gitShaKey and keyGitSha from Key to Annex.Export since they're
only used for export/import.

Documented GIT keys in backends, since they do appear in the git-annex
branch now.

This commit was sponsored by Graham Spencer on Patreon.
2021-03-05 14:12:11 -04:00
Joey Hess
ea63d1dfe3
E variant of external backend keys 2020-07-29 17:12:22 -04:00
Joey Hess
f75be32166
external backends wip
It's able to start them up, the only thing not implemented is generating
and verifying keys. And, the key translation for HasExt.
2020-07-29 15:23:18 -04:00
Ilya_Shlyakhter
d4b700196b added a hyperlink to key_format 2019-11-30 17:00:58 +00:00
Joey Hess
0c6b7e288d
Add BLAKE2BP512 and BLAKE2BP512E backends
using a blake2 variant optimised for 4-way CPUs

This had been deferred because the Debian package of cryptonite, and
possibly other builds, was broken for blake2bp, but I've confirmed #892855
is fixed.

This commit was sponsored by Brett Eisenberg on Patreon.
2019-07-05 15:30:03 -04:00
Ilya_Shlyakhter
209184a798 added details about the URL backend 2018-10-12 20:28:25 +00:00
Joey Hess
e0b1e0ef7b
comment and clarification 2018-09-11 12:55:31 -04:00
Joey Hess
47cd6923b4
mention new limitation 2018-06-22 10:30:10 -04:00
Joey Hess
050ada746f
Added backends for the BLAKE2 family of hashes.
There are a lot of different variants and sizes, I suppose we might as well
export all the common ones.

Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support.
Even android has 0.16 or newer.

On Debian, Blake2bp_512 is buggy, so I have omitted it for now.
http://bugs.debian.org/892855

This commit was sponsored by andrea rota.
2018-03-13 16:23:42 -04:00
Joey Hess
4c1e3210fa
annex.backend is the new name for what was annex.backends
It takes a single key-value backend, rather than the unncessary and confusing list.
The old option still works if set.

Simplified some old old code too.

This commit was sponsored by Thomas Hochstein on Patreon.
2017-05-09 15:04:07 -04:00
Joey Hess
622b3fface
devblog 2017-02-24 20:03:36 -04:00
Joey Hess
1f0d0ab4b3
Revert "pointer to a todo"
This reverts commit ae3f6705eb.

todo is not ready yet
2017-02-24 15:40:28 -04:00
Joey Hess
ae3f6705eb
pointer to a todo 2017-02-24 13:41:29 -04:00
Joey Hess
6346704a04
clarify that annex.backends is used when adding new files
Even if annex.backends does not include a backend, that does not prevent
git-annex commands from acting on a file using the missing backend.

(There's really no reason at all for annex.backends to be a list.)
2017-02-24 11:53:59 -04:00
Øyvind A. Holm
5c0c39115c backends.mdwn: Typo fix, 'SHA_512E' → 'SHA3_512E'
Really awesome that git-annex got SHA3 support the day after the
standard was released.
2015-08-13 04:19:11 +02:00
Joey Hess
0ec9bc2200 Added support for SHA3 hashed keys (in 8 varieties), when git-annex is built using the cryptonite library.
While cryptohash has SHA3 support, it has not been updated for the final
version of the spec. Note that cryptonite has not been ported to all arches
that cryptohash builds on yet.
2015-08-06 15:02:25 -04:00
Joey Hess
8eb01bc894 Added MD5 and MD5E backends. 2015-02-04 13:47:54 -04:00
Joey Hess
b84070b5e9 note about windows path length issues with larger hashes 2015-01-06 22:33:57 -04:00
Joey Hess
9711d529c8 WORM backend: Switched to include the relative path to the file inside the repository, rather than just the file's base name. Note that if you're relying on such things to keep files separate with WORM, you should really be using a better backend. 2014-09-11 14:50:18 -04:00
Joey Hess
bfdbc63b50 mention SKEIN*E variants 2014-07-16 13:31:01 -04:00
Joey Hess
a05b763b01 Added SKEIN256 and SKEIN512 backends
SHA3 is still waiting for final standardization.
Although this is looking less likely given
https://www.cdt.org/blogs/joseph-lorenzo-hall/2409-nist-sha-3

In the meantime, cryptohash implements skein, and it's used by some of the
haskell ecosystem (for yesod sessions, IIRC), so this implementation is
likely to continue working. Also, I've talked with the cryprohash author
and he's a reasonable guy.

It makes sense to have an alternate high security hash, in case some
horrible attack is found against SHA2 tomorrow, or in case SHA3 comes out
and worst fears are realized.

I'd also like to support using skein for HMAC. But no hurry there and
a new version of cryptohash has much nicer HMAC code, so I will probably
wait until I can use that version.
2013-10-01 20:34:36 -04:00
https://www.google.com/accounts/o8/id?id=AItOawkbxlvsA1U_YBe0iNy6_7mhMfgdEp-VUvM
b56c9f6729 Typo: double 'the' 2013-08-18 11:47:39 +00:00
Joey Hess
5af1190ef1 make edits that a commenter apparently didn't realize they could make. this is a wiki.. 2013-07-31 12:31:30 -04:00
Joey Hess
3724344461 SHA256E is new default backend
The default backend used when adding files to the annex is changed from
SHA256 to SHA256E, to simplify interoperability with OSX, media players,
and various programs that needlessly look at symlink targets.

To get old behavior, add a .gitattributes containing: * annex.backend=SHA256
2012-09-12 13:22:16 -04:00
Joey Hess
ef3457196a use SHA256 by default
To get old behavior, add a .gitattributes containing: * annex.backend=WORM

I feel that SHA256 is a better default for most people, as long as their
systems are fast enough that checksumming their files isn't a problem.
git-annex should default to preserving the integrity of data as well as git
does. Checksum backends also work better with editing files via
unlock/lock.

I considered just using SHA1, but since that hash is believed to be somewhat
near to being broken, and git-annex deals with large files which would be a
perfect exploit medium, I decided to go to a SHA-2 hash.

SHA512 is annoyingly long when displayed, and git-annex displays it in a
few places (and notably it is shown in ls -l), so I picked the shorter
hash. Considered SHA224 as it's even shorter, but feel it's a bit weird.

I expect git-annex will use SHA-3 at some point in the future, but
probably not soon!

Note that systems without a sha256sum (or sha256) program will fall back to
defaulting to SHA1.
2011-11-04 15:51:01 -04:00
Joey Hess
025e66e3d3 update to not overstate the danger or WORM 2011-08-29 12:11:05 -04:00
Joey Hess
bbba6c19bd update documentation for new, neutered key-value backends
Backends are now only used to generate keys (and check them); they
are not arbitrary key-value stores for data, because it turned out such
a store is better modeled as a special remote. Updated docs to not
imply backends do more than they do now.

Sometimes I'm tempted to rename "backend" to "keytype" or something,
which would really be more clear. But it would be an annoying transition
for users, with annex.backends etc.
2011-08-28 16:28:38 -04:00
Joey Hess
fb58d1a560 wording 2011-07-01 17:17:51 -04:00
Joey Hess
6ba866ca73 updates for web remote and removing URL backend 2011-07-01 15:39:30 -04:00
Joey Hess
2a8efc7af1 Added filename extension preserving variant backends SHA1E, SHA256E, etc. 2011-05-16 11:46:34 -04:00
Joey Hess
c0fd38bfa9 document S3 remotes 2011-03-27 22:52:13 -04:00
Joey Hess
f8693facab doc update 2011-03-27 17:30:44 -04:00
Joey Hess
1b9c4477fb New backends: SHA512 SHA384 SHA256 SHA224 2011-03-01 17:07:15 -04:00
Joey Hess
ca32c7859b The git-annex-backend attribute has been renamed to annex.backend. 2010-11-28 18:58:03 -04:00
Joey Hess
75d2925082 Merge branch 'master' into checkout
Conflicts:
	debian/changelog
	doc/backends.mdwn
2010-11-09 15:05:08 -04:00
Joey Hess
02a21d7f27 reorg .git/annex 2010-11-08 15:14:54 -04:00
Joey Hess
377bf24d9a documentation for checkout 2010-11-07 21:02:25 -04:00
Joey Hess
f1f4bdcd60 expand docs 2010-11-01 19:11:06 -04:00
Joey Hess
46f9525351 warning about sha1 2010-10-27 15:02:05 -04:00
Joey Hess
7bc4435ffd update 2010-10-19 15:59:40 -04:00