support VURL backend

Not yet implemented is recording hashes on download from web and
verifying hashes.

addurl --verifiable option added with -V short option because I
expect a lot of people will want to use this.

It seems likely that --verifiable will become the default eventually,
and possibly rather soon. While old git-annex versions don't support
VURL, that doesn't prevent using them with keys that use VURL. Of
course, they won't verify the content on transfer, and fsck will warn
that it doesn't know about VURL. So there's not much problem with
starting to use VURL even when interoperating with old versions.

Sponsored-by: Joshua Antonishen on Patreon

doc/backends.mdwn

@@ -54,6 +54,10 @@ in `.gitattributes`:
   `BLAKE2SP224E`, `BLAKE2SP256E`
   -- Fast [Blake2 hash](https://blake2.net/) variants optimised for 
   8-way CPUs.
+  `VURL` -- This is like an `URL` (see below) but the content can
+  be verified with a cryptographically secure checksum that is
+  recorded in the git-annex branch. It's generated when using 
+  eg `git-annex addurl --fast --verifiable`.
 
 ## non-cryptograpgically secure backends
 
@@ -68,10 +72,11 @@ content of an annexed file remains unchanged.
   files or slow systems.
 * `URL` -- This is a key that is generated from the url to a file.
   It's generated when using eg, `git annex addurl --fast`, when the file
-  content is not available for hashing.  The key may not contain the full
-  URL; for long URLs, part of the URL may be represented by a checksum.
+  content is not available for hashing.  
+  The key may not contain the full URL; for long URLs, part of the URL may be
+  represented by a checksum.  
   The URL key may contain `&` characters; be sure to quote the key if
-  passing it to a shell script.  The URL-backend key is distinct from URLs/URIs
+  passing it to a shell script. These types of keys are distinct from URLs/URIs
   that may be attached to a key (using any backend) indicating the key's location
   on the web or in one of [[special_remotes]].
 * `GIT` -- This is used internally by git-annex when exporting trees

doc/git-annex-addurl.mdwn

@@ -42,7 +42,24 @@ be used to get better filenames.
   This is the fastest option, but it still has to access the network
   to check if the url contains embedded media. When adding large numbers
   of urls, using `--relaxed --raw` is much faster.
-  
+
+* `--verifiable` `-V`
+
+  This can be used with the `--fast` or `--relaxed` option. It improves
+  the safety of the resulting annexed file, by letting its content be
+  verified with a checksum when it is transferred between git-annex
+  repositories, as well as by things like `git-annex fsck`.
+
+  When used with --relaxed, content from the web will always be accepted,
+  even if it has changed, and the checksum recorded for later verification.
+
+  When used with --fast, the checksum is recorded the first time the
+  content is downloaded from the web. Once a checksum has been recorded,
+  subsequent downloads from the web must have the same checksum.
+
+  Note that this option currently only has an effect when using the
+  web special remote, not other special remotes that handle urls.
+
 * `--raw`
 
   Prevent special handling of urls by yt-dlp, and by bittorrent

doc/git-annex-importfeed.mdwn

@@ -37,7 +37,7 @@ resulting in the new url being downloaded to such a filename.
 
   Force downloading items it's seen before.
 
-* `--relaxed`, `--fast`, `--raw`, `--raw-except`
+* `--fast`, `--relaxed`, `--verifiable`, `--raw`, `--raw-except`
 
   These options behave the same as when using [[git-annex-addurl]](1).
 

doc/todo/verified_relaxed_urls.mdwn

@@ -24,23 +24,46 @@ the web with its hash recorded, but has since gotten corrupted.
 
 Seems that the only possible way to resolve this problem is to change to a
 new type of url key, that is known to always have its hash recorded on
-download from the web. (Call this a "dynamic" url key.)
+download from the web. (Call this a verifiable url key: a VURL.)
 And handle all existing relaxed url keys as before.
 
-That would leave it up to the user to migrate their relaxed url keys to
-dynamic urls keys, if desired. Now that distributed migration is
+That would leave it up to the user to migrate their URL keys to
+VURL keys, if desired. Now that distributed migration is
 implemented, that seems sufficiently easy.
 
+## addurl --fast 
+
+Using addurl --fast rather than --relaxed records the size but doesn't
+hash. So it has the same problem that data corruption can go unnoticed,
+only the data corruption has to involve bit flips and not truncation.
+
+So it seems that --fast ought to also be handled. The difference being that
+an url added with --fast is expected to always be the same on re-download
+from the web, while an url added with --relaxed may change its content on
+re-download from the web while being still considered the same object.
+
+This can also use a VURL key, but include the size in it. When downloading
+a sized VURL, the web special remote will hash the content, and verify
+that either no hash has been recorded before (and record the hash when the
+size matches), or that it matches the previously recorded hash.
+
+Note that, if an url is added with --fast and that gets committed and
+pulled by another repo, and then later both repos download the content
+from the web, it would be possible for the web to serve up different
+content to the two, and in that case either hash would be treated as
+valid.
+
 ## other special remotes
 
 If the web special remote is what takes care of hashing the content on
 download and recording the hash-based key, what about other special remotes
 that claim an url?
 
-This could also be implemented in the bittorrent special remote, but what
+This could also be implemented in the bittorrent special remote
+(though ), but what
 about external special remotes?
 
-An alternative would be to add a downloadDynamicUrl that is called instead
+An alternative would be to add a downloadVerifiedUrl that is called instead
 of retrieveKeyFile and returns a hash-based key (allowing hashing the
 download on the fly). Then git-annex would take
 care of recording the hash-based key. The external special remote interface
@@ -50,10 +73,26 @@ could be extended to include that.
 
 Should annex.backend gitconfig be used to pick which hash-based key to use?
 The risk is that config changes and several different hash-based keys get
-recorded for a dynamic url. Not really a problem, but would increase the
+recorded for a VURL. Not really a problem, but would increase the
 size of the git-annex branch unncessarily, and require extra work when
 verifying the key.)
 
+What if annex.backend uses WORM or something that is not hash-based?
+Seems it ought to fall back to SHA256 or something then.
+
+To support annex.securehashesonly it would be good if only
+cryptographically secure hashes were recorded for a VURL. But of course,
+which hashes are considered secure can change. Still, let's start by
+only allowing currently secure hashes to be used for VURLs. This way,
+when there are multiple hashes recorded for a VURL, they will all be
+cryptographically secure, and so the VURL can have
+`isCryptographicallySecure = True`. If any of the hashes later becomes
+broken, the VURL will no longer be treated as cryptographically secure,
+because the broken hash can be used to verify its content.
+In that case, the user would probably just migrate to a hash-based key,
+although perhaps something VURL-specific could be built to upgrade its
+hashes.
+
 ## use for other types of keys
 
 It would also be possible to use these new git-annex branch log files