diff --git a/Backend/VURL.hs b/Backend/VURL.hs index f1dfe38a4a..2552d08bfc 100644 --- a/Backend/VURL.hs +++ b/Backend/VURL.hs @@ -31,8 +31,12 @@ backendVURL = Backend -- Normally there will always be an key -- recorded when a VURL's content is available, -- because downloading the content from the web in - -- the first place records one. - [] -> return False + -- the first place records one. But, when the + -- content is downloaded from some other special + -- remote that claims an url, that might not be the + -- case. So, default to True when no key is + -- recorded. + [] -> return True eks -> do let check ek = getbackend ek >>= \case Nothing -> pure False diff --git a/CHANGELOG b/CHANGELOG index e6a89262c3..cdef9716d2 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -3,6 +3,8 @@ git-annex (10.20250103) UNRELEASED; urgency=medium * Improve handing of ssh connection problems during remote annex.uuid discovery. * log: Support --key, as well as --branch and --unused. + * Avoid verification error when addurl --verifiable is used + with an url claimed by a special remote other than the web. -- Joey Hess Fri, 03 Jan 2025 14:30:38 -0400 diff --git a/doc/bugs/VURL_verification_failure_on_first_download.mdwn b/doc/bugs/VURL_verification_failure_on_first_download.mdwn index 7cadb61389..5d37eae73d 100644 --- a/doc/bugs/VURL_verification_failure_on_first_download.mdwn +++ b/doc/bugs/VURL_verification_failure_on_first_download.mdwn @@ -1,3 +1,5 @@ +[[!meta title="VURL equivilant keys not populated when addurl is used with special remotes other than web"]] + ### Please describe the problem. With an external special remote that handles a custom URL scheme, I receive a "Verification of content failed" on the first `git annex get` of a file (i.e. when git-annex cannot know a checksum for the file, yet). diff --git a/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment b/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment new file mode 100644 index 0000000000..f1e146d176 --- /dev/null +++ b/doc/bugs/VURL_verification_failure_on_first_download/comment_1_fe5525d4c03fdafc28dc23ed5ee3fab8._comment @@ -0,0 +1,49 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2025-01-03T19:52:28Z" + content=""" +Looking at [[!commit 55bf01b78888b410a8ac07e834ed7104ffa1f4d0]] it talks +about something like this: + + When downloading a VURL from the web, make sure that the equivilant key + log is populated. + + ... + + Downloads of VURL keys from other special remotes that claim urls, + like bittorrent for example, does not popilate the equivilant key log. + So for now, no checksum verification will be done for those. + +I think at the time, I just punted on the question of how to register the +equivilant key when downloading a VURL from other special remotes. + +Thing is, git-annex just calls retrieveKeyFile at some point. At that +point, it doesn't know if it's retrieving the key from the web or some +other special remote that claims urls. + +It would not do for a `git-annex get` that happens to get the VURL key from +eg, a directory special remote to say hey, we've not populated the +equivilant key log for this VURL yet, let's trust what we downloaded from +the remote is good and populate it now. + +So that's why the code went into the web special remote. It seems to me that to +move it out of the web special remote, there would need to be a way for +git-annex to check if a given special remote claims the url associated with a +VURL. Eg, look up the key's url(s) and check claimUrl. + +Hmm, if it did that, there would still be a problem that retrieveKeyFile can +try to verify the content it downloads, using verifyKeyContent. But when no +equivilant key is yet registered for the VURL, retrieveKeyFile would find that +anything it downloaded failed to verify. So it would fail. Chicken and egg problem. + +But well, it would be easy enough to make this not be treated as a failure. +In Backend.VURL, verifyKeyContent returns False when there are no equivilant +keys. It could just return true. + +That seems like a necessary first step. Just making that change will solve your +problem to the extent that it's ok for no verification of the content of VURLs +to be done when they originate with your special remote. + +So I've done that. Whether it makes sense to do the rest I am not yet sure.. +"""]] diff --git a/doc/git-annex-addurl.mdwn b/doc/git-annex-addurl.mdwn index 769bed69a7..c2247e1de9 100644 --- a/doc/git-annex-addurl.mdwn +++ b/doc/git-annex-addurl.mdwn @@ -50,12 +50,14 @@ be used to get better filenames. verified with a checksum when it is transferred between git-annex repositories, as well as by things like `git-annex fsck`. - When used with --relaxed, content from the web will always be accepted, - even if it has changed, and the checksum recorded for later verification. + When used with --relaxed, content from the web special remote will + always be accepted, even if it has changed, and the checksum recorded + for later verification. When used with --fast, the checksum is recorded the first time the - content is downloaded from the web. Once a checksum has been recorded, - subsequent downloads from the web must have the same checksum. + content is downloaded from the web special remote. Once a checksum has + been recorded, subsequent downloads from the web special remote + must have the same checksum. When addurl was used without this option before, the file it added can be converted to be verifiable by migrating it to the VURL backend. diff --git a/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment b/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment new file mode 100644 index 0000000000..277be82ad4 --- /dev/null +++ b/doc/todo/migration_to_VURL_by_default/comment_2_83b03035f1cae24180038f412e1886f2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2025-01-03T21:17:31Z" + content=""" +I think that is no longer a blocker for this becoming default, it no longer +causes a verification failure, instead the content originating with such a +special remote for a VURL is effectively treated the same as for an URL, +and just doesn't get verified in transit later on. +"""]]