incremental verify for chunked remotes

Simply feed each chunk in turn to the incremental verifier.

When resuming an interrupted retrieve, it does not do incremental
verification. That would need to read the file, up to the resume point,
and feed it to the incremental verifier. That seems easy to get wrong.
Also it would mean extra work done before the transfer can start. Which
would complicate displaying progress, and would perhaps not appear to the
user as if it was resuming from where it left off. Instead, in that
situation, return UnVerified, and let the verification be done in a
separate pass.

Granted, Annex.CopyFile does manage all that, but it's not complicated
by dealing with chunks too.

Sponsored-by: Dartmouth College's DANDI project
This commit is contained in:
Joey Hess 2021-08-11 14:42:49 -04:00
parent c20358b671
commit 7eb3742e4b
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
3 changed files with 47 additions and 30 deletions

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="joey"
subject="""comment 13"""
date="2021-08-11T18:16:41Z"
content="""
Some special remotes now support incremental update. So far, limited to
ones that use the byteRetriever interface.
Others, that use fileRetriever, including external special remotes, still
need work.
"""]]