diff --git a/doc/todo/OPT__58_____34__bundle__34___get_+_check___40__of_checksum__41___in_a_single_operation/comment_4_82707d36ed580d62c73e441f959ccd9e._comment b/doc/todo/OPT__58_____34__bundle__34___get_+_check___40__of_checksum__41___in_a_single_operation/comment_4_82707d36ed580d62c73e441f959ccd9e._comment new file mode 100644 index 0000000000..b5fd006951 --- /dev/null +++ b/doc/todo/OPT__58_____34__bundle__34___get_+_check___40__of_checksum__41___in_a_single_operation/comment_4_82707d36ed580d62c73e441f959ccd9e._comment @@ -0,0 +1,41 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2019-12-30T18:15:40Z" + content=""" +When -J is used, recent versions use a separate pool of worker threads for +the checksumming than the downloading. So even with -J1 checksum of the +previous download will not block the next download. + +I've thought about making this the default without -J.. It relies on +concurrent-output working well, which it sometimes may not, eg when +filenames are not valid unicode, or perhaps on a non-ANSI terminal, and so +far it's been worth not defaulting to -J1 to avoid breaking in such edge +cases. + +Anyway, it seems to me using -J should avoid most of the overhead, except +of course for the remaining checksumming after all downloads finish. + +Incremental checksumming could be done for some of the built-in remotes, +but not others like bittorrent which write out of order. Some transfers +can resume, and the checksumming would have to somehow catch +up to resume point, which adds significant complexity. + +External remotes would need to send the content over a pipe for incremental +checksumming, so it would need a protocol extension. + +git-annex's remote API does have the concept that a remote can sufficiently +verify the content of a file during transfer that additional checksumming +is not necessary. Currently only used for git remotes when hard linking an +object from a sibling remote. I don't think it actually matters what +checksum a remote uses to do such verification, as long as it's +cryptographically secure and runs on the local machine. + +A protocol extension that let an external remote communicate to git-annex +that it had done such verification at the end of transfer is worth thinking +about. + +Re Ilya's security concerns, as long as the external remote runs the +verification on the local machine, it seems there is no added security +impact. +"""]]