From 6e3bcbf4dd769ac907aeeb5d2653d4e8ff3b7322 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 17 Nov 2023 17:27:58 -0400 Subject: [PATCH] Make git-annex copy --from --to --fast actually fast Eg when the destination is logged as containing a file, skip actively checking that it does contain it. Note that --fast does not prevent other verifications of content location that are done in a copy --from --to. Perhaps it could, but this change will already avoid the real unnecessary work of operating on files that are already in the remote. And avoiding other verifications might cause it to fail if the location log thinks that --to does not contain the content but does. Such complications with `git-annex copy --to remote --fast` led to commit d006586cd0b706c9cc92b2747b2ba3487f52c04a which added a note that gets displayed when that fails, mentioning it might be due to --fast being enabled. copy --from --to is already complicated enough without needing to worry about such edge cases, so continuing to doing some verification of content location after the initial --fast filtering seems ok. Sponsored-by: Dartmouth College's DANDI project --- CHANGELOG | 1 + Command/Move.hs | 8 +++++++- ...--fast_--from_--to_checks_destination_files.mdwn | 2 ++ ...ment_1_2a2997e6f914afb0477f2baa69b174fc._comment | 13 +++++++++++++ ...ment_2_f7bf30e8cc2d1995976bde723dfbfe01._comment | 7 +++++++ 5 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_1_2a2997e6f914afb0477f2baa69b174fc._comment create mode 100644 doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_2_f7bf30e8cc2d1995976bde723dfbfe01._comment diff --git a/CHANGELOG b/CHANGELOG index f5f13c3f88..4ca82dcea0 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -19,6 +19,7 @@ git-annex (10.20230927) UNRELEASED; urgency=medium to tune the output of the above added options. * Fix bug in git-annex copy --from --to that skipped files that were locally present. + * Make git-annex copy --from --to --fast actually fast. -- Joey Hess Tue, 10 Oct 2023 13:17:31 -0400 diff --git a/Command/Move.hs b/Command/Move.hs index 3a7dc4859f..2e450b3e4a 100644 --- a/Command/Move.hs +++ b/Command/Move.hs @@ -345,7 +345,13 @@ fromToStart removewhen afile key ai si src dest = starting (describeMoveAction removewhen) (OnlyActionOn key ai) si $ fromToPerform src dest removewhen key afile where - somethingtodo = pure (Remote.uuid src /= Remote.uuid dest) + somethingtodo + | Remote.uuid src == Remote.uuid dest = return False + | otherwise = do + fast <- Annex.getRead Annex.fast + if fast && removewhen == RemoveNever + then not <$> expectedPresent dest key + else return True {- When there is a local copy, transfer it to the dest, and drop from the src. - diff --git a/doc/bugs/copy_--fast_--from_--to_checks_destination_files.mdwn b/doc/bugs/copy_--fast_--from_--to_checks_destination_files.mdwn index 3c140823c6..64cf7c1025 100644 --- a/doc/bugs/copy_--fast_--from_--to_checks_destination_files.mdwn +++ b/doc/bugs/copy_--fast_--from_--to_checks_destination_files.mdwn @@ -84,3 +84,5 @@ and then in conda with `10.20230626-g801c4b7` [[!meta author=yoh]] [[!tag projects/dandi]] + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_1_2a2997e6f914afb0477f2baa69b174fc._comment b/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_1_2a2997e6f914afb0477f2baa69b174fc._comment new file mode 100644 index 0000000000..5682b0cc8a --- /dev/null +++ b/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_1_2a2997e6f914afb0477f2baa69b174fc._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2023-11-17T20:57:19Z" + content=""" +> but it doesn't works out correctly whenever there are some files to actually copy + +I think that was due to the bug you linked, which is now fixed. + +I've confirmed that `--fast` is not actually implemented for `git-annex +copy --from --to`. Explicitly specifying `--not --in destremote` is a +fine workaround. But I've gone ahead and implemented `--fast` for it too. +"""]] diff --git a/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_2_f7bf30e8cc2d1995976bde723dfbfe01._comment b/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_2_f7bf30e8cc2d1995976bde723dfbfe01._comment new file mode 100644 index 0000000000..a12cc7c372 --- /dev/null +++ b/doc/bugs/copy_--fast_--from_--to_checks_destination_files/comment_2_f7bf30e8cc2d1995976bde723dfbfe01._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2023-11-17T21:16:35Z" + content=""" +BTW `git-annex find --print0` is the output eqivilant of -z. +"""]]