From d5aab420b5b2380e14e8c84b28fbae3b2787fa96 Mon Sep 17 00:00:00 2001 From: wting Date: Fri, 13 Nov 2020 06:20:27 +0000 Subject: [PATCH 1/6] --- ...ates_numcopies___40__regression__41__.mdwn | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 doc/bugs/move_violates_numcopies___40__regression__41__.mdwn diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn b/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn new file mode 100644 index 0000000000..757e1803ba --- /dev/null +++ b/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn @@ -0,0 +1,56 @@ +### Please describe the problem. + +Using `git annex move` will reduce the number of copies below `numcopies`. + +Relevant links: + +- [original bug report](https://git-annex.branchable.com/bugs/move_violates_numcopies/) +- [devblog day 496: move numcopies safety revisited](https://git-annex.branchable.com/devblog/day_496__move_numcopies_safety_revisited/) + +### What steps will reproduce the problem? + + ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› + ╰─ git annex whereis + whereis 1403415.jpg (2 copies) + 766c9221-1404-4d68-bfa6-46d064bf4798 -- ting@ringo:/Volumes/apple/backup/annex/test-drop [apple] + 9cfda232-8504-4c3b-b694-4a6c85dc6c9c -- ting@ringo:~/backup/annex/test-drop [ringo] [here] + ok + ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› + ╰─ g annex numcopies + 2 + ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› + ╰─ g annex move 1403415.jpg --to=apple + move 1403415.jpg ok + (recording state in git...) + ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› + ╰─ git annex whereis + whereis 1403415.jpg (1 copy) + 766c9221-1404-4d68-bfa6-46d064bf4798 -- ting@ringo:/Volumes/apple/backup/annex/test-drop [apple] + ok + + + +### What version of git-annex are you using? On what operating system? + +`git-annex` installed via Homebrew on OS X. Version: 8.20201103 + + ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› + ╰─ git annex version + git-annex version: 8.20201103 + build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV + dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.2.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external + operating system: darwin x86_64 + supported repository versions: 8 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 + local repository version: 8 + +### Please provide any additional information below. + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I'm really excited about using git-annex, and have spent 3-4 days reading documentation, past bug reports, setting up repos, transferring data, and manually testing failure cases (e.g. this one :p). + +Thanks for all your hard work! From eaeedfe0ca1c713fba996f307d217b7c0b22e4d9 Mon Sep 17 00:00:00 2001 From: kyle Date: Fri, 13 Nov 2020 16:36:46 +0000 Subject: [PATCH 2/6] Added a comment --- ..._c2857172eba97c0ae6c829c634aadd15._comment | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment new file mode 100644 index 0000000000..7281ad07bb --- /dev/null +++ b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment @@ -0,0 +1,40 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="comment 1" + date="2020-11-13T16:36:45Z" + content=""" +I can trigger this on my end. + +With the script below, it bisects [*] to 0133b7e5a (move: Improve +resuming a move that was interrupted after the object was transferred, +2020-10-21), which was included in the 8.20201103 release. + + [*] Well, almost. I couldn't build the two commits right before + 0133b7e5a (363acfb55 and 62d630272). + +[[!format sh \"\"\" +cd \"$(mktemp -d \"${TMPDIR:-/tmp}\"/gx-XXXXXXX)\" || exit 125 +git annex version +git init a +( + cd a + echo one >one + git annex init a + git annex add one + git commit -mone +) +git clone -o a a b +( + cd b + git annex init b + git annex get one + git annex numcopies 2 + git annex numcopies + git annex whereis +) +git -C b annex move --to=a one || exit 0 +exit 1 +\"\"\"]] + +"""]] From ed7afabdb1de31b8b3a8269e882bace98af994b8 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 13 Nov 2020 13:34:28 -0400 Subject: [PATCH 3/6] fix build on windows --- Utility/LockFile/Windows.hs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Utility/LockFile/Windows.hs b/Utility/LockFile/Windows.hs index a8a0748434..781f8cfd31 100644 --- a/Utility/LockFile/Windows.hs +++ b/Utility/LockFile/Windows.hs @@ -27,7 +27,7 @@ type LockHandle = HANDLE {- Tries to lock a file with a shared lock, which allows other processes to - also lock it shared. Fails if the file is exclusively locked. -} lockShared :: LockFile -> IO (Maybe LockHandle) -lockShared = openLock fILE_SHARE_READ . fromRawFilePath +lockShared = openLock fILE_SHARE_READ {- Tries to take an exclusive lock on a file. Fails if another process has - a shared or exclusive lock. @@ -36,7 +36,7 @@ lockShared = openLock fILE_SHARE_READ . fromRawFilePath - read or write by any other process. So for advisory locking of a file's - content, a separate LockFile should be used. -} lockExclusive :: LockFile -> IO (Maybe LockHandle) -lockExclusive = openLock fILE_SHARE_NONE . fromRawFilePath +lockExclusive = openLock fILE_SHARE_NONE {- Windows considers just opening a file enough to lock it. This will - create the LockFile if it does not already exist. From 89a151608d5078769374df863d328f100be05160 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 13 Nov 2020 13:34:34 -0400 Subject: [PATCH 4/6] hush build warning on OSX --- Assistant/Install/Menu.hs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Assistant/Install/Menu.hs b/Assistant/Install/Menu.hs index f2a532efe1..91fcd3baf5 100644 --- a/Assistant/Install/Menu.hs +++ b/Assistant/Install/Menu.hs @@ -16,7 +16,9 @@ import Utility.Path import System.IO import Utility.SystemDirectory +#ifndef darwin_HOST_OS import System.FilePath +#endif installMenu :: FilePath -> FilePath -> FilePath -> FilePath -> IO () #ifdef darwin_HOST_OS From 98d1cbd1eebb3c2bef07ac4bfbf99133b5f7004a Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 13 Nov 2020 13:49:12 -0400 Subject: [PATCH 5/6] add a test case that move respects numcopies There is already one for drop, but move and drop don't handle numcopies exactly the same, and there was recently a reversion in it checking numcopies properly at all. This commit was sponsored by Noam Kremen on Patreon. --- Test.hs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Test.hs b/Test.hs index b67dff8398..91083d4d64 100644 --- a/Test.hs +++ b/Test.hs @@ -312,6 +312,7 @@ unitTests note = testGroup ("Unit Tests " ++ note) , testCase "get (ssh remote)" test_get_ssh_remote , testCase "move" test_move , testCase "move (ssh remote)" test_move_ssh_remote + , testCase "move (numcopies)" test_move_numcopies , testCase "copy" test_copy , testCase "lock" test_lock , testCase "lock --force" test_lock_force @@ -657,6 +658,20 @@ test_move' setup = setup $ do unannexed ingitfile inmainrepo $ unannexed ingitfile +test_move_numcopies :: Assertion +test_move_numcopies = intmpclonerepo $ do + inmainrepo $ annexed_present annexedfile + git_annex "numcopies" ["2"] @? "setting numcopies failed" + git_annex "move" ["--from", "origin", annexedfile] @? "move of file --from remote with numcopies unsatisfied but not made worse failed unexpectedly" + annexed_present annexedfile + inmainrepo $ annexed_notpresent annexedfile + git_annex "move" ["--to", "origin", annexedfile] @? "move of file --to remote with numcopies unsatisfied but not made worse failed unexpectedly" + annexed_notpresent annexedfile + inmainrepo $ annexed_present annexedfile + git_annex "get" [annexedfile] @? "get of file failed" + git_annex_shouldfail "move" ["--from", "origin", annexedfile] @? "move of file --from remote violated numcopies setting" + git_annex_shouldfail "move" ["--to", "origin", annexedfile] @? "move of file --to remote violated numcopies setting" + test_copy :: Assertion test_copy = intmpclonerepo $ do annexed_notpresent annexedfile From a30030c4a6042cadc07cbca3c54d5a3a62425881 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 13 Nov 2020 14:19:32 -0400 Subject: [PATCH 6/6] move: Fix a regression in the last release that made move --to not honor numcopies settings This commit was sponsored by Svenne Krap on Patreon. --- CHANGELOG | 2 ++ Command/Move.hs | 2 +- ...violates_numcopies___40__regression__41__.mdwn | 2 ++ ...nt_2_98a0c583ed094fec0e8c666e5839c2f7._comment | 15 +++++++++++++++ 4 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment diff --git a/CHANGELOG b/CHANGELOG index be011d78a3..93fa5851db 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,5 +1,7 @@ git-annex (8.20201104) UNRELEASED; urgency=medium + * move: Fix a regression in the last release that made move --to not + honor numcopies settings. * sync --content: Fix a bug where a file that was not preferred content could be transferred to a remote. This happened when the file got deleted after the sync started running. diff --git a/Command/Move.hs b/Command/Move.hs index 85ee923422..1e5a282752 100644 --- a/Command/Move.hs +++ b/Command/Move.hs @@ -151,7 +151,7 @@ toPerform dest removewhen key afile fastcheck isthere = do when fastcheck $ warning "This could have failed because --fast is enabled." stop - Right True -> logMove srcuuid destuuid False key $ \deststartedwithcopy -> + Right True -> logMove srcuuid destuuid True key $ \deststartedwithcopy -> finish deststartedwithcopy $ unlessM (expectedPresent dest key) $ Remote.logStatus dest key InfoPresent diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn b/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn index 757e1803ba..9ab92e08de 100644 --- a/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn +++ b/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn @@ -54,3 +54,5 @@ Relevant links: I'm really excited about using git-annex, and have spent 3-4 days reading documentation, past bug reports, setting up repos, transferring data, and manually testing failure cases (e.g. this one :p). Thanks for all your hard work! + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment new file mode 100644 index 0000000000..a34630e89b --- /dev/null +++ b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2020-11-13T18:01:29Z" + content=""" +Oh what a dumb mistake, literally a copied "False" that should be "True". + +I'm going to move the next release up due to this regression, but +thankfully it doesn't result in data loss, so am not going to do a crash +priority release. Probably Monday. + +The test suite also has a test case for move and numcopies now. It used to be +ok for it to only test drop and numcopies, but that changed a while back +and it should have had one already. +"""]]