git-annex/doc/todo/v7_InodeCache_timestamp_resolution.mdwn

45 lines
1.7 KiB
Text
Raw Normal View History

add small delay to avoid problems on systems with low-resolution mtime I've seen intermittent failures of the test suite with v6 for a long time, it seems to have possibly gotten worse with the changes around v7. Or just being unlucky; all tests failed today. Seen on amd64 and i386 builders, repeatedly but intermittently: unused: FAIL (4.86s) Test.hs:928: git diff did not show changes to unlocked file And I think other such failures, all involving v7/v6 mode tests. I managed to reproduce the unused failure with --keep-failures, and inside the repo, git diff was indeed not showing any changes for the modified unlocked file. The two stats will be the same other than mtime; the old and new files have the same size and inode, since the test case writes to the file and then overwrites it. Indeed, notice the identical timestamps: builder@orca:~/gitbuilder/build/.t/tmprepo335$ echo 1 > foo; stat foo; echo 2 > foo; stat foo File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.894942036 +0000 Change: 2018-10-29 22:14:10.894942036 +0000 Birth: - File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.898942036 +0000 Change: 2018-10-29 22:14:10.898942036 +0000 Birth: - I'm seeing this in Linux VMs; it doesn't happen on my laptop. I've also not experienced the intermittent test suite failures on my laptop. So, I hope that this small delay will avoid the problem. Update: I didn't, indeed I then reproduced the same failure on my laptop, so it must be due to something else. But keeping this change anyway since not needing to worry about lowish-resolution mtime in the test suite seems worthwhile.
2018-10-29 22:42:20 +00:00
InodeCache currently uses modificationTime which has a 1 second resolution.
(And when comparing weakly, further weakens to 2 second resolution.)
In [[!commit c28ca8294f7695c77e5f03762171e829de5d6ea4]], the clean filter
started checking the InodeCache to see if a file is modified.
This means that modifying a file, running `git add`, then modifying again
and `git add` within the same second won't stage the second version of the
file. (Although luckily it also compares file size.)
add small delay to avoid problems on systems with low-resolution mtime I've seen intermittent failures of the test suite with v6 for a long time, it seems to have possibly gotten worse with the changes around v7. Or just being unlucky; all tests failed today. Seen on amd64 and i386 builders, repeatedly but intermittently: unused: FAIL (4.86s) Test.hs:928: git diff did not show changes to unlocked file And I think other such failures, all involving v7/v6 mode tests. I managed to reproduce the unused failure with --keep-failures, and inside the repo, git diff was indeed not showing any changes for the modified unlocked file. The two stats will be the same other than mtime; the old and new files have the same size and inode, since the test case writes to the file and then overwrites it. Indeed, notice the identical timestamps: builder@orca:~/gitbuilder/build/.t/tmprepo335$ echo 1 > foo; stat foo; echo 2 > foo; stat foo File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.894942036 +0000 Change: 2018-10-29 22:14:10.894942036 +0000 Birth: - File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.898942036 +0000 Change: 2018-10-29 22:14:10.898942036 +0000 Birth: - I'm seeing this in Linux VMs; it doesn't happen on my laptop. I've also not experienced the intermittent test suite failures on my laptop. So, I hope that this small delay will avoid the problem. Update: I didn't, indeed I then reproduced the same failure on my laptop, so it must be due to something else. But keeping this change anyway since not needing to worry about lowish-resolution mtime in the test suite seems worthwhile.
2018-10-29 22:42:20 +00:00
I think that optimisation needs to be disabled when inode caches will be
compared weakly, because 2 seconds is just too long. This will mean slow
`git checkout` on FAT and also when a user moves a repo to a different
filesystem. But I don't see a way to avoid it.
> Hmm, on second thought, that would mean every inAnnex on FAT
> would need to checksum the content. That's just too slow to be practical.
> `git annex fsck` will clean up if trusting the timestamps causes
> it to make a mistake on FAT.
add small delay to avoid problems on systems with low-resolution mtime I've seen intermittent failures of the test suite with v6 for a long time, it seems to have possibly gotten worse with the changes around v7. Or just being unlucky; all tests failed today. Seen on amd64 and i386 builders, repeatedly but intermittently: unused: FAIL (4.86s) Test.hs:928: git diff did not show changes to unlocked file And I think other such failures, all involving v7/v6 mode tests. I managed to reproduce the unused failure with --keep-failures, and inside the repo, git diff was indeed not showing any changes for the modified unlocked file. The two stats will be the same other than mtime; the old and new files have the same size and inode, since the test case writes to the file and then overwrites it. Indeed, notice the identical timestamps: builder@orca:~/gitbuilder/build/.t/tmprepo335$ echo 1 > foo; stat foo; echo 2 > foo; stat foo File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.894942036 +0000 Change: 2018-10-29 22:14:10.894942036 +0000 Birth: - File: foo Size: 2 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 3546179 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ builder) Gid: ( 1000/ builder) Access: 2018-10-29 22:14:10.894942036 +0000 Modify: 2018-10-29 22:14:10.898942036 +0000 Change: 2018-10-29 22:14:10.898942036 +0000 Birth: - I'm seeing this in Linux VMs; it doesn't happen on my laptop. I've also not experienced the intermittent test suite failures on my laptop. So, I hope that this small delay will avoid the problem. Update: I didn't, indeed I then reproduced the same failure on my laptop, so it must be due to something else. But keeping this change anyway since not needing to worry about lowish-resolution mtime in the test suite seems worthwhile.
2018-10-29 22:42:20 +00:00
Otherwise, the problem can be fixed by using modificationTimeHiRes.
But! All existing InodeCaches would then appear to have changed. This would
confuse handling of existing v6 repos badly. (And direct mode uses
InodeCache too..)
So, need to detect, when reading a serialized InodeCache,
if it's low res or high res. And when comparing two of different
resolutions, truncate to low res.
readInodeCache currently fails if the mtime contains a decimal, eg
ghci> readInodeCache "1 2 3.1"
Nothing
What would work, w/o breaking back-compat is
ghci> readInodeCache "1 2 3 1"
Just (InodeCache (InodeCachePrim 1 2 3))
So the decimal part of the mtime becomes the 4th word and old
versions of git-annex will ignore it.
> [[fixed|done]] --[[Joey]]