add back inode to directory special remote ContentIdentifier

Directory special remotes with importtree=yes have changed to once more
take inodes into account. This will cause extra work when importing from a
directory on a FAT filesystem that changes inodes on every mount.

To avoid that extra work, set ignoreinodes=yes when initializing a new
directory special remote, or change the configuration of your existing
remote: git-annex enableremote foo ignoreinodes=yes

This will mean a one-time re-import of all contents from every directory
special remote due to the changed setting.

73df633a62 thought
it was too unlikely that there would be modifications that the inode number
was needed to notice. That was probably right; it's very unlikely that a
file will get modified and end up with the same size and mtime as before.
But, what was not considered is that a program like NextCloud might write
two files with different content so closely together that they share the
mtime. The inode is necessary to detect that situation.

Sponsored-by: Max Thoursie on Patreon
This commit is contained in:
Joey Hess 2022-03-21 13:12:02 -04:00
parent d219626794
commit 3e2f1f73cb
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 117 additions and 45 deletions

View file

@ -61,3 +61,5 @@ Unfortunately, I don't have the output of git-annex anymore. However, the used c
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've been successfully using git-annex to manage the growing collection of photos I take for several years now. Synchronizing the collection across several disks while files are renamed and some disks are only rarely synchronized and some contain only a subset of the photos wouldn't have been possible without git-annex. Also, using git-annex gives me the safety that the original photos will never be modified. There is nothing better than using git-annex to verify that all photos have been copied unmodified from a memory card before wiping it.
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,36 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2022-03-21T16:20:27Z"
content="""
The directory special remote use the size and mtime as the
ContentIdentifier of a file. It used to use inode number as well, but
some filesystems make up new inodes on each mount so that was dropped.
So when two files have the same size and mtime, it will import one of them,
and then when it comes to the other one, see that it already has that
ContentIdentifier and use the content it already imported.
For example:
joey@darkstar:/tmp/t>git-annex initremote d type=directory directory=../d importtree=yes exporttree=yes encryption=none
joey@darkstar:/tmp/t>echo hi > ../d/1
joey@darkstar:/tmp/t>echo no > ../d/2
joey@darkstar:/tmp/t>touch --reference=../d/1 ../d/2
joey@darkstar:/tmp/t>git-annex import master --from d
joey@darkstar:/tmp/t>git merge d/master
joey@darkstar:/tmp/t>cmp 1 2
joey@darkstar:/tmp/t>
Perhaps NextCloud is for some reason giving multiple files the same mtime?
This is rather unlikely to happen naturally, especially on modern
filesystems that have high resolution mtimes. Even "echo 1> 1; echo 2>2"
generates files with 2 different mtimes. To get the same mtime, the two
files have to be written by the same process at very close to the same
time.
Probably that's why NextCloud managed to get the same mtime, although why
it would be rewriting the content of the file when the content had not
changed I don't know and perhaps that's a bug of some kind in it.
"""]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2022-03-21T16:38:43Z"
content="""
So this makes me think that [[!73df633a6215faa093e4a7524c6d328aa988aed1]]
needs to be reverted and the inode number used again. But that did fix a
bug of a sort, that importing from a fat filesystem after it was remounted
would re-import everything. So I suppose the best that can be done is to
make it configurable whether inodes should be ignored or not when importing
from a directory special remote. And presumably default to not ignoring
them.
done
"""]]