improve sqlite MultiWriter handling of read after write

This removes a messy caveat that was easy to forget and caused at least one
bug. The price paid is that, after a write to a MultiWriter db, it has to
close the db connection that it had been using to read, and open a new
connection. So it might be a little bit slower. But, writes are usually
batched together, so there's often only a single write, and so there should
not be much of a slowdown. Notice that SingleWriter already closed the db
connection after a write, so paid the same overhead.

This is the second try at fixing a bug: git-annex get when run as the first
git-annex command in a new repo did not populate all unlocked files.
(Reversion in version 8.20210621)

Sponsored-by: Boyd Stephen Smith Jr. on Patreon
This commit is contained in:
Joey Hess 2021-10-19 15:13:29 -04:00
parent ade67b78c5
commit f4bdecc4ec
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 54 additions and 19 deletions

View file

@ -43,3 +43,43 @@ This outputs 1 for foo, followed by annex pointer files for files bar and baz.
The previous fix attempt did make foo get populated, before that none
of the files were populated.
----
`GIT_TRACE=1` shows that git only runs the smudge filter on the first
file, not the other two. And indeed, restagePointerFile is only called
on the first file.
Added debugging to Database.Keys.reconcileStaged, and it adds all 3 files to
the associated files table, but only adds the inode cache of foo.
And that's what I see in the db after the fact too. Which is
not itself a problem, to the extent that the other files are not
populated, and only populated files have an inode cache recorded.
So, Database.Keys.reconcileStaged is called after it gets foo,
but before the other files are present, and in reconcilepointerfile it
calls populatePointerFile and records the inode cache for foo.
That is how foo gets populated.
But, the other 2 files do not have populatePointerFile run on them.
In moveAnnex, it calls getAssociatedFiles and somehow that returns
`[]`, for all 3 files. This does not matter for foo, because it gets
populated by reconcileStaged as explained above. But for the other 2, with
no known associated files of course it fails to populate them.
So: Why is getAssociatedFiles returning `[]`? Those calls come
after Database.Keys.reconcileStaged has added the associated files,
but are somehow not seeing the changes it made.
Ah.. The keys db is opened in MultiWriter mode.
See the comment above the definition of MultiWriter,
which explains that a write to a MultiWriter database,
followed by a flushDbQueue may not be visible when reading
from that same database.
Verified this by making it re-open the db after reconcileStaged,
which did fix the problem.
A better fix is possible: Make MultiWriter mode not have this hidden
gotcha, by re-opening the db after writing to it always. [[done]]
--[[Joey]]