mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

comment and a neat idea

Browse source
This commit is contained in:
Joey Hess 2023-05-30 15:42:34 -04:00
parent f9baf11e11
commit aaeae746f0
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
3 changed files with 75 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

38
doc/bugs/importtree_spends_hours_reading_cidsdb/comment_2_8f609e231fdf315d2b658d30d708d471._comment Normal file
View file

@ -0,0 +1,38 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2023-05-30T18:49:34Z"
content="""
I think I see why a second sync re-writes the cidsdb with information from
the git-annex branch.
The first sync does write to the cidsdb at the same time it writes to the
git-annex branch. So, it seems that it could at the end call
recordAnnexBranchTree to indicate that the information in the git-annex
branch has all been written to the cidsdb. That would avoid the second sync
doing extra work.
But, there could be other processes running at the same time, and one of
them may update the git-annex branch, eg merging a remote git-annex branch
into it. Any cids logs on that merged git-annex branch would not be
reflected in the cidsdb yet. If the sync then called
recordAnnexBranchTree, the cidsdb would never get updated with that merged
information.
I don't think there's a good way to prevent, or to detect that situation.
So, it can't call recordAnnexBranchTree at the end, and has to do extra
work in updateFromLog at the beginning.
What it could do is, only record a cid to the git-annex branch, not to the
cidsdb. Then the updateFromLog would not be doing extra work, but necessary
work. But, it also needs to read cids from the db during import, and if it
doesn't record a cid there, it won't know when a later file has the same
cid. So it will re-import it. Which for other special remotes than
directory, means an expensive second download of the content.
Anyway, the extra work of re-writing the cidsdb is only done on the sync
immediately after the one that did import some new files. And it only
re-writes cids for the new files, not for unchanged files.
I'm not sure that this extra work is what the bug reporter was complaining about though.
"""]]

8
doc/bugs/importtree_spends_hours_reading_cidsdb/comment_3_d54f2c81763b4119c881e283cff80d44._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2023-05-30T19:25:11Z"
content="""
See [[todo/speed_up_import_tree]] for a few ideas of things that would
speed it up.
"""]]