mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions 1
git-annex/doc
History
Joey Hess f6aa097a39
avoid import writing to cidsdb initially 
Speed up importing trees from special remotes somewhat by avoiding
redundant writes to sqlite database.

Before, import would write to both the git-annex branch and also to the
sqlite database. But then the next time it was run, needsUpdateFromLog
would see the branch had changed, so run updateFromLog, which would make
the same writes to the sqlite database a second time.

Now import writes only to the git-annex branch. The next time it's run,
needsUpdateFromLog sees that the branch has changed and so calls
updateFromLog, which updates the sqlite database.

Why defer the write to the sqlite database like this? It seems that it
could write to the database as it goes, and at the end call
recordAnnexBranchTree to indicate that the information in the git-annex
branch has all been written to the cidsdb. That would avoid the second
import doing extra work.

But, there could be other processes running at the same time, and one of
them may update the git-annex branch, eg merging a remote git-annex branch
into it. Any cids logs on that merged git-annex branch would not be
reflected in the cidsdb yet. If the import then called
recordAnnexBranchTree, the cidsdb would never get updated with that merged
information.

I don't think there's a good way to prevent, or to detect that situation.
So, it can't call recordAnnexBranchTree at the end. So it might as well
wait until the next run and do updateFromLog then. It could instead do
updateFromLog at the end, but it's going to check needsUpdateFromLog
at the beginning anyway.

Note that the database writes were queued, so there is already a cidmap
that is used to remember changes that the current process has made.
So, omitting database writes can't change the behavior of the current
process.

Also note that thirdpartypopulatedimport uses recordcidkeyindb, which
reflects what it already did. That code path does not use the cidmap,
but does not need to query it either. It might be possible to make that
code path also only update the git-annex branch and not the db, but I
haven't checked.

Sponsored-by: Noam Kremen on Patreon
2023-05-30 17:05:28 -04:00
..
Android
android
assistant
automatic_conflict_resolution
backends
bare_repositories
bugs avoid import writing to cidsdb initially 2023-05-30 17:05:28 -04:00
chunking comment 2023-02-27 12:03:09 -04:00
coding_style
contribute update comment about updating comments :) 2022-07-27 19:29:43 -04:00
copies
design response 2023-04-07 13:00:56 -04:00
devblog devblog 2023-04-12 15:03:01 -04:00
direct_mode
download
encryption
footer
forum Added a comment 2023-05-19 19:06:01 +00:00
future_proofing
git-annex-add
git-annex-addurl promote comment to bug 2023-03-27 14:10:32 -04:00
git-annex-adjust
git-annex-benchmark comment 2022-12-20 14:30:18 -04:00
git-annex-checkpresentkey
git-annex-config
git-annex-copy
git-annex-direct
git-annex-drop
git-annex-enableremote
git-annex-find findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-annex-import
git-annex-importfeed
git-annex-lookupkey
git-annex-metadata
git-annex-move
git-annex-p2p
git-annex-preferred-content Support "inbackend" in preferred content expressions 2022-09-26 16:06:49 -04:00
git-annex-reinject
git-annex-repair Added a comment: When a weird 'no space left on device' error appears 2022-12-21 16:46:44 +00:00
git-annex-rmurl
git-annex-setpresentkey
git-annex-sync comment and improve docs 2023-04-19 12:20:26 -04:00
git-annex-test
git-annex-unannex Added a comment: Re: Weird behavior of git archive in combination with largefiles configuration 2023-02-26 18:45:31 +00:00
git-annex-undo
git-annex-uninit
git-annex-unlock
git-annex-unused
git-annex-whereis
git-annex-whereused comment 2023-05-05 13:36:57 -04:00
how_it_works
install Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
internals convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license
links
metadata
news add news item for git-annex 10.20230407 2023-04-07 13:37:18 -04:00
not
polls
preferred_content
profiling
projects rename an old closed bug to avoid filename too long on windows checkout 2023-03-18 12:24:58 -04:00
required_content
scalability
security
special_remotes update docs to account for exporttree 2023-04-25 12:15:14 -04:00
stickers
submodules analysis and followup 2023-04-20 14:11:04 -04:00
sync this comment section is not a BTS, part N+1 2023-03-21 13:48:34 -04:00
templates
testimonials
thanks Revert "update" 2023-05-07 12:54:29 -04:00
tips default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
todo update 2023-05-30 15:49:52 -04:00
trust
tuning
upgrades
use_case
users 2023-03-29 22:32:06 +00:00
videos
walkthrough git-annex pull and push 2023-05-16 16:51:07 -04:00
workflow
Android.mdwn
assistant.mdwn
automatic_conflict_resolution.mdwn
backends.mdwn fix typo 2022-12-26 12:36:56 -04:00
bare_repositories.mdwn
bugs.mdwn
builds.mdwn remove appveyor badge, which does not work 2023-03-22 10:55:38 -04:00
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn remove calendar 2023-01-05 14:50:26 -04:00
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn
encryption.mdwn correct documentation, keyid= only works once 2022-08-15 14:25:13 -04:00
favicon.ico
forum.mdwn
future_proofing.mdwn
git-annex-add.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-addunused.mdwn addunused, dropunused: Support --json and --json-error-messages 2023-05-05 14:01:40 -04:00
git-annex-addurl.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-annex-adjust.mdwn support --unlock-present adjustment of view branches 2023-02-27 15:37:57 -04:00
git-annex-assist.mdwn document -m 2023-05-23 11:46:54 -04:00
git-annex-assistant.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-backends.mdwn improve documentation about backends 2022-09-26 15:59:10 -04:00
git-annex-benchmark.mdwn Typo fix unncessary -> unnecessary. 2022-08-20 09:40:19 -04:00
git-annex-calckey.mdwn typo fix 2022-08-03 11:11:10 -04:00
git-annex-checkpresentkey.mdwn
git-annex-common-options.mdwn Improved handling of --time-limit when combined with -J 2022-09-22 12:54:52 -04:00
git-annex-config.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-configremote.mdwn configremote 2023-04-18 15:30:49 -04:00
git-annex-contentlocation.mdwn
git-annex-copy.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dead.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-describe.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-diffdriver.mdwn
git-annex-direct.mdwn
git-annex-drop.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dropkey.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dropunused.mdwn addunused, dropunused: Support --json and --json-error-messages 2023-05-05 14:01:40 -04:00
git-annex-edit.mdwn
git-annex-enable-tor.mdwn
git-annex-enableremote.mdwn enableremote: Support --json and --json-error-messages 2023-05-10 14:09:27 -04:00
git-annex-examinekey.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-expire.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-export.mdwn git-annex pull and push 2023-05-16 16:51:07 -04:00
git-annex-filter-branch.mdwn fix link to git-annex-initremote 2022-08-02 13:51:47 +00:00
git-annex-filter-process.mdwn Typo fix unncessary -> unnecessary. 2022-08-20 09:40:19 -04:00
git-annex-find.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-findkeys.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-findref.mdwn
git-annex-fix.mdwn fix: Support --json and --json-error-messages 2023-05-04 14:28:21 -04:00
git-annex-forget.mdwn
git-annex-fromkey.mdwn
git-annex-fsck.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-fuzztest.mdwn
git-annex-get.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-group.mdwn
git-annex-groupwanted.mdwn
git-annex-import.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-importfeed.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-annex-indirect.mdwn
git-annex-info.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-init.mdwn init: Support --json and --json-error-messages 2023-05-08 14:58:08 -04:00
git-annex-initremote.mdwn initremote: Support --json and --json-error-messages 2023-05-10 14:03:40 -04:00
git-annex-inprogress.mdwn
git-annex-list.mdwn
git-annex-lock.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-log.mdwn log: Support --json and --json-error-messages 2023-05-04 12:36:31 -04:00
git-annex-lookupkey.mdwn
git-annex-map.mdwn
git-annex-matchexpression.mdwn
git-annex-matching-expression.mdwn
git-annex-matching-options.mdwn --anything and --nothing 2022-12-20 15:44:09 -04:00
git-annex-merge.mdwn git-annex pull and push 2023-05-16 16:51:07 -04:00
git-annex-metadata.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-migrate.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex-mincopies.mdwn
git-annex-mirror.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-move.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-multicast.mdwn
git-annex-numcopies.mdwn
git-annex-p2p.mdwn
git-annex-post-receive.mdwn
git-annex-pre-commit.mdwn
git-annex-preferred-content.mdwn document how to include= a path with a space in it 2023-05-15 16:02:06 -04:00
git-annex-proxy.mdwn
git-annex-pull.mdwn sync: Added -g as a short option for --no-content 2023-05-17 12:34:26 -04:00
git-annex-push.mdwn push: Support --cleanup 2023-05-17 12:51:49 -04:00
git-annex-readpresentkey.mdwn
git-annex-registerurl.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-reinit.mdwn reinit: Support --json and --json-error-messages 2023-05-08 15:07:40 -04:00
git-annex-reinject.mdwn reinject: Support --json and --json-error-messages 2023-05-08 15:43:37 -04:00
git-annex-rekey.mdwn rekey, setpresentkey: Support --json and --json-error-messages 2023-05-04 16:03:54 -04:00
git-annex-remotedaemon.mdwn
git-annex-renameremote.mdwn configremote 2023-04-18 15:30:49 -04:00
git-annex-repair.mdwn
git-annex-required.mdwn
git-annex-resolvemerge.mdwn
git-annex-restage.mdwn restage: New git-annex command, handles restaging unlocked files 2022-09-23 16:29:59 -04:00
git-annex-rmurl.mdwn rmurl: Support --json and --json-error-messages 2023-05-04 11:28:27 -04:00
git-annex-schedule.mdwn
git-annex-semitrust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-setkey.mdwn
git-annex-setpresentkey.mdwn rekey, setpresentkey: Support --json and --json-error-messages 2023-05-04 16:03:54 -04:00
git-annex-shell.mdwn
git-annex-smudge.mdwn
git-annex-status.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-sync.mdwn document -m 2023-05-23 11:46:54 -04:00
git-annex-test.mdwn test: Add --test-debug option 2022-11-28 15:12:53 -04:00
git-annex-testremote.mdwn
git-annex-transferkey.mdwn
git-annex-transferkeys.mdwn
git-annex-transferrer.mdwn
git-annex-trust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-unannex.mdwn unannex: Support --json and --json-error-messages 2023-05-03 15:56:20 -04:00
git-annex-undo.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex-ungroup.mdwn
git-annex-uninit.mdwn uninit: Support --json and --json-error-messages 2023-05-11 13:43:02 -04:00
git-annex-unlock.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-unregisterurl.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-untrust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-unused.mdwn unused: Support --json and --json-error-messages 2023-05-08 14:39:57 -04:00
git-annex-upgrade.mdwn upgrade: Support --json and --json-error-messages and --json-progress 2023-05-10 12:54:48 -04:00
git-annex-vadd.mdwn add directory to views for files that lack specified metadata 2023-02-07 16:28:46 -04:00
git-annex-vcycle.mdwn
git-annex-version.mdwn
git-annex-vfilter.mdwn add directory to views for files that lack specified metadata 2023-02-07 16:28:46 -04:00
git-annex-vicfg.mdwn
git-annex-view.mdwn annex.maxextensionlength for view 2023-03-24 14:01:38 -04:00
git-annex-vpop.mdwn
git-annex-wanted.mdwn
git-annex-watch.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-webapp.mdwn
git-annex-whereis.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-whereused.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-remote-tor-annex.mdwn
git-union-merge.mdwn
how_it_works.mdwn
index.mdwn
install.mdwn move a paragraph into the pages that the links it talks about were long ago moved to 2023-02-11 16:07:05 -04:00
internals.mdwn convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license.mdwn
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn
polls.mdwn
preferred_content.mdwn
privacy.mdwn
profiling.mdwn
projects.mdwn
publicrepos.mdwn
related_software.mdwn Add borg2annex to related_software 2023-05-28 07:12:15 +02:00
repomap.png
required_content.mdwn
scalability.mdwn
security.mdwn
shortcuts.mdwn
sidebar.mdwn
sitemap.mdwn
special_remotes.mdwn add NNCP special remote! 2022-09-02 16:55:36 -04:00
stickers.mdwn
submodules.mdwn
summary.mdwn
sync.mdwn comment and improve docs 2023-04-19 12:20:26 -04:00
testimonials.mdwn
thanks.mdwn update 2023-01-07 13:56:06 -04:00
tips.mdwn
todo.mdwn
transferring_data.mdwn
trust.mdwn
tuning.mdwn
upgrades.mdwn v8 repositories automatically upgrade to v9 2022-07-25 16:20:04 -04:00
users.mdwn
videos.mdwn
walkthrough.mdwn
workflow.mdwn