git-annex/Logs
Joey Hess 6821ba8dab
sync: use log to track adjusted branch needs updating
Speeds up sync in an adjusted branch by avoiding re-adjusting the branch
unncessarily, particularly when it is adjusted with --hide-missing or
--unlock-present.

When there are a lot of files, that was the majority of the time of a
--no-content sync.

Uses a log file, which is updated when content presence changes. This
adds a little bit of overhead to every file get/drop when on such an
adjusted branch. The overhead is minimal for get of any size of file,
but might be noticable for drop in some cases. It seems like a reasonable
trade-off. It would be possible to update the log file only at the end, but
then it would not happen if the command is interrupted.

When not in an adjusted branch, there should be no additional overhead.
(getCurrentBranch is an MVar read, and it avoids the MVar read of
getGitConfig.)

Note that this does not deal with situations such as:
git checkout master, git-annex get, git checkout adjusted branch,
git-annex sync. The sync won't know that the adjusted branch needs to be
updated. Dealing with that would add overhead to operation in non-adjusted
branches, which I don't like. Also, there are other situations like having
two adjusted branches that both need to be updated like this, and switching
between them and sync not updating.

This does mean a behavior change to sync, since it did previously deal
with those situations. But, the documentation did not say that it did.
The man pages only talk about sync updating the adjusted branch after
it transfers content.

I did consider making sync keep track of content it transferred (and
dropped) and only update the adjusted branch then, not to catch up to other
changes made previously. That would perform better. But it seemed rather
hard to implement, and also it would have problems with races with a
concurrent get/drop, which this implementation avoids.

And it seemed pretty likely someone had gotten used to get/drop followed by
sync updating the branch. It seems much less likely someone is switching
branches, doing get/drop, and then switching back and expecting sync to update
the branch.

Re-running git-annex adjust still does a full re-adjusting of the branch,
for anyone who needs that.

Sponsored-by: Leon Schuermann on Patreon
2023-06-08 14:35:41 -04:00
..
Chunk update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
ContentIdentifier Fix build with attoparsec-0.14 2021-03-24 12:11:50 -04:00
Difference update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Export Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
MetaData update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
PreferredContent start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Presence split out appending to journal from writing, high level only 2022-07-18 13:22:50 -04:00
Remote Fix build with attoparsec-0.14 2021-03-24 12:11:50 -04:00
SingleValue deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
Trust start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Activity.hs Future proof activity log parsing 2021-06-14 14:18:19 -04:00
AdjustedBranchUpdate.hs sync: use log to track adjusted branch needs updating 2023-06-08 14:35:41 -04:00
Chunk.hs Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
Config.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
ContentIdentifier.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Difference.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Export.hs Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
File.hs fix deadlock in restagePointerFiles 2022-12-08 14:36:11 -04:00
FsckResults.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Group.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Import.hs implement importChanges optimisaton (not used yet) 2023-05-31 16:01:34 -04:00
Line.hs Fix build with attoparsec-0.14 2021-03-24 12:11:50 -04:00
Location.hs handle Annex.Branch.files with read-only unmerged git-annex branches 2021-12-27 15:28:31 -04:00
MapLog.hs deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
MetaData.hs deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
Multicast.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
NumCopies.hs cache negative lookups of global numcopies and mincopies 2023-06-06 14:43:25 -04:00
PreferredContent.hs prevent dropping required content of other file using same content 2021-05-25 11:34:06 -04:00
Presence.hs split out appending to journal from writing, high level only 2022-07-18 13:22:50 -04:00
Remote.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
RemoteState.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
Restage.hs fix deadlock in restagePointerFiles 2022-12-08 14:36:11 -04:00
Schedule.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
SingleValue.hs deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
Smudge.hs add a finalizer to streamLogFile 2022-09-23 13:49:01 -04:00
Transfer.hs rename Git.Filename to Git.Quote 2023-04-12 17:22:03 -04:00
Transitions.hs fix obviously wrong attoparsec parser 2022-02-07 14:15:17 -04:00
Trust.hs info: Added --autoenable option 2022-06-01 14:20:38 -04:00
Unused.hs Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
Upgrade.hs add annex.dbdir (WIP) 2022-08-11 16:58:53 -04:00
UUID.hs one-way escaping of newlines in uuid.log 2023-03-13 14:19:32 -04:00
UUIDBased.hs Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
View.hs understand adjusted view branch names 2023-02-27 14:57:58 -04:00
Web.hs add annex.alwayscompact 2022-07-18 16:39:19 -04:00