CI to automate build of cargo lockfiles on different Alpine releases for git-annex aports
https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/git-annex
a3370ac459
Sqlite doesn't support multiple concurrent writers at all. One of them will fail to write. It's not even possible to have two processes building up separate transactions at the same time. Before using sqlite, incremental fsck could work perfectly well with multiple fsck processes running concurrently. I'd like to keep that working. My partial solution, so far, is to make git-annex buffer writes, and every so often send them all to sqlite at once, in a transaction. So most of the time, nothing is writing to the database. (And if it gets unlucky and a write fails due to a collision with another writer, it can just wait and retry the write later.) This lets multiple processes write to the database successfully. But, for the purposes of concurrent, incremental fsck, it's not ideal. Each process doesn't immediately learn of files that another process has checked. So they'll tend to do redundant work. Only way I can see to improve this is to use some other mechanism for short-term IPC between the fsck processes. Not yet done. ---- Also, make addDb check if an item is in the database already, and not try to re-add it. That fixes an intermittent crash with "SQLite3 returned ErrorConstraint while attempting to perform step." I am not 100% sure why; it only started happening when I moved write buffering into the queue. It seemed to generally happen on the same file each time, so could just be due to multiple files having the same key. However, I doubt my sound repo has many duplicate keys, and I suspect something else is going on. ---- Updated benchmark, with the 1000 item queue: 6m33.808s |
||
|---|---|---|
| Annex | ||
| Assistant | ||
| Backend | ||
| Build | ||
| CmdLine | ||
| Command | ||
| Common | ||
| Config | ||
| Database | ||
| debian | ||
| doc | ||
| Git | ||
| Limit | ||
| Logs | ||
| Messages | ||
| Remote | ||
| RemoteDaemon | ||
| standalone | ||
| static | ||
| templates | ||
| Types | ||
| Upgrade | ||
| Utility | ||
| .ghci | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| Annex.hs | ||
| Assistant.hs | ||
| Backend.hs | ||
| build.bat | ||
| BuildFlags.hs | ||
| CHANGELOG | ||
| Checks.hs | ||
| CmdLine.hs | ||
| Command.hs | ||
| Common.hs | ||
| Config.hs | ||
| configure.hs | ||
| COPYRIGHT | ||
| Creds.hs | ||
| Crypto.hs | ||
| ghci | ||
| git-annex.cabal | ||
| git-annex.hs | ||
| git-union-merge.hs | ||
| Git.hs | ||
| INSTALL | ||
| Limit.hs | ||
| Locations.hs | ||
| Logs.hs | ||
| Makefile | ||
| Messages.hs | ||
| NEWS | ||
| README | ||
| Remote.hs | ||
| Setup.hs | ||
| Test.hs | ||
| Types.hs | ||
| Upgrade.hs | ||
git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, checksumming time, or disk space. For documentation, see doc/ or <https://git-annex.branchable.com/>