git-annex/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn

### Please describe the problem.

When adding plenty of files to my git annex repository, I encounter recurring sqlite errors.

### What steps will reproduce the problem?
Create a git annex repo, add thousands of annexed binary files, and add thousands of small files tracked only with git.


### What version of git-annex are you using? On what operating system?

	>git annex version
	git-annex version: 6.20171003
	build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
	dependency versions: aws-0.13.0 bloomfilter-2.0.1.0 cryptonite-0.10 DAV-1.2 feed-0.3.10.4 ghc-7.10.3 http-client-0.4.26.2 persistent-sqlite-2.2 torrent-10000.0.0 uuid-1.3.11 yesod-1.4.2
	key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
	remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
	local repository version: 6
	supported repository versions: 3 5 6
	upgrade supported from repository versions: 0 1 2 3 4 5
	operating system: linux x86_64

	>lsb_release -d
	Description:    Ubuntu 16.04.3 LTS

	>uname -r
	4.4.0-43-Microsoft
	#(I am using Bash on Windows.)

### Please provide any additional information below.


	# If you can, paste a complete transcript of the problem occurring here.
	>git ls-files --others | grep txt | wc -l
	1953
	>git add $(git ls-files --others | grep txt)
	sqlite worker thread crashed: SQLite3 returned ErrorIO while attempting to perform prepare "SELECT null from content limit 1": disk I/O error
	git-annex: sqlite query crashed
	error: external filter git-annex smudge --clean %f failed 1
	error: external filter git-annex smudge --clean %f failed
	# [...] plenty of errors follow
	>git ls-files --others | grep txt | wc -l
	1953
	# End of transcript or log.


Triying to solve this problem, I found a part of answer in the form of a similar problem encounter here : <https://git-annex.branchable.com/forum/Problem_with_corrupt_SQLite_DB/>

Deleting git annex databases and running git annex fsck didnt do the trick:
	
	rm -rf .git/annex/keys/db .git/annex/keys/db-wal
	git annex fsck --incremental -J4
	git add $(git ls-files --others | grep txt)
	# Again, plenty of sqlite errors :()

### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Oh yeah, I am still discovering this powerfull git annex tool. 

In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other.
2017-11-07 16:03:17 +00:00			`### Please describe the problem.`

			`When adding plenty of files to my git annex repository, I encounter recurring sqlite errors.`

			`### What steps will reproduce the problem?`
			`Create a git annex repo, add thousands of annexed binary files, and add thousands of small files tracked only with git.`


			`### What version of git-annex are you using? On what operating system?`

			`>git annex version`
			`git-annex version: 6.20171003`
			`build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi`
			`dependency versions: aws-0.13.0 bloomfilter-2.0.1.0 cryptonite-0.10 DAV-1.2 feed-0.3.10.4 ghc-7.10.3 http-client-0.4.26.2 persistent-sqlite-2.2 torrent-10000.0.0 uuid-1.3.11 yesod-1.4.2`
			`key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL`
			`remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external`
			`local repository version: 6`
			`supported repository versions: 3 5 6`
			`upgrade supported from repository versions: 0 1 2 3 4 5`
			`operating system: linux x86_64`

			`>lsb_release -d`
			`Description: Ubuntu 16.04.3 LTS`

			`>uname -r`
			`4.4.0-43-Microsoft`
			`#(I am using Bash on Windows.)`

			`### Please provide any additional information below.`


			`# If you can, paste a complete transcript of the problem occurring here.`
			`>git ls-files --others \| grep txt \| wc -l`
			`1953`
			`>git add $(git ls-files --others \| grep txt)`
			`sqlite worker thread crashed: SQLite3 returned ErrorIO while attempting to perform prepare "SELECT null from content limit 1": disk I/O error`
			`git-annex: sqlite query crashed`
			`error: external filter git-annex smudge --clean %f failed 1`
			`error: external filter git-annex smudge --clean %f failed`
			`# [...] plenty of errors follow`
2017-11-07 16:05:15 +00:00			`>git ls-files --others \| grep txt \| wc -l`
			`1953`
2017-11-07 16:03:17 +00:00			`# End of transcript or log.`


			`Triying to solve this problem, I found a part of answer in the form of a similar problem encounter here : <https://git-annex.branchable.com/forum/Problem_with_corrupt_SQLite_DB/>`

			`Deleting git annex databases and running git annex fsck didnt do the trick:`

			`rm -rf .git/annex/keys/db .git/annex/keys/db-wal`
			`git annex fsck --incremental -J4`
			`git add $(git ls-files --others \| grep txt)`
			`# Again, plenty of sqlite errors :()`

			`### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)`
			`Oh yeah, I am still discovering this powerfull git annex tool.`

			`In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other.`