diff --git a/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds.mdwn b/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds.mdwn new file mode 100644 index 0000000000..610c0e9d1a --- /dev/null +++ b/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds.mdwn @@ -0,0 +1,78 @@ +Before I start on what's gone wrong, many thanks for a great program: finally a way of finding out where I've put all those files, and I enjoyed your talk in Australia. Not quite to New Zealand, but a good start :-) + +To play with git-annex, I decided to convert my ~/Downloads directory to a git-annex repository, as there were a wide variety of files in it, mostly easily replaceable, and also handy to have on multiple machines. In hindsight, probably wasn't a great idea, as I'd regularly forget it was a git-annex repo and move files and directories out manually, which caused all sorts of fun trying to sort out the dead symlinks. Then I after one update, the repository format changed, from WORM to xxx256 format which meant there were now both sorts in the GA object store. + +More recently I'd tried converting the repo to direct mode, you get the idea: lots of playing with git-annex commands, and now that I think about it, possibly some git commands too trying to repair missing files. + +Anyway I've ended up with a 27GB git-annex repo that now manages to kill git-annex whenever I try to check it using "git-annex fsck". + +Not only does the fsck subcommand cause it to die, but also "find", "whereis" and "status". It dies on the same file (for find/whereis/fsck). + +e.g. + + ... lots of stuff deleted ... + whereis 1wolf14.zip (2 copies) + 051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop + 2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads) + ok + git-annex: out of memory (requested 2097152 bytes) + +Now "git status" for some repo data: + + myPC:~/Downloads$ git annex status + supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL + supported remote types: git S3 bup directory rsync web webdav glacier hook + repository mode: direct + trusted repositories: 0 + semitrusted repositories: 4 + 00000000-0000-0000-0000-000000000001 -- web + 051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop + 2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads) + 48fbe52a-a1b3-11e1-bb80-ebc15118871d -- netbk + untrusted repositories: 0 + dead repositories: 0 + transfers in progress: none + available local disk space: 100 gigabytes (+1 megabyte reserved) + local annex keys: 1719 + local annex size: 27 gigabytes + known annex keys: git-annex: out of memory (requested 2097152 bytes) + +It always seems to die at about 3.5GB memory usage. This is running on Ubuntu 12.04, using the latest GA release built using cabal: + + git-annex version: 4.20130227 + local repository version: 3 + default repository version: 3 + supported repository versions: 3 4 + upgrade supported from repository versions: 0 1 2 + +There are also dead symlinks that point to directories that have meta-data but not the symlink target (manually line-wrapped): + + myPC:~/Downloads$ ls -l precise* + lrwxrwxrwx 1 nino nino 194 Oct 21 12:48 precise-dvd-i386.iso -> + .git/annex/objects/0x/Xz/SHA256-s3590631424-- + b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0/ + SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0 + +But looking in the symlink destination directory, there's no corresponding object, only metadata: + + myPC:~/Downloads/.git/annex/objects/0x/Xz/SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0$ ls -l + total 8 + -rw-rw-r-- 1 nino nino 30 Jan 8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.cache + -rw-rw-r-- 1 nino nino 49 Jan 8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.map + +But there is another version somewhere else. + + -r--r--r-- 1 nino nino 3590631424 Mar 18 2012 ./.git/annex/objects/Kj/wM/WORM-s3590631424-m1331991509 + --precise-dvd-i386.iso/WORM-s3590631424-m1331991509--precise-dvd-i386.iso + +This actual file does exist in the "Used" directory: + + -rw-r--r-- 1 nino nino 3.4G May 27 2012 precise-dvd-i386.iso + +I'm not so worried about the mangled repo - it's quite possibly because of clueless git/git-annex command usage - but the inability to use the fsck command is concerning + +I could just uninit everything, but as it dies prematurely, I'm not certain that all the contents would be restored. +Any thoughts on how I can get git-annex (esp. fsck) to complete would be appreciated. + +Thanks +Giovanni diff --git a/doc/forum/Out_of_memory_error_in_fsck__44___whereis__44___find_and_status_cmds.mdwn b/doc/forum/Out_of_memory_error_in_fsck__44___whereis__44___find_and_status_cmds.mdwn index 610c0e9d1a..8c54f49da5 100644 --- a/doc/forum/Out_of_memory_error_in_fsck__44___whereis__44___find_and_status_cmds.mdwn +++ b/doc/forum/Out_of_memory_error_in_fsck__44___whereis__44___find_and_status_cmds.mdwn @@ -1,78 +1,5 @@ -Before I start on what's gone wrong, many thanks for a great program: finally a way of finding out where I've put all those files, and I enjoyed your talk in Australia. Not quite to New Zealand, but a good start :-) +Please file bug reports in [[bugs]], not the forum, so they can be closed +when fixed. -To play with git-annex, I decided to convert my ~/Downloads directory to a git-annex repository, as there were a wide variety of files in it, mostly easily replaceable, and also handy to have on multiple machines. In hindsight, probably wasn't a great idea, as I'd regularly forget it was a git-annex repo and move files and directories out manually, which caused all sorts of fun trying to sort out the dead symlinks. Then I after one update, the repository format changed, from WORM to xxx256 format which meant there were now both sorts in the GA object store. - -More recently I'd tried converting the repo to direct mode, you get the idea: lots of playing with git-annex commands, and now that I think about it, possibly some git commands too trying to repair missing files. - -Anyway I've ended up with a 27GB git-annex repo that now manages to kill git-annex whenever I try to check it using "git-annex fsck". - -Not only does the fsck subcommand cause it to die, but also "find", "whereis" and "status". It dies on the same file (for find/whereis/fsck). - -e.g. - - ... lots of stuff deleted ... - whereis 1wolf14.zip (2 copies) - 051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop - 2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads) - ok - git-annex: out of memory (requested 2097152 bytes) - -Now "git status" for some repo data: - - myPC:~/Downloads$ git annex status - supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL - supported remote types: git S3 bup directory rsync web webdav glacier hook - repository mode: direct - trusted repositories: 0 - semitrusted repositories: 4 - 00000000-0000-0000-0000-000000000001 -- web - 051f0b00-e265-11e1-894e-3b0b3f3844f2 -- Laptop - 2c4e11e0-a1b4-11e1-9a02-73e17b04c00f -- here (myPC - Downloads) - 48fbe52a-a1b3-11e1-bb80-ebc15118871d -- netbk - untrusted repositories: 0 - dead repositories: 0 - transfers in progress: none - available local disk space: 100 gigabytes (+1 megabyte reserved) - local annex keys: 1719 - local annex size: 27 gigabytes - known annex keys: git-annex: out of memory (requested 2097152 bytes) - -It always seems to die at about 3.5GB memory usage. This is running on Ubuntu 12.04, using the latest GA release built using cabal: - - git-annex version: 4.20130227 - local repository version: 3 - default repository version: 3 - supported repository versions: 3 4 - upgrade supported from repository versions: 0 1 2 - -There are also dead symlinks that point to directories that have meta-data but not the symlink target (manually line-wrapped): - - myPC:~/Downloads$ ls -l precise* - lrwxrwxrwx 1 nino nino 194 Oct 21 12:48 precise-dvd-i386.iso -> - .git/annex/objects/0x/Xz/SHA256-s3590631424-- - b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0/ - SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0 - -But looking in the symlink destination directory, there's no corresponding object, only metadata: - - myPC:~/Downloads/.git/annex/objects/0x/Xz/SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0$ ls -l - total 8 - -rw-rw-r-- 1 nino nino 30 Jan 8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.cache - -rw-rw-r-- 1 nino nino 49 Jan 8 23:15 SHA256-s3590631424--b08ecdd4846948ec076b23afae7f87be9cfba5218fb9ba4160f26c0b8d4b5dd0.map - -But there is another version somewhere else. - - -r--r--r-- 1 nino nino 3590631424 Mar 18 2012 ./.git/annex/objects/Kj/wM/WORM-s3590631424-m1331991509 - --precise-dvd-i386.iso/WORM-s3590631424-m1331991509--precise-dvd-i386.iso - -This actual file does exist in the "Used" directory: - - -rw-r--r-- 1 nino nino 3.4G May 27 2012 precise-dvd-i386.iso - -I'm not so worried about the mangled repo - it's quite possibly because of clueless git/git-annex command usage - but the inability to use the fsck command is concerning - -I could just uninit everything, but as it dies prematurely, I'm not certain that all the contents would be restored. -Any thoughts on how I can get git-annex (esp. fsck) to complete would be appreciated. - -Thanks -Giovanni +Moved to [[bug report|bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds]]. +--[[Joey]]