From 48824c48fca49b9c6bf9ed34bfd45d260ec8e63b Mon Sep 17 00:00:00 2001 From: "johannes.elferich@796088e47b21d72547837ed62ae5c5cb187830b3" Date: Sat, 4 Jul 2015 02:12:52 +0000 Subject: [PATCH 1/6] Added a comment: Patched git-annex output --- ..._18b3ddff003bcdd78779b35c900964c2._comment | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment diff --git a/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment b/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment new file mode 100644 index 0000000000..3493cb7a49 --- /dev/null +++ b/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="johannes.elferich@796088e47b21d72547837ed62ae5c5cb187830b3" + nickname="johannes.elferich" + subject="Patched git-annex output" + date="2015-07-04T02:12:51Z" + content=""" +You are right something weird is going on with the GIT_ANNEX_SSHOPTION variable even though I have no idea what.... + +This is the output I get after patching: + +›git annex sync +(\"GIT_ANNEX_SSHOPTION\",Nothing) +commit (recording state in git...) +ok +pull origin fatal: protocol error: bad line length character: (\"GI +git-annex.exe: unknown command 35.89.33.17 + +Usage: git-annex command [option ...] + +"""]] From d7d21df960cc26d12561c491e43e4d6f361e7a49 Mon Sep 17 00:00:00 2001 From: "berend.van.berkum@0cc715f45d5a86c134f9192722f7adef9dc63488" Date: Sat, 4 Jul 2015 22:50:34 +0000 Subject: [PATCH 2/6] --- doc/forum/Robustness_on_failing_hardware.mdwn | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 doc/forum/Robustness_on_failing_hardware.mdwn diff --git a/doc/forum/Robustness_on_failing_hardware.mdwn b/doc/forum/Robustness_on_failing_hardware.mdwn new file mode 100644 index 0000000000..ec32cbfe70 --- /dev/null +++ b/doc/forum/Robustness_on_failing_hardware.mdwn @@ -0,0 +1,30 @@ +Hi everyone, + + +I've had a bad git-annex day and I left some musings on my first use of it. There's a question though at the end if you don't need to read that. + + +today I've spend some time trying to deal with a failing SHA256E backend. Or more precise, I'm patienting a refurbished Athlon board[*] without case and that seems to give a lot of errors. I've put up some shielding at the SDRAM which helped a lot on the checksumming, but now its the network department is having a hard time. + +This is basicly an effort to recover an old 1.5TB USB NTFS disks with bit rot, a failed ext4 and recover collections from other aging USB/desktop disks. There is some new hardware on the way, to restore another (proper) box with some ECC memory. That should fix my problems, I think. I've never dealt with a problem like this before. Before git-annex I would have been using rsync -azc or mostly -azu, but I've never had problems like this before. + +It has me pondering a bit though. To help git-annex i've done manual rsync -azc transfers, with some although a very low success--I wonder if its the lack of casing, lack of ground, or simply the non-ecc that is the cause. But that is not really the question. + +rsync does not at first glance have an option to configure the amount of retries, but -azc does try and gives a checksum failure another try before giving up. I had never seen rsync do that before even though I've been using it for a decade. I guess that shows how rare this problem should be, but its still creeping me a bit. I've never had checksummed media file collections, so I really cant be sure about integrity, can I? + +The behaviour of the SHA256E backend is a bit hysterical in this environment. Running a full annex fsck always results in some bad files. The bigger the file the harder it is to get a checksum match. I guess if I wanted to continue I should wrap the fsck step in a shell script that can reinject and fsck again and does do not needless re-checks in one run. It should figure it out in the end with a bit of more time. + +My other reaction would be to throw more checksums at it, from simple to complex. Something between filesize and sha256. I guess the whole would then have some different modes of retries and behaviours of failure maybe. Or levels of trust in the integrity. Maybe using simpler checksums would make the situation more bearable, since the problem is not disk-related iow. if the file is there and has checked out ok, then the re-fsck itself and bad memory/shielding are the culprits. + +tldr; + +git-annex has no retry at all. I have seen it can take a -i somewhere. For robustness, it would be nice if there was a rsync -c mode. It would confirm the object was transferred correctly too. My workflow is to 'annex fsck' at the remote after copying files there. Is it correct to presume that using 'annex move' in my case, would have made the sending repo dropping files even though the remote still has only corrupted copies? + +I'm new to git-annex but been wanting something like git annex for a long time. Made some attempts myself, and too pondered how to use venerable GIT but never figured something out. What I'm saying though is I may want to dig around to see what I can do with Haskell. + +I'm thinking a backend with some more checksums, maybe then some option to fsck to request only partial check of the key. Has something like a faster/less-thorough fsck using smaller checksums ever come up? And might this be a way to go at this? What do you think. + + + + +[*] Asus M2A7A-AM SE w/ AMD Athlon 64 X2 5200+ Dual Core 2.7 From 8c57b9fff7786c254c7e8756df544f100b03cde8 Mon Sep 17 00:00:00 2001 From: "xelez0@57a58225d4e5b260555ebc4a9d0a8df85d1e971a" Date: Sun, 5 Jul 2015 13:19:43 +0000 Subject: [PATCH 3/6] Added a comment: Backend of specified file --- .../comment_14_57154dcd1041a33f220f9105b709be89._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment diff --git a/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment b/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment new file mode 100644 index 0000000000..446960264b --- /dev/null +++ b/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="xelez0@57a58225d4e5b260555ebc4a9d0a8df85d1e971a" + nickname="xelez0" + subject="Backend of specified file" + date="2015-07-05T13:19:43Z" + content=""" +How can I determine backend of specified file? Looking over man pages and can't find it. +"""]] From 22a352bdf7a2b8a2122977940f6c95e95710aee1 Mon Sep 17 00:00:00 2001 From: CandyAngel Date: Sun, 5 Jul 2015 14:54:19 +0000 Subject: [PATCH 4/6] Added a comment --- ...nt_15_b3445fd1f379346c642a27211c6c798b._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment diff --git a/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment b/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment new file mode 100644 index 0000000000..3fc12afee6 --- /dev/null +++ b/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="CandyAngel" + subject="comment 15" + date="2015-07-05T14:54:18Z" + content=""" +It's not explicit, but 'git annex info $FILE' tells you the key, which has the backend as its first component: + + ## git annex info CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip + file: CG Cookie/Compositing in Blender/01_CompositingInBlender_SourceFiles.zip + size: 744.51 megabytes + key: SHA256E-s744506832--08d2daced60b5eb6509044d5eefca82e7a6899350f49adc0083014229739515e.zip + +I don't think there are any situations where the first component of the key isn't the backend, but don't hold me to that, please :) +"""]] From 4fcee356963bf710ebb3f4e1afd0cb4b37a8fb97 Mon Sep 17 00:00:00 2001 From: CandyAngel Date: Sun, 5 Jul 2015 15:00:46 +0000 Subject: [PATCH 5/6] Added a comment --- ...t_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment diff --git a/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment b/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment new file mode 100644 index 0000000000..64c0102e28 --- /dev/null +++ b/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="CandyAngel" + subject="comment 16" + date="2015-07-05T15:00:46Z" + content=""" +Or I could not be an idiot and tell you the command specifically looking up a key for a file: lookupkey + + ## git annex lookupkey CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip + SHA256E-s744506832--08d2daced60b5eb6509044d5eefca82e7a6899350f49adc0083014229739515e.zip + +So to get the backend (if the first component is always the backend): + + ## git annex lookupkey CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip | cut -d- -f1 + SHA256E +"""]] From 5ad3c9bbfd87e3dc60b57003a13bdf9396291b02 Mon Sep 17 00:00:00 2001 From: Mattt Date: Sun, 5 Jul 2015 19:40:30 +0000 Subject: [PATCH 6/6] Added a comment --- .../comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment diff --git a/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment b/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment new file mode 100644 index 0000000000..da8067ab29 --- /dev/null +++ b/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="Mattt" + subject="comment 3" + date="2015-07-05T19:40:30Z" + content=""" +Thanks CandyAngel and Joey. That helped. I have transferred pretty much my entire repo into git annex now and have it sorted the way I want. Yay! +"""]]