From 06c49539a2c25958468b367a8885968d34a03849 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 16:32:32 +0000 Subject: [PATCH 01/10] Added a comment: Version --- ...ent_3_341b47663d133411587ec70ef2b178c6._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_3_341b47663d133411587ec70ef2b178c6._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_3_341b47663d133411587ec70ef2b178c6._comment b/doc/forum/Recover_files__44___annex_stuck/comment_3_341b47663d133411587ec70ef2b178c6._comment new file mode 100644 index 0000000000..77a7c93c8b --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_3_341b47663d133411587ec70ef2b178c6._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="Version" + date="2014-06-18T16:32:32Z" + content=""" +Hi Joeyh, + +Thanks for the reply. I am using git version 2.0.0.390.gcb682f8, not sure what version of git-annex, but I downloaded it from github about 20 minutes ago. + +Thanks! + +-Mike +"""]] From 450dd10b8f28cf2eccbbd549298c6b15696c09a7 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 18 Jun 2014 16:35:29 +0000 Subject: [PATCH 02/10] Added a comment --- ...comment_4_66c0d9284d5edbac189a64b03c4fe50a._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_4_66c0d9284d5edbac189a64b03c4fe50a._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_4_66c0d9284d5edbac189a64b03c4fe50a._comment b/doc/forum/Recover_files__44___annex_stuck/comment_4_66c0d9284d5edbac189a64b03c4fe50a._comment new file mode 100644 index 0000000000..5d153e2481 --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_4_66c0d9284d5edbac189a64b03c4fe50a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.203" + subject="comment 4" + date="2014-06-18T16:35:29Z" + content=""" +You can find out the version of git-annex by running: git-annex version + +You can find out if your repository is in direct or indirect mode by running: git config annex.direct +"""]] From 377614c8d13a07e6b6c05a541f128202b35b413a Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 16:40:14 +0000 Subject: [PATCH 03/10] Added a comment: Versions --- ...5_8b32f6597f447f88bee7a80698fb4df6._comment | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_5_8b32f6597f447f88bee7a80698fb4df6._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_5_8b32f6597f447f88bee7a80698fb4df6._comment b/doc/forum/Recover_files__44___annex_stuck/comment_5_8b32f6597f447f88bee7a80698fb4df6._comment new file mode 100644 index 0000000000..cf656f3eae --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_5_8b32f6597f447f88bee7a80698fb4df6._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="Versions" + date="2014-06-18T16:40:14Z" + content=""" +``git-annex version`` returns: + + git-annex version: 5.20140618-gc2f1c63 + build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash + key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL + remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external + local repository version: unknown + supported repository version: 5 + upgrade supported from repository versions: 0 1 2 4 + +``git config annex.direct`` exits with error code 1 and doesn't return any information, however I never explicitly set direct mode, and the repository is all symlinked, so my assumption is that it is in indirect mode. Would direct mode be better for such a large repo? +"""]] From 62880aee47258cdeb492b48339b6418310c38c74 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 16:40:24 +0000 Subject: [PATCH 04/10] Added a comment: Versions --- ...6_8a14468908f99be2ca4afd73f8a4fcc7._comment | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment b/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment new file mode 100644 index 0000000000..d5eff4fc61 --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="Versions" + date="2014-06-18T16:40:24Z" + content=""" +``git-annex version`` returns: + + git-annex version: 5.20140618-gc2f1c63 + build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash + key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL + remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external + local repository version: unknown + supported repository version: 5 + upgrade supported from repository versions: 0 1 2 4 + +``git config annex.direct`` exits with error code 1 and doesn't return any information, however I never explicitly set direct mode, and the repository is all symlinked, so my assumption is that it is in indirect mode. Would direct mode be better for such a large repo? +"""]] From 2c8d24cc5f7f1cfc7f2fe912c3aa7606f107cc8f Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 16:41:23 +0000 Subject: [PATCH 05/10] removed --- ...6_8a14468908f99be2ca4afd73f8a4fcc7._comment | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment b/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment deleted file mode 100644 index d5eff4fc61..0000000000 --- a/doc/forum/Recover_files__44___annex_stuck/comment_6_8a14468908f99be2ca4afd73f8a4fcc7._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" - nickname="Mike" - subject="Versions" - date="2014-06-18T16:40:24Z" - content=""" -``git-annex version`` returns: - - git-annex version: 5.20140618-gc2f1c63 - build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash - key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL - remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external - local repository version: unknown - supported repository version: 5 - upgrade supported from repository versions: 0 1 2 4 - -``git config annex.direct`` exits with error code 1 and doesn't return any information, however I never explicitly set direct mode, and the repository is all symlinked, so my assumption is that it is in indirect mode. Would direct mode be better for such a large repo? -"""]] From ea5c5e77b5ec86e6652f9cde049133ab1fdf4781 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 16:49:35 +0000 Subject: [PATCH 06/10] Added a comment: Reinstall GHC or Cabal? --- .../comment_6_4cc81169e99a453cdb6e83e57e638f37._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_6_4cc81169e99a453cdb6e83e57e638f37._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_6_4cc81169e99a453cdb6e83e57e638f37._comment b/doc/forum/Recover_files__44___annex_stuck/comment_6_4cc81169e99a453cdb6e83e57e638f37._comment new file mode 100644 index 0000000000..52e04dfc85 --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_6_4cc81169e99a453cdb6e83e57e638f37._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="Reinstall GHC or Cabal?" + date="2014-06-18T16:49:35Z" + content=""" +Do I need to reinstall ghc or cabal with rtsopts enabled somehow in order to be able to compile git-annex with -K1000m? +"""]] From 6ff890dddc6381c1ab889cb8b4bbf9f200a4be57 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 18 Jun 2014 17:14:40 +0000 Subject: [PATCH 07/10] Added a comment --- ..._7_2d104cf4682e04906f8ca0ced7288cf1._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_7_2d104cf4682e04906f8ca0ced7288cf1._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_7_2d104cf4682e04906f8ca0ced7288cf1._comment b/doc/forum/Recover_files__44___annex_stuck/comment_7_2d104cf4682e04906f8ca0ced7288cf1._comment new file mode 100644 index 0000000000..6ea7b45f64 --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_7_2d104cf4682e04906f8ca0ced7288cf1._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.203" + subject="comment 7" + date="2014-06-18T17:14:40Z" + content=""" +Ok, so the repository is in indirect mode, and this rules out a large quantity of problems that could have been caused by direct mode (no, I don't recommend using direct mode). + +If you want to build git-annex with the +RTS option enabled, you just need to pass -rtsopts to ghc when building git-annex. (Not -with-rtsopts ...) +That *might* let you pump up the memory and bypass whatever the problem is, or at least find out how much memory it's trying to allocate, which might be a useful clue. But I would be much more interested in debugging and fixing the actual problem, since git-annex should not normally need to allocate a 8+ mb chunk of memory. + +The \"No HEAD commit to compare with (yet)\" failure mode was removed from git in 2011. You must have been using old versions of git and git-annex before you upgraded. Perhaps they have left the repository in some broken state. + +What size does `du -hsc .git/objects` report? How about `du -h .git/index`? + +Are git commands that do not involve git-annex still taking a long time to run or failing in some way? (Note that `git commit` has a hook that runs git-annex; you can bypass that with `git commit --no-verify`) +"""]] From ba37e076679e0128000bf074ab8ff245bef373b5 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" Date: Wed, 18 Jun 2014 17:31:58 +0000 Subject: [PATCH 08/10] Added a comment --- ..._d356c4fce9f1197e5292f9dedf85bbc9._comment | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_8_d356c4fce9f1197e5292f9dedf85bbc9._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_8_d356c4fce9f1197e5292f9dedf85bbc9._comment b/doc/forum/Recover_files__44___annex_stuck/comment_8_d356c4fce9f1197e5292f9dedf85bbc9._comment new file mode 100644 index 0000000000..df1af4f06b --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_8_d356c4fce9f1197e5292f9dedf85bbc9._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="comment 8" + date="2014-06-18T17:31:58Z" + content=""" +The git available through yum is git 1.7.1, which looks like it was released in 2010 or earlier. (I really wish I had a different version of linux on this server). It is possible that in some way screwed up the repo. + +I figured out how to compile cabal and git-annex with rtsopts, so I can now set higher memory levels, but I am happy to help debug the problem too, as I would really love a fully functional git-annex. + +git commands now run quickly, thanks to the new git I think. + +``du -hsc .git/objects`` returns: ``8.1G .git/objects`` + +``du -h .git/index`` returns: ``437M .git/index`` + +I am currently running the command ``git-annex +RTS -K1000m -RTS add .``, it is chugging away doing something, but is not printing any messages yet after 11 minutes of running, it is a 6TB directory though, and there are a lot of concurrent IO operations on that disk right now. + +I am also running ``du -h --max-depth=1`` on the root repo directory, and also ``find | wc -l``, so that I can tell you the exact size of the dir and the total number of files too. These operations combined may take more than an hour though, I will send details when the commands complete. + +Let me know if you want me to stop the ``git-annex +RTS -K1000m -RTS add .`` command and run git-annex some other way. +"""]] From 5aeba7f90909f0cd1ec0dc1dfa76e63e9979833d Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 18 Jun 2014 17:36:44 +0000 Subject: [PATCH 09/10] Added a comment --- ...mment_9_856c7e1575f5d99530ecd54004315487._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/Recover_files__44___annex_stuck/comment_9_856c7e1575f5d99530ecd54004315487._comment diff --git a/doc/forum/Recover_files__44___annex_stuck/comment_9_856c7e1575f5d99530ecd54004315487._comment b/doc/forum/Recover_files__44___annex_stuck/comment_9_856c7e1575f5d99530ecd54004315487._comment new file mode 100644 index 0000000000..c092d99f50 --- /dev/null +++ b/doc/forum/Recover_files__44___annex_stuck/comment_9_856c7e1575f5d99530ecd54004315487._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.203" + subject="comment 9" + date="2014-06-18T17:36:44Z" + content=""" +Both of those du's look extremely large. How many files are listed by `git ls-files --cached | wc -l` ? + +I don't think that there's any point in running `git annex add` while you're still having some problem. I am curious though how much memory the git-annex add you have running has used. + +If I were you, I'd look in .git/objects for large files (> 100kb, say). +"""]] From cef5c48e8792ffbcc32c453de6f978cdea6934ed Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 18 Jun 2014 18:12:47 +0000 Subject: [PATCH 10/10] Added a comment --- ..._3f779c4d0c9fb27532b2981bd3ad4eee._comment | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/bugs/files_lost_during_upgrade/comment_1_3f779c4d0c9fb27532b2981bd3ad4eee._comment diff --git a/doc/bugs/files_lost_during_upgrade/comment_1_3f779c4d0c9fb27532b2981bd3ad4eee._comment b/doc/bugs/files_lost_during_upgrade/comment_1_3f779c4d0c9fb27532b2981bd3ad4eee._comment new file mode 100644 index 0000000000..b510d92c1f --- /dev/null +++ b/doc/bugs/files_lost_during_upgrade/comment_1_3f779c4d0c9fb27532b2981bd3ad4eee._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.203" + subject="comment 1" + date="2014-06-18T18:12:47Z" + content=""" +So [[bugs/git-annex_auto_upgrade_is_redundant]] strikes again. I have fixed up the versions to avoid this upgrade loop again. I would rather discuss any problems with where the automatic upgrade code puts git-annex in a separate bug report than this one.. (If you're running git-annex as a user that is not allowed to write to the directory where you installed it in the first place then yes, it cannot write there and will instead unpack itself into the home directory.) + +There are many strange things in the log, but this is probably the strangest: + +> fatal: Could not switch to '/home/jwiklund/Documents/.git/annex/merge/': No such file or directory + +This is the gitAnnexMergeDir used for direct mode merges. Since git-annex always creates that directory before starting a merge, I don't know how it could fail to exist. + +Other strange things: + +* The assistant fixes stale index.lock files on startup. But the log shows the assistant starting up and shortly thereafter there's a stale index.lock file. All I can think is that perhaps two git processes are trying to commit at the same time. + +* \"fatal: pathspec 'music.txt' did not match any files\", apparently output by git-add. + +* \"error: duplicate parent 294b61a3dce1e87a62e4d675deac2a9130b819e6 ignored\", which would happen if git-commit-tree were passed duplicate parent refs. AFAICS, the only place in git-annex that might do that is when it commits the git-annex branch. (While this says it's an error, it really is ignored, and git-commit-tree continues and makes a valid commit.) +"""]]