From d40584c63619eba7e81b5e3a6bc9dd20f7107f80 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 18:49:05 +0000 Subject: [PATCH 01/10] Added a comment --- ...nt_3_412941e9385f63153b23695641e71deb._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_3_412941e9385f63153b23695641e71deb._comment diff --git a/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_3_412941e9385f63153b23695641e71deb._comment b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_3_412941e9385f63153b23695641e71deb._comment new file mode 100644 index 0000000000..6349500caa --- /dev/null +++ b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_3_412941e9385f63153b23695641e71deb._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 3" + date="2013-09-19T18:49:05Z" + content=""" +git-annex sync is a wrapper around git commit. But not -a! git commit -a will stage every one of your large files directly into the git repository, wasting much memory and worse, disk space. It is ok to use `git commit` or `git commit --staged` in direct mode after eg `git annex add`. But not `git commit -a` or `git commit even-only-one-of-the-files`. It's best to just use `git annex sync` rather than `git commit`, as it avoids finger memory causing you to run the wrong type of commit command. Please see [[direct_mode]] for the details. + +I was able to make pre-commit take a lot of memory by committing a 1 gb file directly to git. git-annex was buffering the whole file content in memory +due to not thinking to check first if it was a symlink. I have fixed that bug. + +So I think you must have run that command you showed, and you now have a lot of data stored in your git repository that you had meant git-annex to handle. You might need to use git-filter-branch to remove it.. + +This kind of thing is why I need to write the [[todo/direct_mode_guard]]. +"""]] From 7e317bcf039d195de3d3c003758504ac6a67fdb8 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 18:53:14 +0000 Subject: [PATCH 02/10] Added a comment --- ...comment_3_9f83ef190547b291a715cda55b7977d4._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/merge_causes_out_of_memory_on_large_repos/comment_3_9f83ef190547b291a715cda55b7977d4._comment diff --git a/doc/bugs/merge_causes_out_of_memory_on_large_repos/comment_3_9f83ef190547b291a715cda55b7977d4._comment b/doc/bugs/merge_causes_out_of_memory_on_large_repos/comment_3_9f83ef190547b291a715cda55b7977d4._comment new file mode 100644 index 0000000000..11f03b39d9 --- /dev/null +++ b/doc/bugs/merge_causes_out_of_memory_on_large_repos/comment_3_9f83ef190547b291a715cda55b7977d4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 3" + date="2013-09-19T18:53:14Z" + content=""" +I just fixed a memory leak that occurred when large files were checked directly into git (perhaps by accident by committing files manually when using direct mode). However, I don't think it affected `git annex merge`. + +Still need information about how to reproduce this bug... +"""]] From a618301675e933bda0c6d422b1176c9f1f82a8b8 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 18:56:39 +0000 Subject: [PATCH 03/10] Added a comment: Ed's right --- .../comment_2_3df7fcbcd482bb9377ead238b314995b._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Forcing_one_repo_to_contain_a_copy_of_all_files/comment_2_3df7fcbcd482bb9377ead238b314995b._comment diff --git a/doc/forum/Forcing_one_repo_to_contain_a_copy_of_all_files/comment_2_3df7fcbcd482bb9377ead238b314995b._comment b/doc/forum/Forcing_one_repo_to_contain_a_copy_of_all_files/comment_2_3df7fcbcd482bb9377ead238b314995b._comment new file mode 100644 index 0000000000..e59af71fbd --- /dev/null +++ b/doc/forum/Forcing_one_repo_to_contain_a_copy_of_all_files/comment_2_3df7fcbcd482bb9377ead238b314995b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="Ed's right" + date="2013-09-19T18:56:39Z" + content=""" +What we don't have is something to enforce a particular preferred content setting. Ie, refuse to allow dropping a file from a repo if it's preferred content there. Currently you have to remember to use `git annex drop --auto` to cause the preferred content to be honored. +"""]] From 5d4a568478ba62793098c75c7b75d37c46ba265e Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 18:59:42 +0000 Subject: [PATCH 04/10] Added a comment --- ...comment_3_a07105226ef3488b97731db004651976._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status/comment_3_a07105226ef3488b97731db004651976._comment diff --git a/doc/bugs/Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status/comment_3_a07105226ef3488b97731db004651976._comment b/doc/bugs/Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status/comment_3_a07105226ef3488b97731db004651976._comment new file mode 100644 index 0000000000..e4795e5a83 --- /dev/null +++ b/doc/bugs/Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status/comment_3_a07105226ef3488b97731db004651976._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 3" + date="2013-09-19T18:59:42Z" + content=""" +It turns out that if you are using direct mode and you manually `git commit some-file` that is really big, that file is checked directly into git. Which is why direct mode's docs say not to do that. + +Once you've got an enormous file in git, either on purpose on by mistake, it turns out that git-annex tries to buffer the whole file content in some situations. I am in the process of fixing those memory leaks. +"""]] From cf756ff7b3fcf3af2b0d9555e574584634cb4884 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 19:07:04 +0000 Subject: [PATCH 05/10] Added a comment --- .../comment_7_b91f4a87b6d29ae6b4262922fd65a79d._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds/comment_7_b91f4a87b6d29ae6b4262922fd65a79d._comment diff --git a/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds/comment_7_b91f4a87b6d29ae6b4262922fd65a79d._comment b/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds/comment_7_b91f4a87b6d29ae6b4262922fd65a79d._comment new file mode 100644 index 0000000000..c7832477c5 --- /dev/null +++ b/doc/bugs/Out_of_memory_error_in_fsck_whereis_find_and_status_cmds/comment_7_b91f4a87b6d29ae6b4262922fd65a79d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 7" + date="2013-09-19T19:07:03Z" + content=""" +[[Huge_annex_out_of_memory_on_switch_to_indirect_mode_and_status]] is a duplicate of this bug. +"""]] From ea5ff63c0f061bdb27750e32deb1adbf9d2859f7 Mon Sep 17 00:00:00 2001 From: ringprince Date: Thu, 19 Sep 2013 20:06:39 +0000 Subject: [PATCH 06/10] Added a comment --- ...comment_4_112ad140d9006c530db2121bec24de30._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_4_112ad140d9006c530db2121bec24de30._comment diff --git a/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_4_112ad140d9006c530db2121bec24de30._comment b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_4_112ad140d9006c530db2121bec24de30._comment new file mode 100644 index 0000000000..e5ebfa3281 --- /dev/null +++ b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_4_112ad140d9006c530db2121bec24de30._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="ringprince" + ip="134.76.140.110" + subject="comment 4" + date="2013-09-19T20:06:39Z" + content=""" +Let me ask a simple question: All files are regular files (no symlinks). If I can live with loosing the history, is it save to just remove the .git directory and start over? Or do I risk anything? + +PS: .git is about 7GB +"""]] From 1f28ae379574999ed1d6b00e05fc35a39b9f5fa8 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 20:57:19 +0000 Subject: [PATCH 07/10] Added a comment --- .../comment_5_9178023b95683a649355f291165a1467._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_5_9178023b95683a649355f291165a1467._comment diff --git a/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_5_9178023b95683a649355f291165a1467._comment b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_5_9178023b95683a649355f291165a1467._comment new file mode 100644 index 0000000000..7f5047ddd6 --- /dev/null +++ b/doc/forum/git-annex_pre-commit_eats_all_my_4GB_of_ram/comment_5_9178023b95683a649355f291165a1467._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 5" + date="2013-09-19T20:57:18Z" + content=""" +You can safely delete the git repository in the situation you describe. Better yet: Tar it up, and check the tarball into git-annex with the other files when you start over. :) Then if I'm wrong, you still have that data. +"""]] From b764f52b510c13daeffe258330fdd998da413c61 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 20:59:32 +0000 Subject: [PATCH 08/10] Added a comment --- ...mment_1_12b578689eb8d5d38c06261ec65e2109._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/purge_files_with_no_copies/comment_1_12b578689eb8d5d38c06261ec65e2109._comment diff --git a/doc/forum/purge_files_with_no_copies/comment_1_12b578689eb8d5d38c06261ec65e2109._comment b/doc/forum/purge_files_with_no_copies/comment_1_12b578689eb8d5d38c06261ec65e2109._comment new file mode 100644 index 0000000000..0ea81ecfe7 --- /dev/null +++ b/doc/forum/purge_files_with_no_copies/comment_1_12b578689eb8d5d38c06261ec65e2109._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 1" + date="2013-09-19T20:59:32Z" + content=""" +This will output all files in your repository that git-annex thinks have no copies left: + +`git annex find --not --copies=1` + +Piping it to git rm is left as an excercise for the reader.. +"""]] From 591d6a3281e52d932e0b6f3a5d85bf58db2416c7 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 21:01:30 +0000 Subject: [PATCH 09/10] Added a comment --- .../comment_1_6813fdc7ecc98765a5d35d34163a1712._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment diff --git a/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment new file mode 100644 index 0000000000..89fe4c0690 --- /dev/null +++ b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 1" + date="2013-09-19T21:01:30Z" + content=""" +That might be useful in some cases. git-annex can only tell how much free space is available on remotes that are mounted to the local filesystem. Did you use case involve removable drives? +"""]] From 475270f4019e9cd9655a16a763c7e5ca4d890a6c Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 19 Sep 2013 21:07:35 +0000 Subject: [PATCH 10/10] Added a comment --- ...ent_4_59857879abaae22bde444a215e00bf18._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment diff --git a/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment b/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment new file mode 100644 index 0000000000..db217cff97 --- /dev/null +++ b/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 4" + date="2013-09-19T21:07:35Z" + content=""" +If you have the XMPP flag in your git-annex build, it will support XMPP. Are you sure you set up the xmpp creds file and the xmpp special remote correctly on the ARM machine? (I assume it has no webapp, so you had to set that up manually..) + +Here's how you can do that manually: + +1. Run git-annex on a machine with the webapp, set up XMPP, and copy the .git/annex/creds/xmpp to the machine without the webapp. +2. On the machine without the webapp, add a git remote that has its \"url = xmpp::loginname@xmppserver.com\" and its annex-uuid set to the annex.uuid of the repository on the first machine. +3. Run git-annex assistant on the machine without the webapp. +"""]]