From 99a1e6efe2e598769ed33c94d29f15a1572f7ee3 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 9 Jun 2017 13:43:53 -0400 Subject: [PATCH] close as dup --- ...n_adding_large_files_to_v6_repository.mdwn | 2 ++ ..._b9154c38406fca40c4c0edb716707c3a._comment | 19 +++++++++++++++++++ doc/todo/smudge.mdwn | 6 ++++++ 3 files changed, 27 insertions(+) create mode 100644 doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment diff --git a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn index ae5c0e6e44..568b2bf5f4 100644 --- a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn +++ b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn @@ -53,3 +53,5 @@ I have no idea why it needs to do that, though. ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) I've been using git-annex v5 repositories without any issues, and with smaller files, repository v6 works great! + +> dup; [[done]] --[[Joey]] diff --git a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment new file mode 100644 index 0000000000..9ef59ed710 --- /dev/null +++ b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2017-06-09T17:34:40Z" + content=""" +Unfortunately, git add tries to load the whole file content +into memory (or at least allocates a buffer for it all), +even if it's only going to stream it through the clean filter +used in v6 mode, and even though the git-annex smudge +filter reads the file content from disk itself. + +I submitted a patch to git over a year ago, that IIRC fixes this kind of +problem, but it was not accepted. Getting an updated version of that patch +accepted into git is the main blocker for v6 repositories to not be +experimental. + +[[todo/smudge]] documents this problem. I'm going to close this bug +as it's a duplicate of stuff discussed there. +"""]] diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn index 6117224903..266a5d5933 100644 --- a/doc/todo/smudge.mdwn +++ b/doc/todo/smudge.mdwn @@ -71,6 +71,12 @@ git-annex should use smudge/clean filters. avoid the problem for git checkout, since it would use the new interface and not the smudge filter.) +* When `git add` is run with a large file, it allocates memory for + the whole file content, even though it's only going + to stream it to the clean filter. My proposed smudge/clean + interface patch also fixed this problem, since it made git not read + the file at all. + * Eventually (but not yet), make v6 the default for new repositories. Note that the assistant forces repos into direct mode; that will need to be changed then, and it should enable annex.thin instead.