From 4124862ae0d75423f21b4050bb18f999f86f315e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 12 Oct 2020 15:47:46 -0400 Subject: [PATCH 1/2] comment This commit was sponsored by Ethan Aubin. --- ..._774d540ce6f5c3ffda924159e146721e._comment | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment diff --git a/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment b/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment new file mode 100644 index 0000000000..5371142eac --- /dev/null +++ b/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment @@ -0,0 +1,43 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2020-10-12T19:19:40Z" + content=""" +Thinking a little more about this, the lazy bytestring it reads is probably +in around 32kb chunks. The git ls-files --stage output segment for a file +is 50 bytes plus the filename, so probably under 200 bytes. + +The lazy bytestring is split into those segments, and then each segment +is coopied to a strict bytestring. The attoparsec parser I think does not +copy, so the parsed result will be the size of the original strict +bytestring. + +But hmm, does L.toStrict copy the whole chunk or chunks of the lazy +bytestring and make a strict bytestring out of that? If it does, +that means each 32kb chunk will get copied many times, probably 150+! + +Well, how does a lazy bytestring get split on null? L.split uses L.take. +L.take uses S.take on the chunk. S.take simply updates the length of +the bytestring, but the result still keeps the rest of it allocated. +(And similar for drop I assume.) + +So, if L.toStrict is run on a lazy bytestring consisting of a single chunk +that's a strict bytestring, that's had its size reduced by L.take, the +rest is still allocated. And in L.toStrict, there's a special case for a +single chunk input, that bypasses the usual copying: + + goLen1 _ bs Empty = bs + +So, that keeps the original strict bytestring, not copying it. And so +the rest of it, after the NULL, remains allocated! + +This is surprising behavior. Could even be a bug. L.toStrict does +say that it copies all the data, but not that it pins data that is not +even part of the input bytestring as far as the user is concerned. + +So that explains the PINNED memory use. + +So, I think git-annex needs to stop using L.toStrict here +(and probably everywhere involving streaming any amount of data), +there are some other ones. +"""]] From c61d282b72ffe581a3af7884534a29a9a9f0c486 Mon Sep 17 00:00:00 2001 From: Maximilian Date: Mon, 12 Oct 2020 19:53:27 +0000 Subject: [PATCH 2/2] Added a comment --- ...comment_3_57ee489f24b26b73b28f64566e2b8ff5._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/Cannot_get_unlocked_file_contents/comment_3_57ee489f24b26b73b28f64566e2b8ff5._comment diff --git a/doc/forum/Cannot_get_unlocked_file_contents/comment_3_57ee489f24b26b73b28f64566e2b8ff5._comment b/doc/forum/Cannot_get_unlocked_file_contents/comment_3_57ee489f24b26b73b28f64566e2b8ff5._comment new file mode 100644 index 0000000000..b48377726d --- /dev/null +++ b/doc/forum/Cannot_get_unlocked_file_contents/comment_3_57ee489f24b26b73b28f64566e2b8ff5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Maximilian" + avatar="http://cdn.libravatar.org/avatar/e7add028a8da87278c36a3d7b9f1cd60" + subject="comment 3" + date="2020-10-12T19:53:27Z" + content=""" +@Lukey: thanks for the tip. Will try this at some point with a new repo. For now I just copied the most recent version of the file from /annex/objects/ of my central repo. + +@joey: I am afraid I committed the hash-link contents: 'git annex get' didn't change anything. Still I have no idea how this could happen; I'll dig through the history and report back :) +"""]]