diff --git a/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_4_94b241ec93018adce716ceeed4bffd44._comment b/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_4_94b241ec93018adce716ceeed4bffd44._comment new file mode 100644 index 0000000000..e1cf41e901 --- /dev/null +++ b/doc/bugs/git-annex-import_stalls_and_uses_all_ram_available/comment_4_94b241ec93018adce716ceeed4bffd44._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2024-11-15T19:29:52Z" + content=""" +FWIW, I've made some improvements that should make it need around 80% less +memory in this case. Which might be enough to let it import. + +Still don't have filtering on preferred contents on the fly though. +"""]] diff --git a/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_8_017cf9156e94b1587f1853504d6c2de1._comment b/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_8_017cf9156e94b1587f1853504d6c2de1._comment new file mode 100644 index 0000000000..e6b9800971 --- /dev/null +++ b/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_8_017cf9156e94b1587f1853504d6c2de1._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 8""" + date="2024-11-15T17:48:08Z" + content=""" +Did same memory optimisation for the versioned case, and the results are +striking! Running the command until it had made 45 API requests, it was +using 592788 kb of memory. Now it uses only 110968 kb. + +Of that, about 78900 kb are used at startup, so it grew 29836 kb. +At that point, it has gathered 23537 changes. So about 1 kb is used per +change. That seems a bit more memory than really should be needed, +each change takes about 75 bytes of data, eg: + + "y3RixvrmLvr1oWJ7meEa4vWK6B.C.aad",3340,"dandisets/000003/draft/dandiset.jsonld",2021-09-28 02:12:39 UTC + +I did try some further memory optimisation, making it avoid storing the +same filename repeatedly in memory when gathering versioned changes. Which +oddly didn't save any memory. + +Memory profiling might let this be improved further, but needing 1 gb of +memory to import a million changes to files doesn't seem too bad. +"""]]