git-annex/doc/todo/improve_memory_usage_of_--all.mdwn

`git annex unused --from=$remote` and `git annex info $remote`
buffer the list of keys that have uncommitted journalled changes
in memory. This is due to Annex.Branch.files's which reads all the
files in the journal into a buffer.

Note that the list of keys in the branch *does* stream in, so this
is only really a problem when using annex.alwayscommit=false to build
up big git-annex branch commits via the journal. Or using annex.private,
since the private journal can build up a lot of keys in it.

An attempt at making it stream via unsafeInterleaveIO failed miserably
and that is not the right approach. This would be a good place to use
ResourceT, but it might need some changes to the Annex monad to allow
combining the two. --[[Joey]]

> This used to also affect --all and using git-annex in a bare repo, but
> that was avoided by using the overBranchFileContents interface. This
> suggests that changing to that interface in unused and info would be a
> solution.

[[!tag confirmed]]

[[!meta title="improve memory usage of unused and info when the journal contains a lot of files"]]
Improve memory use of --all when using annex.private This does not improve Annex.Branch.files at all, since it still uses ++ to combine the lists, so forcing all but the last one. But when there are a lot of files in the private journal, it does avoid --all (or a bare repo) from buffering the filenames in memory. See commit 653b7194727a27dfd185cdb0b752f8e190250fc4 for prior discussion of this buffering. Sponsored-by: Graham Spencer on Patreon 2023-10-24 17:06:54 +00:00			`git annex unused --from=$remote` and `git annex info $remote`
			`buffer the list of keys that have uncommitted journalled changes`
			`in memory. This is due to Annex.Branch.files's which reads all the`
			`files in the journal into a buffer.`
back out incorrect IO interleaving change Fix regression in last release that crashes when using --all or running git-annex in a bare repository. May have also affected git-annex unused and git-annex info. Reversed the order of the (++) in Annex.Branch.files so --all will stream lazily still when there are not a bunch of uncommitted journal files. Added a todo to maybe improve this later. This commit was sponsored by Trenton Cronholm on Patreon. 2018-05-08 17:54:42 +00:00
			`Note that the list of keys in the branch does stream in, so this`
			`is only really a problem when using annex.alwayscommit=false to build`
Improve memory use of --all when using annex.private This does not improve Annex.Branch.files at all, since it still uses ++ to combine the lists, so forcing all but the last one. But when there are a lot of files in the private journal, it does avoid --all (or a bare repo) from buffering the filenames in memory. See commit 653b7194727a27dfd185cdb0b752f8e190250fc4 for prior discussion of this buffering. Sponsored-by: Graham Spencer on Patreon 2023-10-24 17:06:54 +00:00			`up big git-annex branch commits via the journal. Or using annex.private,`
			`since the private journal can build up a lot of keys in it.`
back out incorrect IO interleaving change Fix regression in last release that crashes when using --all or running git-annex in a bare repository. May have also affected git-annex unused and git-annex info. Reversed the order of the (++) in Annex.Branch.files so --all will stream lazily still when there are not a bunch of uncommitted journal files. Added a todo to maybe improve this later. This commit was sponsored by Trenton Cronholm on Patreon. 2018-05-08 17:54:42 +00:00
			`An attempt at making it stream via unsafeInterleaveIO failed miserably`
			`and that is not the right approach. This would be a good place to use`
			`ResourceT, but it might need some changes to the Annex monad to allow`
			`combining the two. --[[Joey]]`
tagged the past 2 years of open todos and followed up to a few of them also moved some that were really bug reports to bugs/ and closed a couple 2020-01-30 19:22:05 +00:00
Improve memory use of --all when using annex.private This does not improve Annex.Branch.files at all, since it still uses ++ to combine the lists, so forcing all but the last one. But when there are a lot of files in the private journal, it does avoid --all (or a bare repo) from buffering the filenames in memory. See commit 653b7194727a27dfd185cdb0b752f8e190250fc4 for prior discussion of this buffering. Sponsored-by: Graham Spencer on Patreon 2023-10-24 17:06:54 +00:00			`> This used to also affect --all and using git-annex in a bare repo, but`
			`> that was avoided by using the overBranchFileContents interface. This`
			`> suggests that changing to that interface in unused and info would be a`
			`> solution.`

tagged the past 2 years of open todos and followed up to a few of them also moved some that were really bug reports to bugs/ and closed a couple 2020-01-30 19:22:05 +00:00			`[[!tag confirmed]]`
Improve memory use of --all when using annex.private This does not improve Annex.Branch.files at all, since it still uses ++ to combine the lists, so forcing all but the last one. But when there are a lot of files in the private journal, it does avoid --all (or a bare repo) from buffering the filenames in memory. See commit 653b7194727a27dfd185cdb0b752f8e190250fc4 for prior discussion of this buffering. Sponsored-by: Graham Spencer on Patreon 2023-10-24 17:06:54 +00:00
			`[[!meta title="improve memory usage of unused and info when the journal contains a lot of files"]]`