Avoid using a lot of memory when large objects are present in the git repository
.. and have to be checked to see if they are a pointed to an annexed file. Cases where such memory use could occur included, but were not limited to: - git commit -a of a large unlocked file (in v5 mode) - git-annex adjust when a large file was checked into git directly Generally, any use of catKey was a potential problem. Fix by using git cat-file --batch-check to check size before catting. This adds another git batch process, which is included in the CatFileHandle for simplicity. There could be performance impact, anywhere catKey is used. Particularly likely to affect adjusted branch generation speed, and operations on unlocked files in v6 mode. Hopefully since the --batch-check and --batch read the same data, disk buffering will avoid most overhead. Leaving only the overhead of talking to the process over the pipe and whatever computation --batch-check needs to do. This commit was sponsored by Bruno BEAUFILS on Patreon.
This commit is contained in:
parent
672e53bded
commit
34530e59d9
5 changed files with 117 additions and 30 deletions
|
@ -49,6 +49,11 @@ catObject ref = do
|
|||
h <- catFileHandle
|
||||
liftIO $ Git.CatFile.catObject h ref
|
||||
|
||||
catObjectMetaData :: Git.Ref -> Annex (Maybe (Integer, ObjectType))
|
||||
catObjectMetaData ref = do
|
||||
h <- catFileHandle
|
||||
liftIO $ Git.CatFile.catObjectMetaData h ref
|
||||
|
||||
catTree :: Git.Ref -> Annex [(FilePath, FileMode)]
|
||||
catTree ref = do
|
||||
h <- catFileHandle
|
||||
|
@ -89,7 +94,14 @@ catFileStop = do
|
|||
|
||||
{- From ref to a symlink or a pointer file, get the key. -}
|
||||
catKey :: Ref -> Annex (Maybe Key)
|
||||
catKey ref = parseLinkOrPointer <$> catObject ref
|
||||
catKey ref = go =<< catObjectMetaData ref
|
||||
where
|
||||
go (Just (sz, _))
|
||||
-- Avoid catting large files, that cannot be symlinks or
|
||||
-- pointer files, which would require buffering their
|
||||
-- content in memory, as well as a lot of IO.
|
||||
| sz <= maxPointerSz = parseLinkOrPointer <$> catObject ref
|
||||
go _ = return Nothing
|
||||
|
||||
{- Gets a symlink target. -}
|
||||
catSymLinkTarget :: Sha -> Annex String
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue