mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Annex/Content/PointerFile.hs

76 lines
2.3 KiB
Haskell
Raw Normal View History

split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
{- git-annex pointer files
-
- Copyright 2010-2018 Joey Hess <id@joeyh.name>
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
-}
{-# LANGUAGE CPP #-}
module Annex.Content.PointerFile where
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
#if ! defined(mingw32_HOST_OS)
import System.Posix.Files
#else
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
import System.PosixCompat.Files
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
#endif
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
import Annex.Common
import Annex.Perms
import Annex.Link
import Annex.ReplaceFile
import Annex.InodeSentinal
import Annex.Content.LowLevel
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
import Utility.InodeCache
avoid build warning on windows
2020-01-01 18:40:00 +00:00
#if ! defined(mingw32_HOST_OS)
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
import Utility.Touch
avoid build warning on windows
2020-01-01 18:40:00 +00:00
#endif
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
fix race If a pointer file is being populated and something modifies it at the same time, there was a race there the modified file's InodeCache could get added into the keys database. Note that replaceFile normally renames the temp file into place, so the inode cache caculated for the temp file will still be good. If it has to fall back to a copy, the worktree file won't be put in the inode cache. This has the same result as if the worktree file gets touched, and will be handled the same way. Eg, when dropping, isUnmodified will do an expensive comparison and notice that the worktree file does have the same content, and so drop it. This commit was supported by the NSF-funded DataLad project.
2018-08-22 19:22:52 +00:00
{- Populates a pointer file with the content of a key.
-
- If the file already has some other content, it is not modified.
-
- Returns an InodeCache if it populated the pointer file.
-}
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
populatePointerFile :: Restage -> Key -> RawFilePath -> RawFilePath -> Annex (Maybe InodeCache)
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
populatePointerFile restage k obj f = go =<< liftIO (isPointerFile f)
where
go (Just k') | k == k' = do
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
let f' = fromRawFilePath f
destmode <- liftIO $ catchMaybeIO $ fileMode <$> getFileStatus f'
liftIO $ nukeFile f'
(ic, populated) <- replaceFile f' $ \tmp -> do
use filepath-bytestring for annex object manipulations git-annex find is now RawFilePath end to end, no string conversions. So is git-annex get when it does not need to get anything. So this is a major milestone on optimisation. Benchmarks indicate around 30% speedup in both commands. Probably many other performance improvements. All or nearly all places where a file is statted use RawFilePath now.
2019-12-11 18:12:22 +00:00
let tmp' = toRawFilePath tmp
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
ok <- linkOrCopy k (fromRawFilePath obj) tmp destmode >>= \case
optimisation: avoid stat call This commit was sponsored by Paul Walmsley on Patreon.
2018-09-05 21:26:12 +00:00
Just _ -> thawContent tmp >> return True
use filepath-bytestring for annex object manipulations git-annex find is now RawFilePath end to end, no string conversions. So is git-annex get when it does not need to get anything. So this is a major milestone on optimisation. Benchmarks indicate around 30% speedup in both commands. Probably many other performance improvements. All or nearly all places where a file is statted use RawFilePath now.
2019-12-11 18:12:22 +00:00
Nothing -> liftIO (writePointerFile tmp' k destmode) >> return False
ic <- withTSDelta (liftIO . genInodeCache tmp')
fix race If a pointer file is being populated and something modifies it at the same time, there was a race there the modified file's InodeCache could get added into the keys database. Note that replaceFile normally renames the temp file into place, so the inode cache caculated for the temp file will still be good. If it has to fall back to a copy, the worktree file won't be put in the inode cache. This has the same result as if the worktree file gets touched, and will be handled the same way. Eg, when dropping, isUnmodified will do an expensive comparison and notice that the worktree file does have the same content, and so drop it. This commit was supported by the NSF-funded DataLad project.
2018-08-22 19:22:52 +00:00
return (ic, ok)
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
maybe noop (restagePointerFile restage f) ic
fix race If a pointer file is being populated and something modifies it at the same time, there was a race there the modified file's InodeCache could get added into the keys database. Note that replaceFile normally renames the temp file into place, so the inode cache caculated for the temp file will still be good. If it has to fall back to a copy, the worktree file won't be put in the inode cache. This has the same result as if the worktree file gets touched, and will be handled the same way. Eg, when dropping, isUnmodified will do an expensive comparison and notice that the worktree file does have the same content, and so drop it. This commit was supported by the NSF-funded DataLad project.
2018-08-22 19:22:52 +00:00
if populated
then return ic
else return Nothing
go _ = return Nothing
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
{- Removes the content from a pointer file, replacing it with a pointer.
-
- Does not check if the pointer file is modified. -}
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
depopulatePointerFile :: Key -> RawFilePath -> Annex ()
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
depopulatePointerFile key file = do
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
let file' = fromRawFilePath file
st <- liftIO $ catchMaybeIO $ getFileStatus file'
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
let mode = fmap fileMode st
wip RawFilePath 2x git-annex find speedup Finally builds (oh the agoncy of making it build), but still very unmergable, only Command.Find is included and lots of stuff is badly hacked to make it compile. Benchmarking vs master, this git-annex find is significantly faster! Specifically: num files old new speedup 48500 4.77 3.73 28% 12500 1.36 1.02 66% 20 0.075 0.074 0% (so startup time is unchanged) That's without really finishing the optimization. Things still to do: * Eliminate all the fromRawFilePath, toRawFilePath, encodeBS, decodeBS conversions. * Use versions of IO actions like getFileStatus that take a RawFilePath. * Eliminate some Data.ByteString.Lazy.toStrict, which is a slow copy. * Use ByteString for parsing git config to speed up startup. It's likely several of those will speed up git-annex find further. And other commands will certianly benefit even more.
2019-11-26 19:27:22 +00:00
secureErase file'
liftIO $ nukeFile file'
ic <- replaceFile file' $ \tmp -> do
liftIO $ writePointerFile (toRawFilePath tmp) key mode
preserve unlocked file mtime when dropping When dropping an unlocked file, preserve its mtime, which avoids git status unncessarily running the clean filter on the file. If the index file has close to the same mtime as a work tree file, git will not trust the index to be up-to-date, and re-runs the clean filter unncessarily. Preserving the mtime when depopulating a pointer file avoids git status doing a little (or maybe a lot) of unncessary work. There are other places that the mtime could be preserved, including other places where pointer files are written perhaps, but also populatePointerFile. But, I don't know of cases where those lead to git status doing unncessary work, so I just fixed the one I'm aware of for now.
2019-10-08 18:01:12 +00:00
#if ! defined(mingw32_HOST_OS)
-- Don't advance mtime; this avoids unncessary re-smudging
-- by git in some cases.
liftIO $ maybe noop
(\t -> touch tmp t False)
(fmap modificationTimeHiRes st)
#endif
use filepath-bytestring for annex object manipulations git-annex find is now RawFilePath end to end, no string conversions. So is git-annex get when it does not need to get anything. So this is a major milestone on optimisation. Benchmarks indicate around 30% speedup in both commands. Probably many other performance improvements. All or nearly all places where a file is statted use RawFilePath now.
2019-12-11 18:12:22 +00:00
withTSDelta (liftIO . genInodeCache (toRawFilePath tmp))
split out modules from Annex.Content
2018-08-22 18:41:09 +00:00
maybe noop (restagePointerFile (Restage True) file) ic
Reference in a new issue Copy permalink