2013-05-12 23:19:28 +00:00
|
|
|
{- Temporary files and directories.
|
2011-10-16 04:31:25 +00:00
|
|
|
-
|
2015-01-21 16:50:09 +00:00
|
|
|
- Copyright 2010-2013 Joey Hess <id@joeyh.name>
|
2011-10-16 04:31:25 +00:00
|
|
|
-
|
2014-05-10 14:01:27 +00:00
|
|
|
- License: BSD-2-clause
|
2011-10-16 04:31:25 +00:00
|
|
|
-}
|
|
|
|
|
2013-12-29 20:25:12 +00:00
|
|
|
{-# LANGUAGE CPP #-}
|
2015-05-10 20:31:50 +00:00
|
|
|
{-# OPTIONS_GHC -fno-warn-tabs #-}
|
2013-12-29 20:25:12 +00:00
|
|
|
|
2013-05-12 23:19:28 +00:00
|
|
|
module Utility.Tmp where
|
2011-10-16 04:31:25 +00:00
|
|
|
|
|
|
|
import System.IO
|
2013-05-12 20:38:00 +00:00
|
|
|
import Control.Monad.IfElse
|
2014-01-29 19:21:02 +00:00
|
|
|
import System.FilePath
|
2016-04-28 19:18:11 +00:00
|
|
|
import System.Directory
|
2014-07-29 20:22:19 +00:00
|
|
|
import Control.Monad.IO.Class
|
2015-12-16 00:20:37 +00:00
|
|
|
#ifndef mingw32_HOST_OS
|
|
|
|
import System.Posix.Temp (mkdtemp)
|
|
|
|
#endif
|
2011-10-16 04:31:25 +00:00
|
|
|
|
2012-02-03 20:47:24 +00:00
|
|
|
import Utility.Exception
|
Fix a few bugs involving filenames that are at or near the filesystem's maximum filename length limit.
Started with a problem when running addurl on a really long url,
because the whole url is munged into the filename. Ended up doing
a fairly extensive review for places where filenames could get too large,
although it's hard to say I'm not missed any..
Backend.Url had a 128 character limit, which is fine when the limit is 255,
but not if it's a lot shorter on some systems. So check the pathconf()
limit. Note that this could result in fromUrl creating different keys
for the same url, if run on systems with different limits. I don't see
this is likely to cause any problems. That can already happen when using
addurl --fast, or if the content of an url changes.
Both Command.AddUrl and Backend.Url assumed that urls don't contain a
lot of multi-byte unicode, and would fail to truncate an url that did
properly.
A few places use a filename as the template to make a temp file.
While that's nice in that the temp file name can be easily related back to
the original filename, it could lead to `git annex add` failing to add a
filename that was at or close to the maximum length.
Note that in Command.Add.lockdown, the template is still derived from the
filename, just with enough space left to turn it into a temp file.
This is an important optimisation, because the assistant may lock down
a bunch of files all at once, and using the same template for all of them
would cause openTempFile to iterate through the same set of names,
looking for an unused temp file. I'm not very happy with the relatedTemplate
hack, but it avoids that slowdown.
Backend.WORM does not limit the filename stored in the key.
I have not tried to change that; so git annex add will fail on really long
filenames when using the WORM backend. It seems better to preserve the
invariant that a WORM key always contains the complete filename, since
the filename is the only unique material in the key, other than mtime and
size. Since nobody has complained about add failing (I think I saw it
once?) on WORM, probably it's ok, or nobody but me uses it.
There may be compatability problems if using git annex addurl --fast
or the WORM backend on a system with the 255 limit and then trying to use
that repo in a system with a smaller limit. I have not tried to deal with
those.
This commit was sponsored by Alexander Brem. Thanks!
2013-07-30 21:49:11 +00:00
|
|
|
import Utility.FileSystemEncoding
|
2014-01-29 19:21:02 +00:00
|
|
|
import Utility.PosixFiles
|
2011-10-16 04:31:25 +00:00
|
|
|
|
2013-05-12 20:38:00 +00:00
|
|
|
type Template = String
|
|
|
|
|
2011-10-16 04:31:25 +00:00
|
|
|
{- Runs an action like writeFile, writing to a temp file first and
|
|
|
|
- then moving it into place. The temp file is stored in the same
|
|
|
|
- directory as the final file to avoid cross-device renames. -}
|
2017-08-31 18:24:32 +00:00
|
|
|
viaTmp :: (MonadMask m, MonadIO m) => (FilePath -> v -> m ()) -> FilePath -> v -> m ()
|
2014-11-12 18:59:24 +00:00
|
|
|
viaTmp a file content = bracketIO setup cleanup use
|
2014-06-09 19:24:05 +00:00
|
|
|
where
|
|
|
|
(dir, base) = splitFileName file
|
|
|
|
template = base ++ ".tmp"
|
|
|
|
setup = do
|
|
|
|
createDirectoryIfMissing True dir
|
|
|
|
openTempFile dir template
|
unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.
Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.
Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.
However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.
Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-08 01:55:44 +00:00
|
|
|
cleanup (tmpfile, h) = do
|
|
|
|
_ <- tryIO $ hClose h
|
2014-06-09 19:24:05 +00:00
|
|
|
tryIO $ removeFile tmpfile
|
unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.
Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.
Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.
However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.
Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-08 01:55:44 +00:00
|
|
|
use (tmpfile, h) = do
|
2014-11-12 18:59:24 +00:00
|
|
|
liftIO $ hClose h
|
2014-06-09 19:24:05 +00:00
|
|
|
a tmpfile content
|
2014-11-12 18:59:24 +00:00
|
|
|
liftIO $ rename tmpfile file
|
2012-01-21 06:24:12 +00:00
|
|
|
|
2013-05-12 20:38:00 +00:00
|
|
|
{- Runs an action with a tmp file located in the system's tmp directory
|
|
|
|
- (or in "." if there is none) then removes the file. -}
|
2014-07-29 20:22:19 +00:00
|
|
|
withTmpFile :: (MonadIO m, MonadMask m) => Template -> (FilePath -> Handle -> m a) -> m a
|
2013-05-12 23:19:28 +00:00
|
|
|
withTmpFile template a = do
|
2014-07-29 20:22:19 +00:00
|
|
|
tmpdir <- liftIO $ catchDefaultIO "." getTemporaryDirectory
|
2013-05-12 23:19:28 +00:00
|
|
|
withTmpFileIn tmpdir template a
|
2013-05-12 20:38:00 +00:00
|
|
|
|
|
|
|
{- Runs an action with a tmp file located in the specified directory,
|
|
|
|
- then removes the file. -}
|
2014-07-29 20:22:19 +00:00
|
|
|
withTmpFileIn :: (MonadIO m, MonadMask m) => FilePath -> Template -> (FilePath -> Handle -> m a) -> m a
|
2013-05-12 23:19:28 +00:00
|
|
|
withTmpFileIn tmpdir template a = bracket create remove use
|
2012-11-12 21:43:10 +00:00
|
|
|
where
|
2014-07-29 20:22:19 +00:00
|
|
|
create = liftIO $ openTempFile tmpdir template
|
unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.
Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.
Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.
However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.
Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-08 01:55:44 +00:00
|
|
|
remove (name, h) = liftIO $ do
|
|
|
|
hClose h
|
2012-11-12 21:43:10 +00:00
|
|
|
catchBoolIO (removeFile name >> return True)
|
unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.
Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.
Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.
However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.
Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-08 01:55:44 +00:00
|
|
|
use (name, h) = a name h
|
2012-09-10 18:09:13 +00:00
|
|
|
|
2013-05-12 20:38:00 +00:00
|
|
|
{- Runs an action with a tmp directory located within the system's tmp
|
|
|
|
- directory (or within "." if there is none), then removes the tmp
|
|
|
|
- directory and all its contents. -}
|
2014-11-12 18:59:24 +00:00
|
|
|
withTmpDir :: (MonadMask m, MonadIO m) => Template -> (FilePath -> m a) -> m a
|
2013-05-12 23:19:28 +00:00
|
|
|
withTmpDir template a = do
|
2015-12-16 00:20:37 +00:00
|
|
|
topleveltmpdir <- liftIO $ catchDefaultIO "." getTemporaryDirectory
|
|
|
|
#ifndef mingw32_HOST_OS
|
|
|
|
-- Use mkdtemp to create a temp directory securely in /tmp.
|
|
|
|
bracket
|
|
|
|
(liftIO $ mkdtemp $ topleveltmpdir </> template)
|
|
|
|
removeTmpDir
|
|
|
|
a
|
|
|
|
#else
|
|
|
|
withTmpDirIn topleveltmpdir template a
|
|
|
|
#endif
|
2013-05-12 20:38:00 +00:00
|
|
|
|
|
|
|
{- Runs an action with a tmp directory located within a specified directory,
|
|
|
|
- then removes the tmp directory and all its contents. -}
|
2014-11-12 18:59:24 +00:00
|
|
|
withTmpDirIn :: (MonadMask m, MonadIO m) => FilePath -> Template -> (FilePath -> m a) -> m a
|
2015-12-16 00:20:37 +00:00
|
|
|
withTmpDirIn tmpdir template = bracketIO create removeTmpDir
|
2012-11-12 21:43:10 +00:00
|
|
|
where
|
|
|
|
create = do
|
|
|
|
createDirectoryIfMissing True tmpdir
|
2013-05-12 20:38:00 +00:00
|
|
|
makenewdir (tmpdir </> template) (0 :: Int)
|
|
|
|
makenewdir t n = do
|
|
|
|
let dir = t ++ "." ++ show n
|
2015-12-06 20:29:36 +00:00
|
|
|
catchIOErrorType AlreadyExists (const $ makenewdir t $ n + 1) $ do
|
|
|
|
createDirectory dir
|
|
|
|
return dir
|
Fix a few bugs involving filenames that are at or near the filesystem's maximum filename length limit.
Started with a problem when running addurl on a really long url,
because the whole url is munged into the filename. Ended up doing
a fairly extensive review for places where filenames could get too large,
although it's hard to say I'm not missed any..
Backend.Url had a 128 character limit, which is fine when the limit is 255,
but not if it's a lot shorter on some systems. So check the pathconf()
limit. Note that this could result in fromUrl creating different keys
for the same url, if run on systems with different limits. I don't see
this is likely to cause any problems. That can already happen when using
addurl --fast, or if the content of an url changes.
Both Command.AddUrl and Backend.Url assumed that urls don't contain a
lot of multi-byte unicode, and would fail to truncate an url that did
properly.
A few places use a filename as the template to make a temp file.
While that's nice in that the temp file name can be easily related back to
the original filename, it could lead to `git annex add` failing to add a
filename that was at or close to the maximum length.
Note that in Command.Add.lockdown, the template is still derived from the
filename, just with enough space left to turn it into a temp file.
This is an important optimisation, because the assistant may lock down
a bunch of files all at once, and using the same template for all of them
would cause openTempFile to iterate through the same set of names,
looking for an unused temp file. I'm not very happy with the relatedTemplate
hack, but it avoids that slowdown.
Backend.WORM does not limit the filename stored in the key.
I have not tried to change that; so git annex add will fail on really long
filenames when using the WORM backend. It seems better to preserve the
invariant that a WORM key always contains the complete filename, since
the filename is the only unique material in the key, other than mtime and
size. Since nobody has complained about add failing (I think I saw it
once?) on WORM, probably it's ok, or nobody but me uses it.
There may be compatability problems if using git annex addurl --fast
or the WORM backend on a system with the 255 limit and then trying to use
that repo in a system with a smaller limit. I have not tried to deal with
those.
This commit was sponsored by Alexander Brem. Thanks!
2013-07-30 21:49:11 +00:00
|
|
|
|
2015-12-16 00:20:37 +00:00
|
|
|
{- Deletes the entire contents of the the temporary directory, if it
|
|
|
|
- exists. -}
|
|
|
|
removeTmpDir :: MonadIO m => FilePath -> m ()
|
|
|
|
removeTmpDir tmpdir = liftIO $ whenM (doesDirectoryExist tmpdir) $ do
|
|
|
|
#if mingw32_HOST_OS
|
|
|
|
-- Windows will often refuse to delete a file
|
|
|
|
-- after a process has just written to it and exited.
|
|
|
|
-- Because it's crap, presumably. So, ignore failure
|
|
|
|
-- to delete the temp directory.
|
|
|
|
_ <- tryIO $ removeDirectoryRecursive tmpdir
|
|
|
|
return ()
|
|
|
|
#else
|
|
|
|
removeDirectoryRecursive tmpdir
|
|
|
|
#endif
|
|
|
|
|
Fix a few bugs involving filenames that are at or near the filesystem's maximum filename length limit.
Started with a problem when running addurl on a really long url,
because the whole url is munged into the filename. Ended up doing
a fairly extensive review for places where filenames could get too large,
although it's hard to say I'm not missed any..
Backend.Url had a 128 character limit, which is fine when the limit is 255,
but not if it's a lot shorter on some systems. So check the pathconf()
limit. Note that this could result in fromUrl creating different keys
for the same url, if run on systems with different limits. I don't see
this is likely to cause any problems. That can already happen when using
addurl --fast, or if the content of an url changes.
Both Command.AddUrl and Backend.Url assumed that urls don't contain a
lot of multi-byte unicode, and would fail to truncate an url that did
properly.
A few places use a filename as the template to make a temp file.
While that's nice in that the temp file name can be easily related back to
the original filename, it could lead to `git annex add` failing to add a
filename that was at or close to the maximum length.
Note that in Command.Add.lockdown, the template is still derived from the
filename, just with enough space left to turn it into a temp file.
This is an important optimisation, because the assistant may lock down
a bunch of files all at once, and using the same template for all of them
would cause openTempFile to iterate through the same set of names,
looking for an unused temp file. I'm not very happy with the relatedTemplate
hack, but it avoids that slowdown.
Backend.WORM does not limit the filename stored in the key.
I have not tried to change that; so git annex add will fail on really long
filenames when using the WORM backend. It seems better to preserve the
invariant that a WORM key always contains the complete filename, since
the filename is the only unique material in the key, other than mtime and
size. Since nobody has complained about add failing (I think I saw it
once?) on WORM, probably it's ok, or nobody but me uses it.
There may be compatability problems if using git annex addurl --fast
or the WORM backend on a system with the 255 limit and then trying to use
that repo in a system with a smaller limit. I have not tried to deal with
those.
This commit was sponsored by Alexander Brem. Thanks!
2013-07-30 21:49:11 +00:00
|
|
|
{- It's not safe to use a FilePath of an existing file as the template
|
|
|
|
- for openTempFile, because if the FilePath is really long, the tmpfile
|
|
|
|
- will be longer, and may exceed the maximum filename length.
|
|
|
|
-
|
|
|
|
- This generates a template that is never too long.
|
|
|
|
- (Well, it allocates 20 characters for use in making a unique temp file,
|
|
|
|
- anyway, which is enough for the current implementation and any
|
|
|
|
- likely implementation.)
|
|
|
|
-}
|
|
|
|
relatedTemplate :: FilePath -> FilePath
|
|
|
|
relatedTemplate f
|
|
|
|
| len > 20 = truncateFilePath (len - 20) f
|
|
|
|
| otherwise = f
|
|
|
|
where
|
|
|
|
len = length f
|