ddd46db09a
Started with a problem when running addurl on a really long url, because the whole url is munged into the filename. Ended up doing a fairly extensive review for places where filenames could get too large, although it's hard to say I'm not missed any.. Backend.Url had a 128 character limit, which is fine when the limit is 255, but not if it's a lot shorter on some systems. So check the pathconf() limit. Note that this could result in fromUrl creating different keys for the same url, if run on systems with different limits. I don't see this is likely to cause any problems. That can already happen when using addurl --fast, or if the content of an url changes. Both Command.AddUrl and Backend.Url assumed that urls don't contain a lot of multi-byte unicode, and would fail to truncate an url that did properly. A few places use a filename as the template to make a temp file. While that's nice in that the temp file name can be easily related back to the original filename, it could lead to `git annex add` failing to add a filename that was at or close to the maximum length. Note that in Command.Add.lockdown, the template is still derived from the filename, just with enough space left to turn it into a temp file. This is an important optimisation, because the assistant may lock down a bunch of files all at once, and using the same template for all of them would cause openTempFile to iterate through the same set of names, looking for an unused temp file. I'm not very happy with the relatedTemplate hack, but it avoids that slowdown. Backend.WORM does not limit the filename stored in the key. I have not tried to change that; so git annex add will fail on really long filenames when using the WORM backend. It seems better to preserve the invariant that a WORM key always contains the complete filename, since the filename is the only unique material in the key, other than mtime and size. Since nobody has complained about add failing (I think I saw it once?) on WORM, probably it's ok, or nobody but me uses it. There may be compatability problems if using git annex addurl --fast or the WORM backend on a system with the 255 limit and then trying to use that repo in a system with a smaller limit. I have not tried to deal with those. This commit was sponsored by Alexander Brem. Thanks!
93 lines
3.2 KiB
Haskell
93 lines
3.2 KiB
Haskell
{- GHC File system encoding handling.
|
|
-
|
|
- Copyright 2012-2013 Joey Hess <joey@kitenet.net>
|
|
-
|
|
- Licensed under the GNU GPL version 3 or higher.
|
|
-}
|
|
|
|
module Utility.FileSystemEncoding (
|
|
fileEncoding,
|
|
withFilePath,
|
|
md5FilePath,
|
|
decodeW8,
|
|
encodeW8,
|
|
truncateFilePath,
|
|
) where
|
|
|
|
import qualified GHC.Foreign as GHC
|
|
import qualified GHC.IO.Encoding as Encoding
|
|
import Foreign.C
|
|
import System.IO
|
|
import System.IO.Unsafe
|
|
import qualified Data.Hash.MD5 as MD5
|
|
import Data.Word
|
|
import Data.Bits.Utils
|
|
|
|
{- Sets a Handle to use the filesystem encoding. This causes data
|
|
- written or read from it to be encoded/decoded the same
|
|
- as ghc 7.4 does to filenames etc. This special encoding
|
|
- allows "arbitrary undecodable bytes to be round-tripped through it". -}
|
|
fileEncoding :: Handle -> IO ()
|
|
fileEncoding h = hSetEncoding h =<< Encoding.getFileSystemEncoding
|
|
|
|
{- Marshal a Haskell FilePath into a NUL terminated C string using temporary
|
|
- storage. The FilePath is encoded using the filesystem encoding,
|
|
- reversing the decoding that should have been done when the FilePath
|
|
- was obtained. -}
|
|
withFilePath :: FilePath -> (CString -> IO a) -> IO a
|
|
withFilePath fp f = Encoding.getFileSystemEncoding
|
|
>>= \enc -> GHC.withCString enc fp f
|
|
|
|
{- Encodes a FilePath into a String, applying the filesystem encoding.
|
|
-
|
|
- There are very few things it makes sense to do with such an encoded
|
|
- string. It's not a legal filename; it should not be displayed.
|
|
- So this function is not exported, but instead used by the few functions
|
|
- that can usefully consume it.
|
|
-
|
|
- This use of unsafePerformIO is belived to be safe; GHC's interface
|
|
- only allows doing this conversion with CStrings, and the CString buffer
|
|
- is allocated, used, and deallocated within the call, with no side
|
|
- effects.
|
|
-}
|
|
{-# NOINLINE _encodeFilePath #-}
|
|
_encodeFilePath :: FilePath -> String
|
|
_encodeFilePath fp = unsafePerformIO $ do
|
|
enc <- Encoding.getFileSystemEncoding
|
|
GHC.withCString enc fp $ GHC.peekCString Encoding.char8
|
|
|
|
{- Encodes a FilePath into a Md5.Str, applying the filesystem encoding. -}
|
|
md5FilePath :: FilePath -> MD5.Str
|
|
md5FilePath = MD5.Str . _encodeFilePath
|
|
|
|
{- Converts a [Word8] to a FilePath, encoding using the filesystem encoding.
|
|
-
|
|
- w82c produces a String, which may contain Chars that are invalid
|
|
- unicode. From there, this is really a simple matter of applying the
|
|
- file system encoding, only complicated by GHC's interface to doing so.
|
|
-}
|
|
{-# NOINLINE encodeW8 #-}
|
|
encodeW8 :: [Word8] -> FilePath
|
|
encodeW8 w8 = unsafePerformIO $ do
|
|
enc <- Encoding.getFileSystemEncoding
|
|
GHC.withCString Encoding.char8 (w82s w8) $ GHC.peekCString enc
|
|
|
|
{- Useful when you want the actual number of bytes that will be used to
|
|
- represent the FilePath on disk. -}
|
|
decodeW8 :: FilePath -> [Word8]
|
|
decodeW8 = s2w8 . _encodeFilePath
|
|
|
|
{- Truncates a FilePath to the given number of bytes (or less),
|
|
- as represented on disk.
|
|
-
|
|
- Avoids returning an invalid part of a unicode byte sequence, at the
|
|
- cost of efficiency when running on a large FilePath.
|
|
-}
|
|
truncateFilePath :: Int -> FilePath -> FilePath
|
|
truncateFilePath n = go . reverse
|
|
where
|
|
go f =
|
|
let bytes = decodeW8 f
|
|
in if length bytes <= n
|
|
then reverse f
|
|
else go (drop 1 f)
|