git-annex/doc/todo/optimize_by_converting_String_to_ByteString.mdwn

git-annex uses FilePath (String) extensively. That's a slow data type.
Converting to ByteString, and RawFilePath, should speed it up
significantly, according to [[/profiling]].

I've made a test branch, `bs`, to see what kind of performance improvement
to expect. Most commands don't built yet in that branch, but `git annex
find` does. Speedups range from 28-66%. The files fly by much more
snappily.

As well as adding back all the code that was disabled to get it to build,
the `bs` branch has quite a lot of things still needing work, including:

* Eliminate all the fromRawFilePath, toRawFilePath, encodeBS,
  decodeBS conversions. Or at least most of them. There are likely
  quite a few places where a value is converted back and forth several times.

  As a first step, profile and look for the hot spots. For example, keyFile
  uses fromRawFilePath and that adds around 3% overhead in `git-annex find`. 
  Converting it to a RawFilePath needs a version of `</>` for RawFilePaths.

* System.FilePath is not available for RawFilePath, and many of the
  conversions are to get a FilePath in order to use that library.

  It should be entirely straightforward to make a version of System.FilePath
  that can operate on RawFilePath, except possibly there could be some
  complications due to Windows.

* Use versions of IO actions like getFileStatus that take a RawFilePath,
  avoiding a conversion. Note that these are only available on unix, not
  windows, so a compatability shim will be needed.
  (I can't seem to find any library that provides one.)
todo for bs branch 2019-11-26 20:11:55 +00:00			`git-annex uses FilePath (String) extensively. That's a slow data type.`
			`Converting to ByteString, and RawFilePath, should speed it up`
			`significantly, according to [[/profiling]].`

			I've made a test branch, `bs`, to see what kind of performance improvement
			to expect. Most commands don't built yet in that branch, but `git annex
			find` does. Speedups range from 28-66%. The files fly by much more
			`snappily.`

			`As well as adding back all the code that was disabled to get it to build,`
			the `bs` branch has quite a lot of things still needing work, including:

			`* Eliminate all the fromRawFilePath, toRawFilePath, encodeBS,`
			`decodeBS conversions. Or at least most of them. There are likely`
			`quite a few places where a value is converted back and forth several times.`

update based on profiling While L.toStrict copies, profiling showed it was only around 0.3% of git-annex find runtime. Does not seem worth optimising that, which would probably involve either a major refactoring, or a use of UnsafeInterleaveIO. Also, it seems to me that the latter would need to read chunks, and preappend the leftover part to the next chunk. But a strict ByteString append itself is a copy, so I'm not convinced that would be faster than L.toStrict. 2019-11-27 18:09:11 +00:00			`As a first step, profile and look for the hot spots. For example, keyFile`
			uses fromRawFilePath and that adds around 3% overhead in `git-annex find`.
			Converting it to a RawFilePath needs a version of `</>` for RawFilePaths.
todo for bs branch 2019-11-26 20:11:55 +00:00
			`* System.FilePath is not available for RawFilePath, and many of the`
			`conversions are to get a FilePath in order to use that library.`

			`It should be entirely straightforward to make a version of System.FilePath`
			`that can operate on RawFilePath, except possibly there could be some`
			`complications due to Windows.`

			`* Use versions of IO actions like getFileStatus that take a RawFilePath,`
			`avoiding a conversion. Note that these are only available on unix, not`
			`windows, so a compatability shim will be needed.`
			`(I can't seem to find any library that provides one.)`