diff --git a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn b/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn new file mode 100644 index 0000000000..184a6ce9eb --- /dev/null +++ b/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn @@ -0,0 +1,49 @@ +### Please describe the problem. + +I have a special directory remote with exporttree=yes (encryption=none) on an USB hard drive. Both `git annex sync --content` and `git annex export` only write around 400 KiB/s. Thus an export of a 9GB DVD iso takes a whole night. + +The drive is not blazing fast, but: + +- `sync; dd if=/dev/zero of=tempfile bs=1M count=10; sync` gives something around 10MB/s (don't recall the exact number) +- rsync (with --progress turned on) copies files with 2.35MB/s + +`mount` for this drive shows: + +> /dev/sdc1 on /media/thk/thk-sg1 type ext4 (rw,nosuid,nodev,relatime,sync,stripe=8191,uhelper=udisks2) + +I tried to mount the drive without sync but failed. Even with the usdisks2 service stopped I could not manually mount the drive without sync (or with async). It always ended up being mounted with sync. + +### What steps will reproduce the problem? + +TODO(thk): try other drive and other laptop once the current transfer finishes... + +### What version of git-annex are you using? On what operating system? + +- git-annex version: 8.20200227-gf56dfe791 +- Debian testing with Kernel 5.2.17 + +### Please provide any additional information below. + +I now learned that there is no Linux kernel primitive to copy a file but that this is actually a high art: + + + +I was surprised to see the implementation of `meteredWrite` in *Utility/Metered.hs*. I hoped that there would be some haskell standard library for efficient file copying? I wonder how rsync implements its progress meter? And whether the progress meter is the reason why rsync had slower write speed than dd. + +Maybe it would make sense to call out to the *cp* command and just issue a *stat()* every few seconds for the progress meter? This is what I do to monitor cp progress manually. + +I have no clue, but maybe these could help for fast file copying in Haskell? + +- +- +- reddit: [What is your take on conduits, pipes, and streams?](https://www.reddit.com/r/haskell/comments/7w79q1/what_is_your_take_on_conduits_pipes_and_streams/) + +### Have you had any luck using git-annex before? + +Well, I'm coming back to git-annex after several years. So far it is better than I remembered: + +- tor support is great and solves the need for a central server +- I hope that the sqlite integration will now make large collections of files managable +- Finally we have exporttree, yeah! + +