better HTTP connection reuse

Enable HTTP connection reuse across multiple files, when git-annex
uses http-conduit. Before, a new Manager was created each time
Utility.Url used it. Now, a single Manager gets created the first time,
so connections are reused.

Doesn't help when external programs are used for url download,
but does speed up addurl --fast, fsck --from web, etc.

Testing fsck --fast --from web with 3 files, over high-latency
satellite internet, it sped up from 19.37s to 14.96s.

This commit was supported by the NSF-funded DataLad project.
This commit is contained in:
Joey Hess 2018-04-04 15:15:12 -04:00
parent 2ec07bc29f
commit 9b98d3f630
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
12 changed files with 61 additions and 53 deletions

View file

@ -150,7 +150,7 @@ downloadFeed url
| Url.parseURIRelaxed url == Nothing = giveup "invalid feed url"
| otherwise = do
showOutput
Url.withUrlOptions $ \ou ->
Url.withUrlOptions $ \uo ->
liftIO $ withTmpFile "feed" $ \f h -> do
hClose h
ifM (Url.download url f uo)
@ -167,7 +167,8 @@ performDownload opts cache todownload = case location todownload of
then do
urlinfo <- if relaxedOption (downloadOptions opts)
then pure Url.assumeUrlExists
else Url.withUrlOptions (Url.getUrlInfo url)
else Url.withUrlOptions $
liftIO . Url.getUrlInfo url
let dlopts = (downloadOptions opts)
-- force using the filename
-- chosen here