From b8f0b7309fbceb04c3d5ffd525274589d016f6a2 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 27 Mar 2015 10:22:32 -0400 Subject: [PATCH] Work around curl bug when asked to download an empty url to a file. In this situation, curl -o exits successfully without creating the output file. There was already a workaround for curl file:/// but I did not realize this also affected regular url downloads. To fix it, pre-create the destination file before starting curl. Since we cannot always know the size of an url before trying to download it, let's always do this. Note that since curl is told -C -, we have to consider if this makes curl try to do a ranged download, which might fail on some servers where a regular download would have succeeded. My testing indicates this isn't a problem; since the file is empty, curl seems to not try to do a ranged download. Original report: https://github.com/datalad/datalad/issues/79 Curl bug report: https://github.com/bagder/curl/issues/183 --- Utility/Url.hs | 11 ++++++----- debian/changelog | 1 + 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Utility/Url.hs b/Utility/Url.hs index ddf5eea408..07be955e25 100644 --- a/Utility/Url.hs +++ b/Utility/Url.hs @@ -209,9 +209,6 @@ download' quiet url file uo = case parseURIRelaxed url of Just u | uriScheme u == "file:" -> do - -- curl does not create destination file - -- for an empty file:// url, so pre-create - writeFile file "" curl | otherwise -> ifM (inPath "wget") (wget , curl) _ -> return False @@ -240,8 +237,12 @@ download' quiet url file uo = - the remainder to download as the whole file, - and not indicating how much percent was - downloaded before the resume. -} - curl = go "curl" $ headerparams ++ quietopt "-s" ++ - [Params "-f -L -C - -# -o"] + curl = do + -- curl does not create destination file + -- if the url happens to be empty, so pre-create. + writeFile file "" + go "curl" $ headerparams ++ quietopt "-s" ++ + [Params "-f -L -C - -# -o"] go cmd opts = boolSystem cmd $ addUserAgent uo $ reqParams uo++opts++[File file, File url] quietopt s diff --git a/debian/changelog b/debian/changelog index c41a3c0b51..bb487279b9 100644 --- a/debian/changelog +++ b/debian/changelog @@ -17,6 +17,7 @@ git-annex (5.20150318) UNRELEASED; urgency=medium * assistant: Committing a whole lot of files at once could overflow command-line length limits and cause the commit to fail. This only happened when using the assistant in an indirect mode repository. + * Work around curl bug when asked to download an empty url to a file. -- Joey Hess Thu, 19 Mar 2015 17:05:32 -0400