improve url download failure display

* When downloading urls fail, explain which urls failed for which
  reasons.
* web: Avoid displaying a warning when downloading one url failed
  but another url later succeeded.

Some other uses of downloadUrl use urls that are effectively internal use,
and should not all be displayed to the user on failure. Eg, Remote.Git
tries different urls where content could be located depending on how the
remote repo is set up. Exposing those urls to the user would lead to wild
goose chases. So had to parameterize it to control whether it displays urls
or not.

A side effect of this change is that when there are some youtube urls
and some regular urls, it will try regular urls first, even if the
youtube urls are listed first. This seems like an improvement if
anything, but in any case there's no defined order of urls that it's
supposed to use.

Sponsored-by: Dartmouth College's Datalad project

Annex/Content.hs

@@ -632,18 +632,20 @@ saveState nocommit = doSideAction $ do
 			Annex.Branch.commit =<< Annex.Branch.commitMessage
 
 {- Downloads content from any of a list of urls, displaying a progress
- - meter. -}
-downloadUrl :: Key -> MeterUpdate -> Maybe IncrementalVerifier -> [Url.URLString] -> FilePath -> Url.UrlOptions -> Annex Bool
-downloadUrl k p iv urls file uo = 
+ - meter.
+ -
+ - Only displays error message if all the urls fail to download.
+ - When listfailedurls is set, lists each url and why it failed.
+ - Otherwise, only displays one error message, from one of the urls
+ - that failed.
+ -}
+downloadUrl :: Bool -> Key -> MeterUpdate -> Maybe IncrementalVerifier -> [Url.URLString] -> FilePath -> Url.UrlOptions -> Annex Bool
+downloadUrl listfailedurls k p iv urls file uo = 
 	-- Poll the file to handle configurations where an external
 	-- download command is used.
-	meteredFile file (Just p) k (go urls Nothing)
+	meteredFile file (Just p) k (go urls [])
   where
-  	-- Display only one error message, if all the urls fail to
-	-- download.
-	go [] (Just err) = warning err >> return False
-	go [] Nothing = return False
-	go (u:us) _ = Url.download' p iv u file uo >>= \case
+	go (u:us) errs = Url.download' p iv u file uo >>= \case
 		Right () -> return True
 		Left err -> do
 			-- If the incremental verifier was fed anything
@@ -655,7 +657,14 @@ downloadUrl k p iv urls file uo =
 					Just n | n > 0 -> unableIncremental iv'
 					_ -> noop
 				Nothing -> noop
-			go us (Just err)
+			go us ((u, err) : errs)
+	go [] [] = return False
+	go [] errs@((_, err):_) = do
+		if listfailedurls
+			then warning $ unlines $ flip map errs $ \(u, err') ->
+				u ++ " " ++ err'
+			else warning err
+		return False
 
 {- Copies a key's content, when present, to a temp file.
  - This is used to speed up some rsyncs. -}