git-annex/doc/bugs/check_for_curl_in_configure.hs.mdwn

[[!meta title="arbitrary/configurable backends"]]

(Retitling as this has drifted..)

---

I thought this might be useful, since curl is being used for the URL backend, it might be worth checking for it's existence.

<pre>
diff --git a/configure.hs b/configure.hs
index 772ba54..1a563e0 100644
--- a/configure.hs
+++ b/configure.hs
@@ -13,6 +13,7 @@ tests = [
        , TestCase "uuid generator" $ selectCmd "uuid" ["uuid", "uuidgen"]
        , TestCase "xargs -0" $ requireCmd "xargs_0" "xargs -0 </dev/null"
        , TestCase "rsync" $ requireCmd "rsync" "rsync --version >/dev/null"
+       , TestCase "curl" $ requireCmd "curl" "curl --version >/dev/null"
        , TestCase "unicode FilePath support" $ unicodeFilePath
        ] ++ shaTestCases [1, 256, 512, 224, 384]
</pre>

> Well, curl is an optional extra, so requireCmd is too strong. Changed
> to testCmd and applied, thank you!
>
> I thought about actually *using* the resulting SysConfig.curl
> to disable the URL backend if False.. but probably it's better
> to just let it fail if curl is not available. Although, if we wanted
> to add a check for wget or something and use it when curl was not
> available, that might be worth doing. --[[Joey]] 

>> I was thinking that is it worth doing a generic "stat", "delete", "get" 
>> and "put" options, I do like the idea of having the possibility of 
>> being about to use completely arbitrary storage systems or arbitrary 
>> transfer systems. If there was the capability of doing so it would be 
>> interesting to see possibilities of using aria2 for using something 
>> like bittorrent as backend, or using something like irods or some 
>> grid storage system as the storage archive. It's just an idea as 
>> I have seen it implemented quite well in irods.

>>> I'm unsure about the idea of having a backend where that is
>>> parameterized. It would mean that one annex's GENERIC-foo key
>>> might be entirely different from another's key with the same backend
>>> and details. And a misconfiguration could get data the wrong
>>> way and get the wrong data, etc.
>>>
>>> I mostly look at the URL backend as an example that can be modified to
>>> make this kind of custom backend. You already probably know enough to
>>> make a TORRENT backend where keys are the urls to torrents to download
>>> with `aria2c --follow-torrent=mem`.
>>>
>>> I am also interested in doing backends that use eg, cloud storage.
>>> A S3 backend that could upload files to S3 in addition to downloading
>>> them, for example, would be handy. --[[Joey]]

>>>> So, rather than use backends to do this, it instead made more sense
>>>> to make them [[special_remotes]]. The URL backend remains a bit
>>>> of a special case, and a bittorrent backend that downloaded a file
>>>> from a bittorrent url would still be a good use of backend, but for
>>>> storing files in external data stores like S3, making it a remote
>>>> makes better sense. I think I can close this bug now, [[done]]
>>>> --[[Joey]] 

also in Backend/URL.hs is it worth making a minor change to the way curl is called (I'm not sure if the following is correct or not)

> It's correct, typewise, but I don't see any real reason to bother
> with the change. But I do appreciate patches, which have been rare
> so far, probaby because of Haskell.. :) --[[Joey]] 

>> heh agreed

<pre>
diff --git a/Backend/URL.hs b/Backend/URL.hs
index 29dc8fe..4afcf86 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -50,10 +50,13 @@ dummyFsck _ _ _ = return True
 dummyOk :: Key -> Annex Bool
 dummyOk _ = return True
 
+curl :: [CommandParam] -> IO Bool
+curl = boolSystem "curl"
+
 downloadUrl :: Key -> FilePath -> Annex Bool
 downloadUrl key file = do
        showNote "downloading"
        showProgress -- make way for curl progress bar
-       liftIO $ boolSystem "curl" [Params "-# -o", File file, File url]
+       liftIO $ curl [Params "-# -o", File file, File url]
        where
                url = join ":" $ drop 1 $ split ":" $ show key 
</pre>