improve special remotes list
In several cases, there's an old, maybe unmaintained special remote, and rclone also supports the service. I don't know which is better, so I added tips pages for these, listing both possibilies, so users can fill in the details. Added some things listed in special_remotes/rclone that were not in the cloud provider list, so users will see it supports those. But, omitted S3 and webdav, since it's probably better for users to use what's built into git-annex than going via rclone. The rclone list is out of date apparently, as rclone.org has more, so I added a note at the bottom for users to check. This commit was sponsored by Ethan Aubin.
This commit is contained in:
Joey Hess
parent
97a8ce0191
commit
5995eba531
20 changed files with 82 additions and 290 deletions
|
|
@ -17,7 +17,6 @@ the git history is not stored in them.
|
|||
* [[gcrypt]] (encrypted git repositories!)
|
||||
* [[git-lfs]]
|
||||
* [[hook]]
|
||||
* [[rclone]]
|
||||
* [[rsync]]
|
||||
* [[S3]] (Amazon S3, and other compatible services)
|
||||
* [[tahoe]]
|
||||
|
|
@ -30,34 +29,43 @@ the git history is not stored in them.
|
|||
The above special remotes are built into git-annex, and can be used
|
||||
to tie git-annex into many cloud services.
|
||||
|
||||
Here are specific instructions
|
||||
for using git-annex with various services:
|
||||
Here are specific instructions for using git-annex with various services:
|
||||
|
||||
* [[Amazon_Glacier|tips/using_Amazon_Glacier]]
|
||||
* [Amazon Cloud drive](https://github.com/DanielDent/git-annex-remote-rclone)
|
||||
* [[Amazon Cloud drive|rclone]]
|
||||
* [[Amazon_S3|tips/using_Amazon_S3]]
|
||||
* [Backblaze's B2](https://github.com/encryptio/git-annex-remote-b2)
|
||||
* [[Backblaze B2|tips/using_Backblaze_B2]]
|
||||
* [[Box.com|tips/using_box.com_as_a_special_remote]]
|
||||
* [Ceph](https://github.com/mhameed/git-annex-remote-ceph)
|
||||
* [chef-vault](https://github.com/3ofcoins/knife-annex/)
|
||||
* [Dropbox](https://github.com/DanielDent/git-annex-remote-rclone)
|
||||
* [[Dropbox|rclone]]
|
||||
* [[FTP|rclone]]
|
||||
* [[Flickr|tips/flickrannex]]
|
||||
* [Freenet](https://github.com/xloem/gitlakepy)
|
||||
* [[Google Cloud Storage|tips/using_Google_Cloud_Storage]]
|
||||
* [Google Drive](https://github.com/Lykos153/git-annex-remote-googledrive)
|
||||
* [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
|
||||
* [[Google Drive|tips/using_Google_Drive]]
|
||||
* [[hubiC|tips/using_Hubic]]
|
||||
* [[IMAP|forum/special_remote_for_IMAP]]
|
||||
* [[tips/Internet_Archive_via_S3]]
|
||||
* [[ipfs]]
|
||||
* [[Mega.co.nz|tips/megaannex]]
|
||||
* [Microsoft One Drive](https://github.com/DanielDent/git-annex-remote-rclone)
|
||||
* [Openstack Swift / Rackspace cloud files / Memset Memstore](https://github.com/DanielDent/git-annex-remote-rclone)
|
||||
* [[Jottacloud|rclone]]
|
||||
* [[Mega|tips/megaannex]]
|
||||
* [[Microsoft Azure Blob Storage|rclone]]
|
||||
* [[Microsoft OneDrive|rclone]]
|
||||
* [[OpenDrive|rclone]]
|
||||
* [[Openstack Swift / Rackspace cloud files / Memset Memstore|rclone]]
|
||||
* [[OwnCloud|tips/owncloudannex]]
|
||||
* [pCloud](https://github.com/tochev/git-annex-remote-pcloud)
|
||||
* [[pCloud|rclone]]
|
||||
* [[QingStor|rclone]]
|
||||
* [[SFTP|rclone]]
|
||||
* [[SkyDrive|tips/skydriveannex]]
|
||||
* [smb / sftp](https://github.com/grawity/code/blob/master/net/git-annex-remote-gvfs)
|
||||
* [[Usenet|forum/nntp__47__usenet special remote]]
|
||||
* [Yandex Disk](https://github.com/DanielDent/git-annex-remote-rclone)
|
||||
* [[Yandex Disk|rclone]]
|
||||
|
||||
If a service is not mentioned above, it's worth checking if
|
||||
[rclone](https://rclone.org/) supports it, then you can use the
|
||||
[[rclone_special_remote|rclone]].
|
||||
|
||||
Want to add support for something else? [[Write your own!|external]]
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
[rclone](https://rclone.org/) is a command line program to sync files and directories to and from a vast variety of cloud provider and protocols. At the time of writing, this includes the following services:
|
||||
|
||||
[rclone](https://rclone.org/) is a command line program to sync files and
|
||||
directories to and from a vast variety of cloud provider and protocols. At
|
||||
the time of writing, this includes the following services:
|
||||
|
||||
* Amazon Drive ([See note](https://rclone.org/amazonclouddrive/#status))
|
||||
* Amazon S3 / Dreamhost / Ceph / Minio / Wasabi
|
||||
|
|
@ -24,6 +25,7 @@
|
|||
* Yandex Disk
|
||||
* The local filesystem
|
||||
|
||||
That list is regularly expanding. git-annex supports all of those through the use of the [rclone special remote](https://github.com/DanielDent/git-annex-remote-rclone).
|
||||
That list is regularly expanding. git-annex supports all of those through
|
||||
the use of the [rclone special remote](https://github.com/DanielDent/git-annex-remote-rclone).
|
||||
|
||||
See their documentation for more concrete examples.
|
||||
|
|
|
|||
|
|
@ -1,35 +0,0 @@
|
|||
** The software documented below is not being maintained. **
|
||||
Instead, use <https://github.com/Lykos153/git-annex-remote-googledrive>
|
||||
|
||||
googledriveannex 0.2.0
|
||||
=========
|
||||
|
||||
Hook program for gitannex to use Google Drive as backend
|
||||
|
||||
# Requirements:
|
||||
|
||||
python2
|
||||
python-httplib2
|
||||
|
||||
Credit for the googledrive api interface goes to google
|
||||
|
||||
# Install
|
||||
Clone the git repository in your home folder.
|
||||
|
||||
git clone git://github.com/TobiasTheViking/googledriveannex.git
|
||||
|
||||
This should make a ~/googledriveannex folder
|
||||
|
||||
# Setup
|
||||
Make the file executable, and link it into PATH
|
||||
|
||||
cd ~/googledriveannex; chmod +x git-annex-remote-googledrive; sudo ln -sf `pwd`/git-annex-remote-googledrive /usr/local/bin/git-annex-remote-googledrive
|
||||
|
||||
# Commands for gitannex:
|
||||
|
||||
git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
|
||||
|
||||
An oauth authentication link should now be launched in the default browser. Authenticate and you will be proved with a code.
|
||||
|
||||
OAUTH='authentication code' git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
|
||||
git annex describe googledrive "the googledrive library"
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 10"""
|
||||
date="2015-08-11T17:22:48Z"
|
||||
content="""
|
||||
I have not heard from TobiasTheViking in a while, unfortunately.
|
||||
|
||||
Whether this could be supported in git-anenx depends on whether a haskell
|
||||
API comes available.
|
||||
|
||||
Google Cloud Storage might be a better option; it is supported in git-annex
|
||||
via the S3 remote, as well as having an actively maintained external
|
||||
special remote.
|
||||
"""]]
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="lykos@d125a37d89b1cfac20829f12911656c40cb70018"
|
||||
nickname="lykos"
|
||||
avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2"
|
||||
subject="comment 12"
|
||||
date="2017-07-21T23:56:12Z"
|
||||
content="""
|
||||
I couldn't get this to run and had a lot of performance issues with rclone on Google Drive, so I adapted the rclone wrapper to [gdrive](https://github.com/prasmussen/gdrive). It's running fine so far, so I thought I share it:
|
||||
|
||||
[[https://github.com/Lykos153/git-annex-remote-gdrive]]
|
||||
"""]]
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="emanuele.olivetti@47d88ed185b03191e25329caa6fabc2efb3118b2"
|
||||
nickname="emanuele.olivetti"
|
||||
subject="googledriveannex fails: No handlers could be found for logger "oauth2client.util""
|
||||
date="2016-07-14T14:58:29Z"
|
||||
content="""
|
||||
Following the instructions above, when issuing:
|
||||
|
||||
git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
|
||||
|
||||
googleannexdrive fails and I get:
|
||||
|
||||
initremote googledrive (encryption setup) (encryption key stored in git repository) No handlers could be found for logger \"oauth2client.util\"
|
||||
git-annex: You need to set OAUTH environment variables and folder and encryption parameters when running initremote.
|
||||
|
||||
Am I making some obvious mistake?
|
||||
|
||||
If not, I'd like to share my positive experience with git-annex-remote-rclone (<https://github.com/DanielDent/git-annex-remote-rclone>), which wraps rclone (<https://rclone.org>) that supports Google Drive: it worked for me.
|
||||
"""]]
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawmlnEK_po7A0xMC9Xdp0FdcBgYllsCORXM"
|
||||
nickname="Johnny"
|
||||
subject="Shared google drive on multiple hosts"
|
||||
date="2014-03-14T16:40:11Z"
|
||||
content="""
|
||||
I have now configured my Googledrive on Laptop1 according to the guide above, how do I add the same remote on Laptop2? It would be nice if this was described in the guide, include where the encryption keys are stored.
|
||||
|
||||
"""]]
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawl64jV2rE8GMogJ6XuqESSkz78RVBgVdGw"
|
||||
nickname="Mesut"
|
||||
subject="I can't add google drive as remote"
|
||||
date="2014-04-10T07:55:56Z"
|
||||
content="""
|
||||
Hi,
|
||||
|
||||
I am new to git-annex and I want to use google drive as remote but I can't.
|
||||
|
||||
I create syslink to `googledriveannex` in `/usr/local/bin`.
|
||||
|
||||
When I execute below command, command waiting but not make anything:
|
||||
|
||||
`$ git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex`
|
||||
|
||||
`initremote googledrive (encryption setup)` # Waiting but does not do anything.
|
||||
|
||||
What I am doing wrong?
|
||||
|
||||
Thanks for helps
|
||||
|
||||
"""]]
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="http://joeyh.name/"
|
||||
ip="209.250.56.191"
|
||||
subject="comment 3"
|
||||
date="2014-04-17T20:58:41Z"
|
||||
content="""
|
||||
@Mesut, I think you're doing everything right. It can take a long time for the highly secure gpg key to be generated. Sit tight and let it finish, or you can pass --fast to generate a key that is a tiny bit less secure.
|
||||
"""]]
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo"
|
||||
nickname="donkeyicydragon"
|
||||
subject="Googledrive annex on second repository"
|
||||
date="2014-04-19T20:45:23Z"
|
||||
content="""
|
||||
Hi Johnny,
|
||||
|
||||
I wrote a patch for googledriveannex that fixed this problem for me. First you add the google drive special remote in repo1 then you clone repo1 into repo2. In repo2 you do \"git annex enableremote googldrivespecialremotename\" and it should work.
|
||||
The problem was that the init method, that is called by git annex when a special remote is first created but also when it is enabled somewhere else, did not factor in the possibility that it had already been created.
|
||||
I will simultaneously submit the patch to the author of the special remote plugin but here it is for you to quickly get going:
|
||||
|
||||
diff --git a/git-annex-remote-googledrive b/git-annex-remote-googledrive
|
||||
index 49cd917..c8e70f3 100755
|
||||
--- a/git-annex-remote-googledrive
|
||||
+++ b/git-annex-remote-googledrive
|
||||
@@ -330,13 +330,16 @@ def initremote(line):
|
||||
oauth = os.getenv(\"OAUTH\") or \"\"
|
||||
encryption = common.getConfig(\"encryption\")
|
||||
myfolder = common.getConfig(\"folder\")
|
||||
- stored_creds = sys.modules[\"__main__\"].login({\"oauth\": oauth})
|
||||
- if len(myfolder) and stored_creds:
|
||||
- common.sprint('SETCONFIG myfolder ' + myfolder + '')
|
||||
- common.sprint('SETCONFIG stored_creds ' + json.dumps(stored_creds) + '')
|
||||
- common.sprint('INITREMOTE-SUCCESS')
|
||||
+ if not common.getConfig(\"stored_creds\"):
|
||||
+ stored_creds = sys.modules[\"__main__\"].login({\"oauth\": oauth})
|
||||
+ if len(myfolder) and stored_creds:
|
||||
+ common.sprint('SETCONFIG myfolder ' + myfolder + '')
|
||||
+ common.sprint('SETCONFIG stored_creds ' + json.dumps(stored_creds) + '')
|
||||
+ common.sprint('INITREMOTE-SUCCESS')
|
||||
+ else:
|
||||
+ common.sprint('INITREMOTE-FAILURE You need to set OAUTH environment variables and folder and encryption parameters when running initremote.')
|
||||
else:
|
||||
- common.sprint('INITREMOTE-FAILURE You need to set OAUTH environment variables and folder and encryption parameters when running initremote.')
|
||||
+ common.sprint('INITREMOTE-SUCCESS')
|
||||
common.log(\"Done\")
|
||||
|
||||
def prepare(line):
|
||||
--
|
||||
"""]]
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="hugo"
|
||||
ip="37.160.8.232"
|
||||
subject="'content-length' error"
|
||||
date="2014-10-05T19:40:23Z"
|
||||
content="""
|
||||
I moved a big PDF to Google Drive (with shared encryption).
|
||||
|
||||
Now, when I try to get it again:
|
||||
|
||||
get Documents/Guyau - The Non-Religion of the Future, nonreligionoffut00guyarich.pdf (from googledrive...) (gpg)
|
||||
Traceback (most recent call last):
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 411, in <module>
|
||||
common.startRemote()
|
||||
File \"/usr/share/googledriveannex-git/lib/CommonFunctions.py\", line 555, in startRemote
|
||||
sys.modules[\"__main__\"].transfer(line)
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 372, in transfer
|
||||
if getFile(line[2], \" \".join(line[3:]), folder):
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 257, in getFile
|
||||
ret = common.fetchPage({\"link\": download_url, \"headers\": [(\"Authorization\", \"Bearer \" + credentials.access_token)], \"progress\": \"true\"})
|
||||
File \"/usr/share/googledriveannex-git/lib/CommonFunctions.py\", line 207, in fetchPage
|
||||
totalsize = int(con.headers['content-length'])
|
||||
File \"/usr/lib/python2.7/rfc822.py\", line 388, in __getitem__
|
||||
return self.dict[name.lower()]
|
||||
KeyError: 'content-length'
|
||||
|
||||
It works for smaller files. Is there a limit on the file size?
|
||||
|
||||
"""]]
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="hugo"
|
||||
ip="37.160.8.232"
|
||||
subject="Re: 'content-length' error "
|
||||
date="2014-10-05T19:43:04Z"
|
||||
content="""
|
||||
The Google Drive interface tells me:
|
||||
|
||||
Size 29,776,826 bytes
|
||||
"""]]
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="http://joeyh.name/"
|
||||
ip="209.250.56.54"
|
||||
subject="comment 7"
|
||||
date="2014-10-06T15:23:59Z"
|
||||
content="""
|
||||
Hugo, I didn't write this code, but it looks to me like you could work around the problem by changing line 207 of lib/CommonFunctions.py:
|
||||
|
||||
<pre>
|
||||
diff --git a/lib/CommonFunctions.py b/lib/CommonFunctions.py
|
||||
index 050b93e..083f5d6 100644
|
||||
--- a/lib/CommonFunctions.py
|
||||
+++ b/lib/CommonFunctions.py
|
||||
@@ -204,7 +204,7 @@ def fetchPage(params={}):
|
||||
if get(\"progress\"):
|
||||
data = False
|
||||
tdata = \"\"
|
||||
- totalsize = int(con.headers['content-length'])
|
||||
+ totalsize = 0
|
||||
chunksize = totalsize / 100
|
||||
if chunksize < 4096:
|
||||
chunksize = 4096
|
||||
</pre>
|
||||
|
||||
Probably the API used to return a content-length header, and no longer does, or doesn't do so reliably. It does not seem to be used for anything too important -- this change will break git-annex's progress display a little bit, perhaps.
|
||||
"""]]
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="hugo"
|
||||
ip="79.212.203.201"
|
||||
subject="Re: comment 7 Re: 'content-length' error "
|
||||
date="2014-10-06T16:45:03Z"
|
||||
content="""
|
||||
Thanks Joey, I was able to get the file after this modification. I’ll make a pull request on Github.
|
||||
"""]]
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4"
|
||||
subject="any chance to get a native google drive annex support?"
|
||||
date="2015-08-11T04:12:10Z"
|
||||
content="""
|
||||
the referenced custom remote project is pretty much dead, PRs are hanging without merge, issues lists growing (https://github.com/TobiasTheViking/googledriveannex/issues) with e.g. grave ones like dropping content removed on the drive as well...
|
||||
Just wondered ;-)
|
||||
"""]]
|
||||
|
|
@ -1,29 +1,14 @@
|
|||
megaannex 0.2.0
|
||||
=========
|
||||
For using [Mega](https://mega.nz)
|
||||
as a special remote, there are currently three choices:
|
||||
|
||||
Hook program for git-annex to use mega.co.nz as backend
|
||||
* There is a dedicated special remote,
|
||||
<https://github.com/dxtr/megaannex-go>
|
||||
Last updated 2016
|
||||
* Alternatively, rclone supports Mega,
|
||||
so the [[rclone_special_remote|special_remotes/rclone]] can be used.
|
||||
* Alternatively, there's an older dedicated special remote,
|
||||
<https://github.com/TobiasTheViking/megaannex>
|
||||
Reported to be no longer working
|
||||
|
||||
# Requirements:
|
||||
|
||||
python2
|
||||
requests>=0.10
|
||||
pycrypto
|
||||
|
||||
Credit for the mega api interface goes to: https://github.com/richardasaurus/mega.py
|
||||
|
||||
# Install
|
||||
Clone the git repository in your home folder.
|
||||
|
||||
git clone git://github.com/TobiasTheViking/megaannex.git
|
||||
|
||||
This should make a ~/megaannex folder
|
||||
|
||||
# Setup
|
||||
Make the file executable, and link it into PATH
|
||||
|
||||
cd ~/megaannex; chmod +x git-annex-remote-mega; sudo ln -sf `pwd`/git-annex-remote-mega /usr/local/bin/git-annex-remote-mega
|
||||
|
||||
# Commands for gitannex:
|
||||
|
||||
USERNAME="user" PASSWORD="pword" git annex initremote mega type=external externaltype=mega encryption=shared folder=gitannex
|
||||
git annex describe mega "the mega.co.nz library"
|
||||
At this time it's not clear which is better, so if you find one works
|
||||
best, please comment below.
|
||||
|
|
|
|||
10
doc/tips/using_Backblaze_B2.mdwn
Normal file
10
doc/tips/using_Backblaze_B2.mdwn
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
For using Backblaze B2 as a special remote, there are currently two
|
||||
choices:
|
||||
|
||||
* Using [[special_remotes/rclone]]
|
||||
(Actively maintained)
|
||||
* A dedicated special remote, <https://github.com/encryptio/git-annex-remote-b2>
|
||||
(Last updated 2016)
|
||||
|
||||
At this time it's not clear which is better, so if you find one works
|
||||
better than the other, please comment below.
|
||||
|
|
@ -1,9 +1,19 @@
|
|||
[Google Cloud Storage](https://cloud.google.com/products/cloud-storage)
|
||||
supports the same API as Amazon S3, so the
|
||||
[[S3 special remote|special_remotes/S3]] can be used with it.
|
||||
Here is a configuration example:
|
||||
For using [Google Cloud Storage](https://cloud.google.com/products/cloud-storage)
|
||||
as a special remote, there are currently three choices:
|
||||
|
||||
* Google Cloud Storage supports supports the same API as Amazon S3, so
|
||||
git-annex's built-in [[S3 special remote|special_remotes/S3]] can be used
|
||||
with it. You may need to configure Google Cloud Storage to allow
|
||||
"Interoperable Access". Here is how to set up the special remote:
|
||||
|
||||
git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80
|
||||
|
||||
Thanks to jterrance for the [original tip](https://gist.github.com/4576324).
|
||||
--[[Joey]]
|
||||
* Alternatively, rclone supports Google Cloud Storage's native API,
|
||||
so the [[rclone_special_remote|special_remotes/rclone]] can be used.
|
||||
|
||||
* Alternatively, there is a dedicated special remote,
|
||||
<https://github.com/bgilbert/gcsannex>
|
||||
(Last updated 2016)
|
||||
|
||||
At this time it's not clear which is better, so if you find one works
|
||||
best, please comment below.
|
||||
|
|
|
|||
9
doc/tips/using_Google_Drive.mdwn
Normal file
9
doc/tips/using_Google_Drive.mdwn
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
For using [Google Drive](https://google.com/drive)
|
||||
as a special remote, there are currently two choices:
|
||||
|
||||
* A dedicated special remote,
|
||||
<https://github.com/Lykos153/git-annex-remote-googledrive>
|
||||
Includes support for exporttree and other features.
|
||||
* Using [[special_remotes/rclone]]
|
||||
|
||||
If you find one works best, please comment below.
|
||||
9
doc/tips/using_Hubic.mdwn
Normal file
9
doc/tips/using_Hubic.mdwn
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
For using Hubic as a special remote, there are currently two choices:
|
||||
|
||||
* Using [[special_remotes/rclone]]
|
||||
Actively maintained.
|
||||
* A dedicated special remote,
|
||||
<https://github.com/Schnouki/git-annex-remote-hubic>
|
||||
Not actively maintained.
|
||||
|
||||
If you find one works best, please comment below.
|
||||
Loading…
Add table
Reference in a new issue