git-annex/doc/special_remotes.mdwn

git-annex can transfer data to and from configured git remotes.
Normally those remotes are normal git repositories (bare and non-bare;
local and remote), that store the file contents in their own git-annex
directory.

But, git-annex also extends git's concept of remotes, with these special
types of remotes. These can be used just like any normal remote by git-annex.
They cannot be used by other git commands though.

* [[S3]] (Amazon S3, and other compatible services)
* [[Amazon_Glacier|glacier]]
* [[bup]]
* [[directory]]
* [[rsync]]
* [[webdav]]
* [[web]]
* [[xmpp]]
* [[hook]]

The above special remotes can be used to tie git-annex 
into many cloud services. 

There are many use cases for a special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could git annex move --to specialremote large files when your local drive is getting full, and then git annex move the files back when free space is again available. You could have one repository copy files to a special remote, and then git annex get them on another repository, to transfer the files between computers that do not communicate directly. 

The git-annex assistant makes it easy to set up rsync remotes using this last scenario, which is referred to as a transfer repository, and arranges to drop files from the transfer repository once they have been transferred to all known clients.

None of these use cases are particular to particular special remote types. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses.

Here are specific instructions
for various cloud things:

* [[Amazon_S3|tips/using_Amazon_S3]]
* [[Amazon_Glacier|tips/using_Amazon_Glacier]]
* [[tips/Internet_Archive_via_S3]]
* [[tahoe-lafs|forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]]
* [[Box.com|tips/using_box.com_as_a_special_remote]]
* [[Google drive|tips/googledriveannex]]
* [[Google Cloud Storage|tips/using_Google_Cloud_Storage]]
* [[Mega.co.nz|tips/megaannex]]
* [[SkyDrive|tips/skydriveannex]]
* [[OwnCloud|tips/owncloudannex]]
* [[Flickr|tips/flickrannex]]
* [[IMAP|forum/special_remote_for_IMAP]]
* [[Usenet|forum/nntp__47__usenet special remote]]

## Unused content on special remotes

Over time, special remotes can accumulate file content that is no longer
referred to by files in git. Normally, unused content in the current
repository is found by running `git annex unused`. To detect unused content
on special remotes, instead use `git annex unused --from`. Example:

	$ git annex unused --from mys3
	unused mys3 (checking for unused data...) 
	  Some annexed data on mys3 is not used by any files in this repository.
	    NUMBER  KEY
	    1       WORM-s3-m1301674316--foo
	  (To see where data was previously used, try: git log --stat -S'KEY')
	  (To remove unwanted data: git-annex dropunused --from mys3 NUMBER)
	$ git annex dropunused --from mys3 1
	dropunused 12948 (from mys3...) ok
update 2012-04-11 18:22:59 +00:00			`git-annex can transfer data to and from configured git remotes.`
Added support for getting content from git remotes using http (and https). 2011-08-17 01:12:48 +00:00			`Normally those remotes are normal git repositories (bare and non-bare;`
update 2012-04-11 18:22:59 +00:00			`local and remote), that store the file contents in their own git-annex`
Added support for getting content from git remotes using http (and https). 2011-08-17 01:12:48 +00:00			`directory.`
document S3 remotes 2011-03-28 02:52:13 +00:00
			`But, git-annex also extends git's concept of remotes, with these special`
			`types of remotes. These can be used just like any normal remote by git-annex.`
			`They cannot be used by other git commands though.`

added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3. 2011-05-16 06:07:59 +00:00			`* [[S3]] (Amazon S3, and other compatible services)`
update 2012-11-20 21:13:35 +00:00			`* [[Amazon_Glacier\|glacier]]`
document bup special remotes 2011-04-08 18:56:57 +00:00			`* [[bup]]`
add Remote.Directory 2011-03-30 17:18:46 +00:00			`* [[directory]]`
rsync special remote Fully tested and working, including resuming and encryption. (Though not resuming when sending with encryption; gpg doesn't produce identical output each time.) Uses same layout as the directory special remote and the .git/annex/objects/ directory. 2011-04-28 00:06:07 +00:00			`* [[rsync]]`
update 2012-11-20 21:13:35 +00:00			`* [[webdav]]`
documentation for using the web 2011-07-01 20:01:26 +00:00			`* [[web]]`
add page for xmpp 2013-03-24 13:09:41 +00:00			`* [[xmpp]]`
documentation for hook special remotes Releasing before I have quite finished the code. Got a little caught up in Anathem references. Time for a walk and then a tiny bit more coding and possibly testing. 2011-04-28 19:26:21 +00:00			`* [[hook]]`
instructions for using Box.com as a special remote I was sucked in by the 50 gb free lifetime storage offer. Happily, it was pretty easy to get it to work with git-annex. 2012-03-04 00:06:27 +00:00
			`The above special remotes can be used to tie git-annex`
Adding a use case from joeyh's comment http://git-annex.branchable.com/special_remotes/rsync/#comment-910be826fc1e7a3c01840d4063059907 2013-08-22 20:21:47 +00:00			`into many cloud services.`

			There are many use cases for a special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could git annex move --to specialremote large files when your local drive is getting full, and then git annex move the files back when free space is again available. You could have one repository copy files to a special remote, and then git annex get them on another repository, to transfer the files between computers that do not communicate directly.

			`The git-annex assistant makes it easy to set up rsync remotes using this last scenario, which is referred to as a transfer repository, and arranges to drop files from the transfer repository once they have been transferred to all known clients.`

			`None of these use cases are particular to particular special remote types. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses.`

			`Here are specific instructions`
instructions for using Box.com as a special remote I was sucked in by the 50 gb free lifetime storage offer. Happily, it was pretty easy to get it to work with git-annex. 2012-03-04 00:06:27 +00:00			`for various cloud things:`

better descriptions 2013-05-23 23:28:45 +00:00			`* [[Amazon_S3\|tips/using_Amazon_S3]]`
			`* [[Amazon_Glacier\|tips/using_Amazon_Glacier]]`
instructions for using Box.com as a special remote I was sucked in by the 50 gb free lifetime storage offer. Happily, it was pretty easy to get it to work with git-annex. 2012-03-04 00:06:27 +00:00			`* [[tips/Internet_Archive_via_S3]]`
			`* [[tahoe-lafs\|forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]]`
better descriptions 2013-05-23 23:28:45 +00:00			`* [[Box.com\|tips/using_box.com_as_a_special_remote]]`
gar 2013-05-23 23:29:48 +00:00			`* [[Google drive\|tips/googledriveannex]]`
update 2013-05-31 17:56:12 +00:00			`* [[Google Cloud Storage\|tips/using_Google_Cloud_Storage]]`
better descriptions 2013-05-23 23:28:45 +00:00			`* [[Mega.co.nz\|tips/megaannex]]`
fix link 2013-06-12 17:33:50 +00:00			`* [[SkyDrive\|tips/skydriveannex]]`
add OwnCloud 2013-06-11 14:50:53 +00:00			`* [[OwnCloud\|tips/owncloudannex]]`
better descriptions 2013-05-23 23:28:45 +00:00			`* [[Flickr\|tips/flickrannex]]`
			`* [[IMAP\|forum/special_remote_for_IMAP]]`
			`* [[Usenet\|forum/nntp__47__usenet special remote]]`
unused/dropunused: support --from 2011-04-03 00:59:41 +00:00
			`## Unused content on special remotes`

			`Over time, special remotes can accumulate file content that is no longer`
			`referred to by files in git. Normally, unused content in the current`
			repository is found by running `git annex unused`. To detect unused content
			on special remotes, instead use `git annex unused --from`. Example:

			`$ git annex unused --from mys3`
improve unused command's output Display the name of the remote being checked, with "." for the current remote, echoing the way describe takes that to change its description. 2011-05-29 02:20:22 +00:00			`unused mys3 (checking for unused data...)`
unused/dropunused: support --from 2011-04-03 00:59:41 +00:00			`Some annexed data on mys3 is not used by any files in this repository.`
			`NUMBER KEY`
			`1 WORM-s3-m1301674316--foo`
			`(To see where data was previously used, try: git log --stat -S'KEY')`
			`(To remove unwanted data: git-annex dropunused --from mys3 NUMBER)`
			`$ git annex dropunused --from mys3 1`
			`dropunused 12948 (from mys3...) ok`