From 1878745a4693913417f65c7c5182e939512e9b22 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 28 Mar 2011 02:12:05 -0400 Subject: [PATCH] more s3 docs --- debian/changelog | 2 +- doc/git-annex.mdwn | 31 +++++++++++++++++++++++++++ doc/internals.mdwn | 11 ++++++++++ doc/special_remotes.mdwn | 32 +--------------------------- doc/todo/S3.mdwn | 2 ++ doc/walkthrough/using_Amazon_S3.mdwn | 22 ++++++++++++++----- 6 files changed, 63 insertions(+), 37 deletions(-) diff --git a/debian/changelog b/debian/changelog index 0469f2242b..945a4ed6b0 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,6 +1,6 @@ git-annex (0.20110326) UNRELEASED; urgency=low - * Amazon is S3 now supported as a special type of remote. + * Amazon S3 is now supported as a special type of remote. * Generalized remotes handling, laying groundwork for remotes that are not regular git remotes. * Provide a less expensive version of `git annex copy --to`, enabled diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn index 8afe93c109..6960c19662 100644 --- a/doc/git-annex.mdwn +++ b/doc/git-annex.mdwn @@ -132,6 +132,16 @@ Many git-annex commands will stage changes for later `git commit` by you. by uuid. To change the description of the current repository, use "." +* s3bucket name description [datacenter host port] + + Creates a bucket in Amazon S3. The bucket's name can be used + to configure git remote using the bucket. + + The datacenter defaults to "US". Other values include "EU", + "us-west-1", and "ap-southeast-1". + + To use a different, S3-compatable service, specify a host and port. + * fsck [path ...] With no parameters, this command checks the whole annex for consistency, @@ -387,6 +397,25 @@ Here are all the supported configuration settings. Default ssh and rsync options to use if a remote does not have specific options. +* `remote..annex-s3-access-key-id` + + Your S3 Access Key ID. Does not need to be kept private. + If not set, the environment variable `AWS_ACCESS_KEY_ID` + will be used. + +* `remote..annex-s3-secret-access-key` + + Your S3 Secret Access Key. This is a password. + If not set, the environment variable `AWS_SECRET_ACCESS_KEY` + will be used. + +* `remote..annex-s3-storageclass` + + Storage class to use when adding new content to S3. The default + is "STANDARD". If you have configured git-annex to preserve + multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY" + to save money. + * `annex.diskreserve` Amount of disk space to reserve. Disk space is checked when transferring @@ -401,6 +430,8 @@ Here are all the supported configuration settings. Automatically maintained, and used to automate upgrades between versions. + + # CONFIGURATION VIA .gitattributes The backend used when adding a new file to the annex can be configured diff --git a/doc/internals.mdwn b/doc/internals.mdwn index a133320b4b..55b1045a11 100644 --- a/doc/internals.mdwn +++ b/doc/internals.mdwn @@ -30,6 +30,17 @@ space and then the description through to the end of the line. Example: e605dca6-446a-11e0-8b2a-002170d25c55 laptop 26339d22-446b-11e0-9101-002170d25c55 usb disk +## `git-annex/s3.log` + +Associates the UUIDs of Amazon S3 buckets with a bucket nickname and connection +information. Example: + + be72acb8-5901-11e0-b600-002170d25c55 mybucket s3.amazonaws.com 80 + +Note that the actual bucket name used on S3 in the above example +is "mybucket-be72acb8-5901-11e0-b600-002170d25c55". The UUID is included +in the bucket name to ensure it is globally unique. + ## `.git-annex/trust.log` Records the [[trust]] information for repositories. Does not exist unless diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn index 3849b0e841..7dc54fd9bf 100644 --- a/doc/special_remotes.mdwn +++ b/doc/special_remotes.mdwn @@ -9,34 +9,4 @@ They cannot be used by other git commands though. ## Amazon S3 Stores file contents in a bucket in Amazon S3 or a similar service. - -Example of configuring such a remote: - - git config remote.mys3.annex-s3-bucket myannex - export ANNEX_S3_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" - export ANNEX_S3_SECRET_ACCESS_KEY="s3kr1t" - -That creates a remote named "mys3" using the bucket named "myannex", -which will be created if it doesn't already exist. - -Here is the full set of configurable settings for Amazon S3. -Each setting can be configured on a per-remote basis in git-config, -or globally in an environment variable. - -* `remote.$name.annex-s3-access-key-id` `ANNEX_S3_ACCESS_KEY_ID` - Your S3 Access Key ID. Does not need to be kept private. -* `remote.$name.annex-s3-secret-access-key` `ANNEX_S3_SECRET_ACCESS_KEY` - Your S3 Secret Access Key. This is a password. Usually stored in the - environment variable to avoid it being exposed. -* `remote.$name.annex-s3-host` `ANNEX_S3_HOST` - Host to connect to. Default is s3.amazonaws.com. -* `remote.$name.annex-s3-port` `ANNEX_S3_PORT` - Port to connect to. Default is 80. -* `remote.$name.annex-s3-datacenter` `ANNEX_S3_DATACENTER` - Name of the datacenter to use when creating a new bucket. Default is "US"; - other valid values include "EU", "us-west-1", and "ap-southeast-1". -* `remote.$name.annex-s3-storageclass` `ANNEX_S3_STORAGECLASS` - Name of storage class to use when adding new content to the bucket. - Default is "STANDARD". If you have configured git-annex to preserve - multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY" to - save money. +See [[walkthrough/using_Amazon_S3]] for examples. diff --git a/doc/todo/S3.mdwn b/doc/todo/S3.mdwn index 09a64f1a7a..356b2af2eb 100644 --- a/doc/todo/S3.mdwn +++ b/doc/todo/S3.mdwn @@ -1,3 +1,5 @@ +[[done]] + Support Amazon S3 as a file storage backend. There's a haskell library that looks good. Not yet in Debian. diff --git a/doc/walkthrough/using_Amazon_S3.mdwn b/doc/walkthrough/using_Amazon_S3.mdwn index d7222731b3..cadd78582c 100644 --- a/doc/walkthrough/using_Amazon_S3.mdwn +++ b/doc/walkthrough/using_Amazon_S3.mdwn @@ -2,14 +2,26 @@ git-annex extends git's usual remotes with some [[special_remotes]], that are not git repositories. This way you can set up a remote using say, Amazon S3, and use git-annex to transfer files into the cloud. - # git config remote.mys3.annex-s3-bucket myannex - # export ANNEX_S3_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" - # export ANNEX_S3_SECRET_ACCESS_KEY="s3kr1t" +First, export your S3 credentials: + + export ANNEX_S3_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" + export ANNEX_S3_SECRET_ACCESS_KEY="s3kr1t" + +Next, create a bucket, giving it a name and a description: + + git annex s3bucket mybucket "my Amazon S3 bucket" + s3bucket (creating mybucket...) ok + +Finally, configure a git remote to use the bucket you created: + + git config remote.mys3.annex-s3-bucket mybucket + +Now the remote can be used like any other remote. + # git annex copy my_cool_big_file --to mys3 copy my_cool_big_file (to mys3...) ok # git annex move video/hackity_hack_and_kaxxt.mov --to mys3 move video/hackity_hack_and_kaxxt.mov (to mys3...) ok An Amazon S3 remote works just like a ssh remote, except it does not have -a git repository at the other end, and it costs you money. :) For full -details about setting them up, see [[special_remotes]]. +a git repository at the other end, and it costs you money. :)