mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

generalize special remote configuration storage

Browse source
This commit is contained in:
Joey Hess 2011-03-28 19:08:12 -04:00
parent caef7c8221
commit 58af574934
4 changed files with 40 additions and 63 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

39
doc/git-annex.mdwn
View file

@ -132,21 +132,17 @@ Many git-annex commands will stage changes for later `git commit` by you.
by uuid. To change the description of the current repository, use by uuid. To change the description of the current repository, use
"." "."
* s3bucket name gpgkey [datacenter host port] * initremote name [type=value param=value ...]
Create or updates the key of a bucket in Amazon S3. The bucket's Sets up a [[special_remote|special_remotes]] of some type. The remote's
name can be used to configure git remote using the bucket. type and configuration is specified by the parameters. If a remote
with the specified name has already been configured, its configuration
is modified by any values specified. In either case, the remote will be
added added to `.git/config`.
The gpgkey is a value that can be looked up (using gpg -k) to Example Amazon S3 remote:
find a gpg encryption key that will be given access to the bucket.
To disable encryption, specify "unencrypted". Note that additional gpg
keys can be given access to a bucket by running s3bucket on an existing
bucket, with a new key.
The datacenter defaults to "US". Other values include "EU", initremote mys3 type=S3 encryption=none datacenter=EU
"us-west-1", and "ap-southeast-1".
To use a different, S3-compatable service, specify a host and port.
* fsck [path ...] * fsck [path ...]
@ -403,25 +399,6 @@ Here are all the supported configuration settings.
Default ssh and rsync options to use if a remote does not have Default ssh and rsync options to use if a remote does not have
specific options. specific options.
* `remote.<name>.annex-s3-access-key-id`
Your S3 Access Key ID. Does not need to be kept private.
If not set, the environment variable `AWS_ACCESS_KEY_ID`
will be used.
* `remote.<name>.annex-s3-secret-access-key`
Your S3 Secret Access Key. This is a password.
If not set, the environment variable `AWS_SECRET_ACCESS_KEY`
will be used.
* `remote.<name>.annex-s3-storageclass`
Storage class to use when adding new content to S3. The default
is "STANDARD". If you have configured git-annex to preserve
multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
to save money.
* `annex.diskreserve` * `annex.diskreserve`
Amount of disk space to reserve. Disk space is checked when transferring Amount of disk space to reserve. Disk space is checked when transferring

14
doc/internals.mdwn
View file

@ -30,16 +30,14 @@ space and then the description through to the end of the line. Example:
e605dca6-446a-11e0-8b2a-002170d25c55 laptop e605dca6-446a-11e0-8b2a-002170d25c55 laptop
26339d22-446b-11e0-9101-002170d25c55 usb disk 26339d22-446b-11e0-9101-002170d25c55 usb disk
## `git-annex/s3.log` ## `git-annex/remotes.log`
Associates the UUIDs of Amazon S3 buckets with a bucket nickname and connection Holds persistent configuration settings for [[special_remotes]] such as
information. Example: Amazon S3.
be72acb8-5901-11e0-b600-002170d25c55 mybucket s3.amazonaws.com 80 The file format is one line per remote, starting with the uuid of the
remote, followed by a space, the name of the remote, a space, and then
Note that the actual bucket name used on S3 in the above example a series of key=value pairs, each separated by whitespace.
is "mybucket-be72acb8-5901-11e0-b600-002170d25c55". The UUID is included
in the bucket name to ensure it is globally unique.
## `.git-annex/trust.log` ## `.git-annex/trust.log`

40
doc/special_remotes/Amazon_S3.mdwn
View file

@ -3,24 +3,36 @@ or a similar service.
See [[walkthrough/using_Amazon_S3]] for usage examples. See [[walkthrough/using_Amazon_S3]] for usage examples.
## bucket names ## initremote parameters
When `git annex s3bucket` is used to create a new bucket, it generates a A number of parameters can be passed to `git annex initremote` to configure
UUID, and the name of the bucket includes that UUID, as well as the name the S3 remote.
specified by the user. This makes for some unweidly bucket names, but
since S3 requires that bucket names be globally unique, it avoids needing * `encryption` - Either "none" to disable encryption,
to hunt for a unused bucket name. or a value that can be looked up (using gpg -k) to find a gpg encryption
key that will be given access to the remote. Note that additional gpg
keys can be given access to a remote by rerunning initremote with
the new key id.
* `datacenter` - Defaults to "US". Other values include "EU",
"us-west-1", and "ap-southeast-1".
* `storageclass` - Default is "STANDARD". If you have configured git-annex
to preserve multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
to save money.
* `host` and `port` - Specify in order to use a different, S3 compatable
service.
## data security ## data security
When `git annex s3bucket` is used to create an unencrypted bucket, When encryption=none, there is **no** protection against your data being read
there is **no** protection against your data being read as it is sent as it is sent to/from S3, or by Amazon when it is stored in S3. This should
to/from S3, or by Amazon when it is stored in S3. This should only be used only be used for public data.
for public data.
** Encryption is not yet supported. ** ** Encryption is not yet supported. **
When an encrypted bucket is created, all files stored in the bucket are When encryption is enabled, all files stored in the bucket are
encrypted with gpg. Additionally, the filenames themselves are hashed encrypted with gpg. Additionally, the filenames themselves are hashed
to obfuscate them. The size of the encrypted files, and access patterns of to obfuscate them. The size of the encrypted files, and access patterns of
the data, should be the only clues to what type of data you are storing in the data, should be the only clues to what type of data you are storing in
@ -36,9 +48,3 @@ encrypted using one or more gpg public keys. This scheme allows new private
keys to be given access to a bucket's content, after the bucket is created keys to be given access to a bucket's content, after the bucket is created
and is in use. The symmetric cipher is also hashed together with filenames and is in use. The symmetric cipher is also hashed together with filenames
used in the bucket, in order to obfuscate the filenames. used in the bucket, in order to obfuscate the filenames.
To add a new gpg key to an existing bucket, just re-run `git annex
s3bucket`, specifying the new key id. For example:
# git annex s3bucket mybucket 16D0B8EF
s3bucket (adding gpg key 16D0B8EF) ok

10
doc/walkthrough/using_Amazon_S3.mdwn
View file

@ -7,18 +7,14 @@ First, export your S3 credentials:
export ANNEX_S3_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" export ANNEX_S3_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
export ANNEX_S3_SECRET_ACCESS_KEY="s3kr1t" export ANNEX_S3_SECRET_ACCESS_KEY="s3kr1t"
Next, create a bucket, giving it a name and a description: Next, create the remote.
git annex s3bucket mybucket unencrypted git annex initremote mys3 encryption=none
s3bucket (creating mybucket...) (no encryption!) ok initremote (creating bucket mys3-291d2fdc-5990-11e0-909a-002170d25c55...) ok
**Note that encrypted buckets are not (yet) supported. Data sent to S3 **Note that encrypted buckets are not (yet) supported. Data sent to S3
is susceptible to snooping.** is susceptible to snooping.**
Finally, configure a git remote to use the bucket you created:
git config remote.mys3.annex-s3-bucket mybucket
Now the remote can be used like any other remote. Now the remote can be used like any other remote.
# git annex copy my_cool_big_file --to mys3 # git annex copy my_cool_big_file --to mys3