119 lines
5.1 KiB
Markdown
119 lines
5.1 KiB
Markdown
This special remote type stores file contents in a bucket in Amazon S3
|
||
or a similar service.
|
||
|
||
See [[tips/using_Amazon_S3]],
|
||
[[tips/Internet_Archive_via_S3]], and [[tips/using_Google_Cloud_Storage]]
|
||
for usage examples.
|
||
|
||
## configuration
|
||
|
||
The standard environment variables `AWS_ACCESS_KEY_ID` and
|
||
`AWS_SECRET_ACCESS_KEY` are used to supply login credentials
|
||
for S3. You need to set these only when running
|
||
`git annex initremote`, as they will be cached in a file only you
|
||
can read inside the local git repository. If you’re working with
|
||
temporary security credentials, you can also set the `AWS_SESSION_TOKEN`
|
||
environment variable.
|
||
|
||
A number of parameters can be passed to `git annex initremote` to configure
|
||
the S3 remote.
|
||
|
||
* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
|
||
See [[encryption]].
|
||
|
||
* `keyid` - Specifies the gpg key to use for [[encryption]].
|
||
|
||
* `chunk` - Enables [[chunking]] when storing large files.
|
||
`chunk=1MiB` is a good starting point for chunking.
|
||
|
||
* `embedcreds` - Optional. Set to "yes" embed the login credentials inside
|
||
the git repository, which allows other clones to also access them. This is
|
||
the default when gpg encryption is enabled; the credentials are stored
|
||
encrypted and only those with the repository's keys can access them.
|
||
|
||
It is not the default when using shared encryption, or no encryption.
|
||
Think carefully about who can access your repository before using
|
||
embedcreds without gpg encryption.
|
||
|
||
* `datacenter` - Defaults to "US". Other values include "EU" (which is EU/Ireland),
|
||
"us-west-1", "us-west-2", "ap-southeast-1", "ap-southeast-2", and
|
||
"sa-east-1".
|
||
|
||
* `storageclass` - Default is "STANDARD".
|
||
Consult S3 provider documentation for pricing details and available
|
||
storage classes.
|
||
|
||
When using Amazon S3, if you have configured git-annex to preserve
|
||
multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
|
||
to save money.
|
||
|
||
Or, if the remote will be used for backup or archival,
|
||
and so its files are Infrequently Accessed, "STANDARD_IA" is also a
|
||
good choice to save money. (Requires a git-annex built with aws-0.13.0)
|
||
|
||
When using Google Cloud Storage, to make a nearline bucket, set this to
|
||
"NEARLINE". (Requires a git-annex built with aws-0.13.0)
|
||
|
||
Note that changing the storage class of an existing S3 remote will
|
||
affect new objects sent to the remote, but not objects already
|
||
stored there.
|
||
|
||
* `host` and `port` - Specify in order to use a different, S3 compatable
|
||
service.
|
||
|
||
* `requeststyle` - Set to "path" to use path style requests, instead of the
|
||
default DNS style requests. This is needed with some S3 services.
|
||
|
||
If you get an error about a host name not existing, it's a good
|
||
indication that you need to use this.
|
||
|
||
* `bucket` - S3 requires that buckets have a globally unique name,
|
||
so by default, a bucket name is chosen based on the remote name
|
||
and UUID. This can be specified to pick a bucket name.
|
||
|
||
* `exporttree` - Set to "yes" to make this special remote usable
|
||
by [[git-annex export|git-annex-export]].
|
||
It will not be usable as a general-purpose special remote.
|
||
|
||
* `versioning` - Setting this to "yes" along with "exporttree=yes",
|
||
and [manually enabling versioning for the S3 bucket in the AWS console](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-versioning.html)
|
||
allows git-annex to access old versions of files exported to the
|
||
special remote with [[git-annex export|git-annex-export]].
|
||
|
||
Note that git-annex needs to remember S3 version IDs for files
|
||
sent to a remote configured this way, which will make the git-annex
|
||
branch a bit larger.
|
||
|
||
Also note that git-annex does not support dropping content from versioned
|
||
S3 buckets.
|
||
|
||
* `public` - Set to "yes" to allow public read access to files sent
|
||
to the S3 remote. This is accomplished by setting an ACL when each
|
||
file is uploaded to the remote. So, changes to this setting will
|
||
only affect subseqent uploads.
|
||
|
||
* `publicurl` - Configure the URL that is used to download files
|
||
from the bucket. Using this in combination with public=yes allows
|
||
git-annex to download files from the S3 remote without needing to
|
||
know the S3 credentials.
|
||
|
||
* `partsize` - Amazon S3 only accepts uploads up to a certian file size,
|
||
and storing larger files requires a multipart upload process.
|
||
|
||
Setting `partsize=1GiB` is recommended for Amazon S3 when not using
|
||
chunking; this will cause multipart uploads to be done using parts
|
||
up to 1GiB in size. Note that setting partsize to less than 100MiB
|
||
will cause Amazon S3 to reject uploads.
|
||
|
||
This is not enabled by default, since other S3 implementations may
|
||
not support multipart uploads or have different limits,
|
||
but can be enabled or changed at any time.
|
||
|
||
* `fileprefix` - By default, git-annex places files in a tree rooted at the
|
||
top of the S3 bucket. When this is set, it's prefixed to the filenames
|
||
used. For example, you could set it to "foo/" in one special remote,
|
||
and to "bar/" in another special remote, and both special remotes could
|
||
then use the same bucket.
|
||
|
||
* `x-amz-meta-*` are passed through as http headers when storing keys
|
||
in S3. see [the Internet Archive S3 interface documentation](https://archive.org/help/abouts3.txt) for example headers.
|