git-annex/doc/design/encryption.mdwn

This was the design doc for [[/encryption]] and is preserved for
the curious. For an example of using git-annex with an encrypted S3 remote,
see [[tips/using_Amazon_S3]].

[[!toc]]

## encryption key management

[[!template id=note text="""
The basis of this scheme was originally developed by Lars Wirzenius et al
[for Obnam](http://liw.fi/obnam/encryption/).
"""]]

Data is encrypted by GnuPG, using a symmetric cipher. The cipher is
generated by GnuPG when the special remote is created. By default the
best entropy pool is used, hence the generation may take a while; One
can use `initremote` with the `--fast` option
to speed up things, but at the expense of using random numbers of a
lower quality. The generated cipher is then checked into your git
repository, encrypted using one or more OpenPGP public keys. This scheme
allows new OpenPGP private keys to be given access to content that has
already been stored in the remote.

Different encrypted remotes need to be able to each use different ciphers.
Allowing multiple ciphers to be used within a single remote would add a lot
of complexity, so is not supported.
Instead, if you want a new cipher, create a new S3 bucket, or whatever.
There does not seem to be much benefit to using the same cipher for
two different encrypted remotes.

So, the encrypted cipher is just stored with the rest of a remote's
configuration in `remotes.log` (see [[internals]]). When `git
annex intiremote` makes a remote, it generates a random symmetric
cipher, and encrypt it with the specified gpg key. To allow another gpg
public key access, update the encrypted cipher to be encrypted to both gpg
keys.

Note that there's a shared encryption mode where the cipher is not
encrypted. When this mode is used, any clone of the git repository
can decrypt files stored in its special remote.

## filename enumeration

If the names of files are encrypted or securely hashed, or whatever is
chosen, this makes it harder for git-annex (let alone untrusted third parties!)
to get a list of the files that are stored on a given enrypted remote.
But, does git-annex really ever need to do such an enumeration?

Apparently not. `git annex unused --from remote` can now check for
unused data that is stored on a remote, and it does so based only on
location log data for the remote. This assumes that the location log is
kept accurately.

What about `git annex fsck --from remote`? Such a command should be able to,
for each file in the repository, contact the encrypted remote to check
if it has the file. This can be done without enumeration, although it will
mean running gpg once per file fscked, to get the encrypted filename.

So, the files stored in the remote should be encrypted. But, it needs to
be a repeatable encryption, so they cannot just be gpg encrypted, that
would yeild a new name each time. Instead, HMAC is used. Any hash could
be used with HMAC. SHA-1 is the default, but [[other_hashes|/encryption]]
can be chosen for new remotes.

It was suggested that it might not be wise to use the same cipher for both
gpg and HMAC. Being paranoid, it's best not to tie the security of one
to the security of the other. So, the encrypted cipher described above is
actually split in two; the first half is used for HMAC, and the second
half for gpg.

----

Does the HMAC cipher need to be gpg encrypted? Imagine if it were
stored in plainext in the git repository. Anyone who can access
the git repository already knows the actual filenames, and typically also
the content hashes of annexed content. Having access to the HMAC cipher
could perhaps be said to only let them verify that data they already
know.

While this seems a pretty persuasive argument, I'm not 100% convinced, and
anyway, most times that the HMAC cipher is needed, the gpg cipher is also
needed. Keeping the HMAC cipher encrypted does slow down two things:
dropping content from encrypted remotes, and checking if encrypted remotes
really have content. If it's later determined to be safe to not encrypt the
HMAC cipher, the current design allows changing that, even for existing
remotes.

## other use of the symmetric cipher

The symmetric cipher can be used to encrypt other content than the content
sent to the remote. In particular, it may make sense to encrypt whatever
access keys are used by the special remote with the cipher, and store that
in remotes.log. This way anyone whose gpg key has been given access to 
the cipher can get access to whatever other credentials are needed to
use the special remote.

For example, the S3 special remote does this if configured with
embedcreds=yet.

## risks

A risk of this scheme is that, once the symmetric cipher has been
obtained, it allows full access to all the encrypted content. Indeed
anyone owning a key that used to be granted access could already have
decrypted the cipher and stored a copy. While it is in possible to
remove a key with `keyid-=`, it is designed for a
[[completely_different_purpose|/encryption]] and does not actually revoke
access.

If git-annex stores the decrypted symmetric cipher in memory, then there
is a risk that it could be intercepted from there by an attacker. Gpg
ameliorates these type of risks by using locked memory. For git-annex, note
that an attacker with local machine access can tell at least all the
filenames and metadata of files stored in the encrypted remote anyway,
and can access whatever content is stored locally.

This design does not address obfuscating the size of files by chunking
them. However, chunking was later added; see [[design/assistant/chunks]].