git-annex/doc/chunking.mdwn

Most [[special_remotes]] have support for breaking large files up into
chunks that are stored on the remote.

This can be useful to work around limitations on the size of files
on the remote.

Chunking also allows for resuming interrupted downloads and uploads.

Note that git-annex has to buffer chunks in memory before they are sent to
a remote. So, using a large chunk size will make it use more memory.

To enable chunking, pass a `chunk=nnMiB` parameter to `git annex
initremote`, specifying the chunk size. 

Good chunk sizes will depend on the remote, but a good starting place
is probably `1MiB`. Very large chunks are problimatic, both because
git-annex needs to buffer one chunk in memory when uploading, and because
a larger chunk will make resuming interrupted transfers less efficient.
On the other hand, when a file is split into a great many chunks,
there can be increased overhead of making many requests to the remote.

To disable chunking of a remote that was using chunking,
pass `chunk=0` to `git annex enableremote`. Any content already stored on
the remote using chunks will continue to be accessed via chunks, this
just prevents using chunks when storing new content.

To change the chunk size, pass a `chunk=nnMiB` parameter to
`git annex enableremote`. This only affects the chunk sized used when
storing new content.

# old-style chunking

Note that older versions of git-annex used a different chunk method, which
was configured by passing `chunksize=nnMib` when initializing a remote.

The old-style chunking had a number of problems, including being less
efficient, and not allowing resumes of encrypted uploads.

It's not possible to change a remote using that old chunking method to the
new one, but git-annex continues to support the old-style chunking to
support such remotes.

See also: [[design document|design/assistant/chunks]]