update
This commit is contained in:
parent
684011175c
commit
a31dc74806
2 changed files with 2 additions and 5 deletions
182
doc/git-annex.mdwn
Normal file
182
doc/git-annex.mdwn
Normal file
|
@ -0,0 +1,182 @@
|
|||
git-annex allows managing files with git, without checking the file
|
||||
contents into git. While that may seem paradoxical, it is useful when
|
||||
dealing with files larger than git can currently easily handle, whether due
|
||||
to limitations in memory, checksumming time, or disk space.
|
||||
|
||||
Even without file content tracking, being able to manage files with git,
|
||||
move files around and delete files with versioned directory trees, and use
|
||||
branches and distributed clones, are all very handy reasons to use git. And
|
||||
annexed files can co-exist in the same git repository with regularly
|
||||
versioned files, which is convenient for maintaining documents, Makefiles,
|
||||
etc that are associated with annexed files but that benefit from full
|
||||
revision control.
|
||||
|
||||
Enough broad picture, here's how it actually looks:
|
||||
|
||||
* `git annex add $file` moves the file into `.git/annex/`, and replaces
|
||||
it with a symlink pointing at the annexed file, and then calls `git add`
|
||||
to version the *symlink*. (If the file has already been annexed, it does
|
||||
nothing.)
|
||||
* If you use normal git push/pull commands, the annexed file content
|
||||
won't be transferred, but the symlinks will be. So different clones of a
|
||||
repository can have different sets of annexed files available.
|
||||
* You can move the symlink around, copy it, delete it, etc, and commit changes
|
||||
as desired using git. Reading the symlink will always get you the annexed
|
||||
file content, or the link may be broken if the content is not currently
|
||||
available.
|
||||
* `git annex push $repository` pushes *all* annexed files to the specified
|
||||
repository.
|
||||
* `git annex pull $repository` pulls *all* annexed files from the specified
|
||||
repository.
|
||||
* `git annex want $file` indicates that you want access to a file's
|
||||
content, without immediatly transferring it.
|
||||
* `git annex get $file` is used to transfer a specified file, and/or
|
||||
files previously indicated with `git annex want`. If a configured
|
||||
repository has it, or it is available from other key/value storage,
|
||||
it will be immediatly downloaded.
|
||||
* `git annex drop $file` indicates that you no longer want the file's
|
||||
content to be available in this repository.
|
||||
* `git annex unannex $file` undoes a `git annex add`. But use `git annex drop`
|
||||
if you're just done with a file; only use `unannex` if you
|
||||
accidentially added a file.
|
||||
|
||||
Oh yeah, "$file" in the above can be any number of files, or directories,
|
||||
same as you'd pass to "git add" or "git rm".
|
||||
So "git annex add ." or "git annex get dir/" work fine.
|
||||
|
||||
## copies
|
||||
|
||||
git-annex can be configured to try to keep N copies of a file's content
|
||||
available across all repositories. By default, N is 1; it is configured by
|
||||
annex.numcopies.
|
||||
|
||||
`git annex drop` attempts to check with other git remotes, to check that N
|
||||
copies of the file exist. If enough repositories cannot be verified to have
|
||||
it, it will retain the file content to avoid data loss.
|
||||
|
||||
For example, consider three repositories: Server, Laptop, and USB. Both Server
|
||||
and USB have a copy of a file, and N=1. If on Laptop, you `git annex get
|
||||
$file`, this will transfer it from either Server or USB (depending on which
|
||||
is available), and there are now 3 copies of the file.
|
||||
|
||||
Suppose you want to free up space on Laptop again, and you `git annex drop` the file
|
||||
there. If USB is connected, or Server can be contacted, git-annex can check
|
||||
that it still has a copy of the file, and the content is removed from
|
||||
Laptop. But if USB is currently disconnected, and Server also cannot be
|
||||
contacted, it can't verify that it is safe to drop the file, and will
|
||||
refuse to do so.
|
||||
|
||||
With N=2, in order to drop the file content from Laptop, it would need access
|
||||
to both USB and Server.
|
||||
|
||||
Note that different repositories can be configured with different values of
|
||||
N. So just because Laptop has N=2, this does not prevent the number of
|
||||
copies falling to 1, when USB and Server have N=1.
|
||||
|
||||
## key/value storage
|
||||
|
||||
git-annex uses a key/value abstraction layer to allow file contents to be
|
||||
stored in different ways. In theory, any key/value storage system could be
|
||||
used to store the file contents, and git-annex would then retrieve them
|
||||
as needed and put them in `.git/annex/`.
|
||||
|
||||
When a file is annexed, a key is generated from its content and/or metadata.
|
||||
The file checked into git symlinks to the key. This key can later be used
|
||||
to retrieve the file's content (its value). This key generation must be
|
||||
stable for a given file content, name, and size.
|
||||
|
||||
Multiple pluggable backends are supported, and more than one can be used
|
||||
to store different files' contents in a given repository.
|
||||
|
||||
* `WORM` ("Write Once, Read Many") This backend stores the file's content
|
||||
only in `.git/annex/`, and assumes that any file with the same basename,
|
||||
size, and modification time has the same content. So with this backend,
|
||||
files can be moved around, but should never be added to or changed.
|
||||
This is the default, and the least expensive backend.
|
||||
* `SHA1` -- This backend stores the file's content in
|
||||
`.git/annex/`, with a name based on its sha1 checksum. This backend allows
|
||||
modifications of files to be tracked. Its need to generate checksums
|
||||
can make it slow for large files.
|
||||
* `URL` -- This backend downloads the file's content from an external URL.
|
||||
|
||||
## location tracking
|
||||
|
||||
git-annex keeps track of on which repository it last saw a file's content.
|
||||
This can be useful when using it for archiving with offline storage. When
|
||||
you indicate you want a file, git-annex will tell you which repositories
|
||||
have the file's content. For example:
|
||||
|
||||
# git annex get myfile
|
||||
git-annex: unable to get: myfile
|
||||
To get that file, need access to one of these remotes: usbdrive
|
||||
|
||||
Location tracking information is stored in `.git-annex/$key.log`.
|
||||
Repositories record their UUID and the date when they get or drop
|
||||
a file's content. (Git is configured to use a union merge for this file,
|
||||
so the lines may be in arbitrary order, but it will never conflict.)
|
||||
|
||||
The optional file `.git-annex/uuid.log` can be created to add a description
|
||||
to a UUID. If git-annex needs a file from some repository, and it cannot find
|
||||
the repository amoung the remotes, it will use the description from this
|
||||
file when asking for the repository to be made available. The file format
|
||||
is a UUID, a space, and the rest of the line is its description. For
|
||||
example:
|
||||
|
||||
UUID d3d2474c-d5c3-11df-80a9-002170d25c55 USB drive in red enclosure
|
||||
UUID 60cf39c8-d5c6-11df-aa8b-93fda39008d6 my colocated server
|
||||
|
||||
## configuration
|
||||
|
||||
* `annex.uuid` -- a unique UUID for this repository
|
||||
* `annex.numcopies` -- number of copies of files to keep (default: 1)
|
||||
* `annex.backends` -- space-separated list of names of
|
||||
the key/value backends to use. The first listed is used to store
|
||||
new files. (default: "WORM SHA1 URL")
|
||||
* `remote.<name>.annex-cost` -- When determining which repository to
|
||||
transfer annexed files from or to, ones with lower costs are preferred.
|
||||
The default cost is 100 for local repositories, and 200 for remote
|
||||
repositories. Note that other factors may be configured when pushing
|
||||
files to repositories, in particular, whether the repository is on
|
||||
a filesystem with sufficient free space.
|
||||
* `remote.<name>.annex-uuid` -- git-annex caches UUIDs of repositories
|
||||
here.
|
||||
|
||||
## issues
|
||||
|
||||
### symlinks
|
||||
|
||||
If the symlink to annexed content is relative, moving it to a subdir will
|
||||
break it. But it it's absolute, moving the git repo (or mounting its drive
|
||||
elsewhere) will break it. Either:
|
||||
|
||||
* Use relative links and need `git annex mv` to move (or post-commit
|
||||
hook that caches moves and updates links).
|
||||
* Use absolute links and need `git annex fixlinks` when location changes;
|
||||
note that would also mean that git would see the symlink targets changed
|
||||
and want to commit the change. And, other clones of the repo would
|
||||
diverge and there would be conflicts on the symlink text. Ugh.
|
||||
|
||||
Hard links are not an option, because git would then happily commit the
|
||||
file content. Amoung other reasons..
|
||||
|
||||
### free space determination
|
||||
|
||||
Need a way to tell how much free space is available on the disk containing
|
||||
a given repository. The repository may be remote, so ssh may need to be
|
||||
used.
|
||||
|
||||
Similarly, need a way to tell the size of a file before downloading it from
|
||||
remote, to check local disk space.
|
||||
|
||||
### auto-drop files on rm
|
||||
|
||||
When git-rm removed a file, it should get dropped too. Of course, it may
|
||||
not be dropped right away, depending on number of copies available.
|
||||
|
||||
### branching
|
||||
|
||||
The use of `.git-annex` to store logs means that if a repo has branches
|
||||
and the user switched between them, git-annex will see different logs in
|
||||
the different branches, and so may miss info about what remotes have which
|
||||
files (though it can re-learn). An alternative would be to
|
||||
store the log data directly in the git repo as `pristine-tar` does.
|
Loading…
Add table
Add a link
Reference in a new issue