mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

improve tip

Browse source
This commit is contained in:
Joey Hess 2020-12-28 19:38:12 -04:00
parent 4262ba3c44
commit 4f64c08f32
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
1 changed files with 24 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

46
doc/tips/using_borg_for_efficient_storage_of_old_annexed_files.mdwn
View file

@ -1,25 +1,31 @@
[Borg](https://www.borgbackup.org/) is a deduplicating archiver If your git-annex repository contains 10 versions of a 100 megabyte file,
with compression and encryption. It can be used with git-annex, as an it will need 1000 megabytes of disk space to store them all. To save space
unusual kind of special remote. Since borg can efficiently store old those old versions can be moved to a remote, but most remotes also don't
versions of files, storing them in borg, rather than the git-annex store similar versions efficiently.
repository can free up disk space.
[Borg](https://www.borgbackup.org/) is a deduplicating archiver
with compression and encryption. This makes it a good solution to this
problem, only the differences between the old versions of the file will be
stored by borg.
Borg can be used with git-annex as an unusual kind of remote.
git-annex is not able to store files in borg itself. Instead the way this git-annex is not able to store files in borg itself. Instead the way this
works is you use borg to store your git-annex repository, and then works is you use borg to store your git-annex repository, and then
`git-annex sync` scans the borg repository to find out what annexed files are `git-annex sync` scans the borg repository to find out what annexed files are
stored in it. And when needed, git-annex can retrieve annexed files from stored in it.
the borg repository.
Let's set that up. Run this from the top directory of your git-annex repository: Let's set that up. Run this from the top directory of your git-annex repository
to create a borg repository next to it that stores all the files in it, and
let git-annex treat it as a remote.
# borg init --encryption=keyfile ../borgrepo # borg init --encryption=keyfile ../borgrepo
# git annex initremote borg type=borg borgrepo=../borgrepo # git annex initremote borg type=borg borgrepo=../borgrepo
# borg create ../borgrepo `pwd`::{now} # borg create ../borgrepo `pwd`::{now}
# git annex sync borg # git annex sync borg
Now git-annex knows that all the files in the repository have been stored Now git-annex knows that all the files in the repository, including all the
in borg. But when you try to drop a file, you'll find that old versions, have been stored in borg. But when you try to drop a file,
git-annex does not trust the borg repository. you'll find that git-annex does not trust the borg repository.
drop file (unsafe) drop file (unsafe)
Could only verify the existence of 0 out of 1 necessary copies Could only verify the existence of 0 out of 1 necessary copies
@ -32,21 +38,17 @@ the content of the file from the borg repository at any time, so git-annex
defaults to not trusting it. This is fine when you're using borg to take defaults to not trusting it. This is fine when you're using borg to take
backups, and need to delete old borg archives to free up space on the backups, and need to delete old borg archives to free up space on the
backup drive. And it can be useful to use git-annex with such borg backups. backup drive. And it can be useful to use git-annex with such borg backups.
But our goal is instead to move old versions of files to borg. So, we need But our goal is instead to move old versions of files to borg. So, you need
to decide not to delete things from the borg repository ourselves, and tell to tell git-annex that you will only use borg to append to the borg
git-annex that we will only use borg to append to the borg repository. repository, not to delete things from it.
# git annex enableremote borg appendonly=yes # git annex enableremote borg appendonly=yes
Now when you use git-annex to drop files, git-annex will treat the borg Now all the old versions of files can be dropped from the git-annex
repository as [[a copy|copies]]. Finally, we can move all the old versions repository, freeing up disk space.
of files to the borg repository.
# git annex unused # git annex unused
# git annex drop --unused # git annex drop --unused
You can continue running `borg create` and `git-annex sync` as your files You can continue running `borg create` and `git-annex sync` to store
changes to store them in borg and let git-annex know what's stored there. changed files in borg and let git-annex know what's stored there.
See [[special_remotes/borg]] for more details about using borg as a special
remote.