mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2012-10-20 16:44:05 -04:00
commit a2385edd29
4 changed files with 132 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

118
doc/forum/Managing_a_large_number_of_files_archived_on_many_pieces_of_read-only_medium___40__E.G._DVDs__41__.mdwn Normal file
View file

@ -0,0 +1,118 @@
I have come up with a moderately complex solution to a particular use case that I have and am posting it here in case it is useful to someone else, and to get suggestions on how to improve it.
#The problem:
I have a large number of files that are accessed infrequently and stored off-line on DVD-Rs. I need to keep track of which files are on which disc so that when I want a file I can find it.
#The solution:
I currently keep a text file to track which files are on which discs. I would like to organize all the files in a proper filesystem using git annex, allowing me better organization and the ability to keep some smaller related files online near the annexed large files.
#Requirements:
1) Easily locate the DVD-R containing any specific offline file
This is easily taken care of with git annex whereis
2) Automatically de-duplicate stored files with the same contents
This is taken care of with one of the hash backends (E.G. SHA256)
3) The DVD-Rs still need to be usable without git or git-annex (E.G. The stored files should retain their normal human readable names)
This requirement rules out dir and rsync special remotes, they store the files named according to their hash. I have settled on making each disc a separate repo which will satisfy this requirement.
#Future goals:
4) Easily incorporate the current DVD-Rs into the new system
I haven't found a way to fulfill this goal yet. I have some convoluted ideas, but nothing so easy as mount disc, run git annex command.
#The solution in detail
Suppose you have the following tree:
<pre>
~/mainrepo/thing1/file1.bin
~/mainrepo/thing1/description1.txt
~/mainrepo/thing2/file2.bin
~/mainrepo/thing2/description2.txt
</pre>
You want to store thing1 on disc1 and thing2 on disc2, but you'd like to keep the descriptions online because they are small and useful for figuring out which thing you want later.
1) Create the main repo and annex the files:
<pre>
cd ~/mainrepo
git init
git annex init mainrepo
git annex add .
git commit -m 'added files'
</pre>
2) Create two new unrelated repos and populate them with their respective data and annex:
<pre>
cd /tmp
mkdir disc1repo disc2repo
cd disc1repo
cp ~/mainrepo/thing1/* .
git init
git annex init disc1
git annex add .
git commit -m 'added files'
cd ../disc2repo
cp ~/mainrepo/thing2/* .
git init
git annex init disc2
git annex add .
git commit -m 'added files'
</pre>
3) This is optional, but after annexing the files in these new repos, I replace the symlinks pointing into to the .git/annex/objects/ directory with hard links. This makes the DVD-Rs usable from operating systems that can't deal with symlinks. (mkisofs handles hard links correctly)
<pre>
cd /tmp
find disc1repo/ disc2repo/ -type l -execdir sh -c "mv -iv {} {}.symlink && ln -L {}.symlink {} && rm {}.symlink" \;
</pre>
4) Burn these repos onto DVD-Rs:
<pre>
cd /tmp
#make isos
mkisofs -volid disc1 -rational-rock -joliet -joliet-long -udf -full-iso9660-filenames -iso-level 3 -o disc1.iso disc1repo/
mkisofs -volid disc2 -rational-rock -joliet -joliet-long -udf -full-iso9660-filenames -iso-level 3 -o disc2.iso disc2repo/
#burn the isos (untested command)
cdrecord -v -dao disc1.iso
cdrecord -v -dao disc2.iso
</pre>
5) Mount the DVD-Rs and add as a remote and fetch, then drop from the mainrepo:
<pre>
cd ~/mainrepo
#disc1
mount /mnt/cdrom
git remote add disc1 /mnt/cdrom
git fetch disc1
git annex drop thing1/thing1.bin
umount /mnt/cdrom
#disc2
mount /mnt/cdrom
git remote add disc2 /mnt/cdrom
git fetch disc2
git annex drop thing2/thing2.bin
umount /mnt/cdrom
</pre>
6) Enjoy! You can now find out what disc things are on simply using git annex whereis, and you can git annex get them or simply use them directly from the disc.
I'd appreciate any comments and helpful suggestions. Especially how to simplify the process or easily integrate all the things I already have stored on discs.
Maybe it would be possible to create a special remote using the hooks for the DVD-Rs.
Even though it is a bit tedious and complicated, the current process could be automated using a script.

10
doc/forum/Managing_a_large_number_of_files_archived_on_many_pieces_of_read-only_medium___40__E.G._DVDs__41__/comment_1_25e65ee3949e7d918376298cf11585f2._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
nickname="John"
subject="Have you seen the dar utility?"
date="2012-10-20T19:03:37Z"
content="""
http://dar.linux.free.fr/doc/index.html
Would be nice to have this as another remote option for git-annex, since I too would like to have static (and possibly incrementally extended) remotes that span multiple DVDs
"""]]

3
doc/forum/New_git-annex_integration_mode_for_Emacs_users.mdwn Normal file
View file

@ -0,0 +1,3 @@
I've started a another project to provide Emacs integration for git-annex: [[https://github.com/jwiegley/git-annex-el]]
My problem with the existing mode is that it didn't feel at all Emacsy, while my mode just piggy backs on existing paradigms, like using `C-x C-q` in a locked file to make it editable, and allowing you to browse and lock/unlock annexed files from Dired.

1
doc/todo/wishlist:_An_--all_option_for_dropunused.mdwn Normal file
View file

@ -0,0 +1 @@
Cleaning out a repository is presently a fairly manual process. Am I missing a UI trick? "dropunsed" with no arguments prints nothing at all; I think in that case it should display the list of what could be dropped.