Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2013-03-13 22:48:14 -04:00
commit 07669f4854
15 changed files with 371 additions and 1 deletions

View file

@ -0,0 +1,100 @@
What steps will reproduce the problem?
Move any file into a git annex repository using "mv". The symlink is created, content is stored in .git/annex, content may even be moved to other repos, but the symlink is not checked into git. A day later, the daily sanity check will add and check-in the link.
What is the expected output? What do you see instead?
After adding content to git-annex, "git status" should show a clean repo with everything checked in.
Instead it looks like this:
~$ touch foobar
~$ rm foobar
~$ echo "foo" > bar
~$ mv bar annex/Incoming/
~$ cd annex
annex$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# Incoming/bar
nothing added to commit but untracked files present (use "git add" to track)
annex$
I ran the assistant with the --debug option and this is what was in the daemon.log when this happened:
[2013-03-13 09:23:36 EDT] read: git ["--git-dir=/Users/ed/annex/.git","--work-tree=/Users/ed/annex","ls-files","--others","--exclude-standard","-z","--","/Users/ed/annex/Incoming/bar"]
[2013-03-13 09:23:37 EDT] read: lsof ["-F0can","+d","/Users/ed/annex/.git/annex/tmp/"]
[2013-03-13 09:23:39 EDT] Committer: Adding bar
(Recording state in git...)
(Recording state in git...)
(Recording state in git...)
(Recording state in git...)
add Incoming/bar (checksum...) recv: resource vanished (Connection reset by peer)
[2013-03-13 09:23:39 EDT] 127.0.0.1 GET /sidebar/NotificationId%201 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20100101 Firefox/19.0
[2013-03-13 09:23:40 EDT] 127.0.0.1 GET /sidebar/NotificationId%201 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20100101 Firefox/19.0
What version of git-annex are you using? On what operating system?
A fresh compile from last night (git hash fe2df85f0db08227556897db3b076ef321e4303a). OS X Lion.
Please provide any additional information below.
I get the same effect when doing this with a newly created "testannex" repo with nothing in it but a newly created file -- I just haven't left the assistant running in a test repo all day to confirm that the file doesn't get checked in till the next daily sanity check.
Here it is on a newly-created, empty repo:
testannex$ git init
Initialized empty Git repository in /Users/ed/testannex/.git/
testannex$ git annex init
init ok
(Recording state in git...)
testannex$ git annex assistant --debug
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","show-ref","git-annex"]
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","show-ref","--hash","refs/heads/git-annex"]
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","log","refs/heads/git-annex..4aeceaff629c83c35c1244da9eeeac4b7c228fef","--oneline","-n1"]
[2013-03-13 09:31:10 EDT] chat: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","cat-file","--batch"]
testannex$ cd
~$ echo "foo" > bar
~$ mv bar testannex/
~$ cd testannex/
testannex$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# bar
nothing added to commit but untracked files present (use "git add" to track)
(meanwhile.....)
~$ cd testannex/.git/annex/
annex$ tail -f daemon.log
[2013-03-13 09:31:10 EDT] TransferWatcher: watching for transfers
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","show-ref","git-annex"]
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","show-ref","--hash","refs/heads/git-annex"]
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","log","refs/heads/git-annex..4aeceaff629c83c35c1244da9eeeac4b7c228fef","--oneline","-n1"]
[2013-03-13 09:31:10 EDT] Merger: watching /Users/ed/testannex/.git/refs
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","ls-tree","-z","--","refs/heads/git-annex","uuid.log","remote.log","trust.log","group.log","preferred-content.log"]
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","ls-tree","-z","--","refs/heads/git-annex","uuid.log","remote.log","trust.log","group.log","preferred-content.log"]
(scanning...) [2013-03-13 09:31:10 EDT] Watcher: Performing startup scan
[2013-03-13 09:31:10 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","ls-files","--deleted","-z","--","/Users/ed/testannex"]
(started...) [2013-03-13 09:31:10 EDT] Watcher: watching .
[2013-03-13 09:31:48 EDT] read: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","ls-files","--others","--exclude-standard","-z","--","/Users/ed/testannex/bar"]
[2013-03-13 09:31:49 EDT] read: lsof ["-F0can","+d","/Users/ed/testannex/.git/annex/tmp/"]
[2013-03-13 09:31:51 EDT] Committer: Adding bar
add bar [2013-03-13 09:31:51 EDT] chat: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","check-attr","-z","--stdin","annex.backend","annex.numcopies","--"]
(checksum...) [2013-03-13 09:31:51 EDT] chat: git ["--git-dir=/Users/ed/testannex/.git","--work-tree=/Users/ed/testannex","cat-file","--batch"]

View file

@ -0,0 +1,18 @@
##What steps will reproduce the problem?
echo text > foo
echo text > bar
git annex add foo bar
git annex unannex foo
##What is the expected output? What do you see instead?
I would expect that the object behind 'bar' remained intact, what happens is that the object is moved out of the annex and 'bar' is left as a dangling symlink, if you are unlucky and don't spot this, it could be potentially dangerous, since you can easily lose data.
##What version of git-annex are you using? On what operating system?
git-annex built from git on Tue Mar 12 15:58:36 2013 -0400
From commit: 70b7555eaf9ac5f88bb137985d93bed8d5a434e8
On Debian Sid

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl1D_4vD5ueaDw8gRsIYPO3UHRKEpFfg9I"
nickname="Дмитрий"
subject="About development in Vim"
date="2013-03-13T02:18:20Z"
content="""
Could you describe your vim configuration for haskell development, what plugins do you use, what is in you .vimrc file and maybe even record a screencast of typical \"session\" of work?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
nickname="Michael"
subject="Global preferences?"
date="2013-03-13T08:00:55Z"
content="""
I just did a \"cabal install git-annex --bindir=$HOME/bin\" on Ubuntu 12.10 and I don't see the global preferences (where you configure the autostart etc.) from the screencast. How can I enable this?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 3"
date="2013-03-13T16:08:35Z"
content="""
The global preferences page is a new feature that will be in the next release. You can use a daily build or build from source to try it now.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmU_2tE75oyG0h2ZPN4lcroIKEMC8G-otE"
nickname="Michael"
subject="comment 4"
date="2013-03-13T16:34:46Z"
content="""
Build from source is too hard for a non-haskell developer, I run into various errors I can not resolve. The daily build gives me \"git-annex: error while loading shared libraries: libyaml-0.so.2: cannot open shared object file: No such file or directory\". For Ubuntu a PPA with daily build would be handy.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 5"
date="2013-03-13T16:58:04Z"
content="""
In the standalone linux daily build, you need to use the \"runshell\" script to start a shell environment in which you can run git-annex.
"""]]

View file

@ -8,6 +8,6 @@ assistant, please instead fill in this poll on behalf of less technically
adept friends or family -- what's preventing you from introducing them to
the assistant?
[[!poll open=yes expandable=yes 7 "I'm using the assistant!" 25 "I need a Windows port" 29 "I need an Android port" 2 "I need an IPhone port (not holding my breath)" 2 "Well, it's still in beta..." 11 "I want to, but have not had the time to try it" 5 "Just inertia. I've got this dropbox/whatever that already works.." 3 "It's too hard to install (please say why in comments)" 2 "Perceived recent increase of bug reports and thus sitting it out." 24 "Initially the lack of direct-mode. Now concerns about the safety of direct mode. Perhaps after the next release." 8 "I haven't always well understood the differences between commandline operation & the assistant, so the differences would confuse me, and I found the command line more understandable & less scary. Now trying to learn to like & trust the assistant. :)" 16 "An Ubuntu PPA would be supercool! Thanks for your great work!!" 18 "Not yet in Debian sid amd64" 4 "Waiting for Fedora/CentOS rpm repository." 2 "throttling transfers, it upsets people when I saturate the connection" 2 "partial content" 1 "Not yet available in macports" 3 "No build yet for Nokia N9" 3 "Using only git-annex webapp to config does not seem to work: Create walkthough?" 5 "No build for OSX 10.6" 4 "Needs more focus on the UI." 1 "Just inertia. I don't have a Dropbox/whatever." 4 "Replaces files with a symlink mess." 2 "configurable option to only annex files meeting certian size or filename criteria" 4 "I'm really confused about how to make it sync with a remote NON-bare repository. I'm even afraid to try `git remote add`, since there is no clear method to completely forget a git-annex remote..." 2 "A build for te raspberry pi would be supercol!" 1 "Would be nice to exclude subfolders from the gui or through a config file" 1 "I wish I had transparently encrypted git repos in the cloud available, like jgit." 1 "too many inodes used in direct mode. maybe it's possible to keep more info as git objects instead?" 2 "I need to be able to restrict in which repo dirs changes get auto-committed" 1 "Provide .deb package" 1 "Better documentation/walkthroughs on using git-annex within an existing git repo. AKA mixed use"]]
[[!poll open=yes expandable=yes 7 "I'm using the assistant!" 25 "I need a Windows port" 29 "I need an Android port" 2 "I need an IPhone port (not holding my breath)" 2 "Well, it's still in beta..." 11 "I want to, but have not had the time to try it" 5 "Just inertia. I've got this dropbox/whatever that already works.." 3 "It's too hard to install (please say why in comments)" 2 "Perceived recent increase of bug reports and thus sitting it out." 24 "Initially the lack of direct-mode. Now concerns about the safety of direct mode. Perhaps after the next release." 8 "I haven't always well understood the differences between commandline operation & the assistant, so the differences would confuse me, and I found the command line more understandable & less scary. Now trying to learn to like & trust the assistant. :)" 18 "An Ubuntu PPA would be supercool! Thanks for your great work!!" 18 "Not yet in Debian sid amd64" 4 "Waiting for Fedora/CentOS rpm repository." 2 "throttling transfers, it upsets people when I saturate the connection" 2 "partial content" 1 "Not yet available in macports" 3 "No build yet for Nokia N9" 3 "Using only git-annex webapp to config does not seem to work: Create walkthough?" 5 "No build for OSX 10.6" 4 "Needs more focus on the UI." 1 "Just inertia. I don't have a Dropbox/whatever." 4 "Replaces files with a symlink mess." 2 "configurable option to only annex files meeting certian size or filename criteria" 4 "I'm really confused about how to make it sync with a remote NON-bare repository. I'm even afraid to try `git remote add`, since there is no clear method to completely forget a git-annex remote..." 2 "A build for te raspberry pi would be supercol!" 1 "Would be nice to exclude subfolders from the gui or through a config file" 1 "I wish I had transparently encrypted git repos in the cloud available, like jgit." 1 "too many inodes used in direct mode. maybe it's possible to keep more info as git objects instead?" 2 "I need to be able to restrict in which repo dirs changes get auto-committed" 1 "Provide .deb package" 1 "Better documentation/walkthroughs on using git-annex within an existing git repo. AKA mixed use"]]
Feel free to write in your own reasons, or add a comment to give me more info.

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
nickname="Jon Ander"
subject="comment 2"
date="2013-03-13T23:52:32Z"
content="""
Hi Joey,
My problem isn't that the repositories don't sync immediately, they are never up at the same time and I understand that the assistant is not constantly checking for changes, so that's not an issue.
The problem is that with this setup I'm experiencing data loss (files being completely removed from both repos), it has happened a few times, but I cant seam to reproduce it reliably. I'll keep investigating and get back to you if I have a proper bug.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmVICFY2CDP08xdsPr3cgmScomy9HA-1sk"
nickname="Andrew"
subject="comment 2"
date="2013-03-13T02:31:52Z"
content="""
It's perfectly possible (though slightly messy) to install git-annex on Ubuntu via [[/install/cabal]], though you need to find and install all the required dependencies yourself (haskell, libidn, libgnutls, libgsasl etc). If you want to get crazy with this look at <http://developer.ubuntu.com/packaging/html/>
"""]]

View file

@ -0,0 +1,17 @@
[[!comment format=mdwn
username="http://sekenre.wordpress.com/"
nickname="sekenre"
subject="Bup remotes in git-annex assistant"
date="2013-03-13T12:54:56Z"
content="""
Hi,
Is the bup remote available via the Assistant user interface?
Unrelated question;
If you are syncing files between two bup repos on local usb drives, does it use git to sync the changes or does it use \"bup split\" to re-add the file? (Basically, is the syncing as efficient as possible using git-annex or would I have to go to a lower level)
Many Thanks,
Sek
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 4"
date="2013-03-13T16:05:50Z"
content="""
I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up.
Your two bup repos would be synced using bup-split.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 5"
date="2013-03-13T16:05:57Z"
content="""
I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up.
Your two bup repos would be synced using bup-split.
"""]]

View file

@ -0,0 +1,111 @@
I've been wrestling with git-annex to try to make it build on Debian, or more specifically, wrestling with Haskell dependencies.
After a fair amount of futzing around, and pestering a bunch of people in the process (thanks for the help! :) ) I finally managed to make it build.
I figured I would post the steps here, since it's not completely trivial, and I expect that a few others might be interested in building newer versions as well.
There appears to currently be two methods:
* Debian packages on Wheezy plus Sid
* Starting out on Wheezy, and then picking the rest from Sid (it seems at least libghc-safesemaphore-dev from Sid is critical for newer git-annex)
* WebDAV suport will not be available with this method
* Cabal packages
#Debian packages on Wheezy plus Sid
##Start off with a clean wheezy chroot
sudo debootstrap weezy debian-wheezy
sudo chroot debian-wheezy
##Install some build tools
apt-get update
apt-get install devscripts git
##Get git-annex (either by cloning or simply moving the source into the chroot)
mkdir /src
cd /src
git clone git://git-annex.branchable.com/source.git git-annex
cd git-annex
##Remove WebDAV dependency which can't be satisfied anywhere
sed '/libghc-dav-dev/d' -i debian/control
##Create dummy build-depends package and install all available Wheezy dependencies using it
mk-build-deps
dpkg -i git-annex-build-deps*.deb
apt-get install -f
(this will remove the build-depends package)
##Add Sid sources and install all available Sid dependencies
echo "deb http://http.debian.net/debian sid main" >>/etc/apt/sources.list
apt-get update
dpkg -i git-annex-build-deps*.deb
apt-get install -f
(the build-depends package should now be fully installed)
##Disable the 'make test' that fails due to missing hothasktags
echo >>debian/rules
echo "override_dh_auto_test:" >>debian/rules
##Build!
debuild -us -uc -Igit
#Cabal packages
##Start off with a clean Sid(/Wheezy) chroot
sudo debootstrap sid debian-sid
sudo chroot debian-sid
##Install a smaller set of tools and build-depends from Debian (cabal needs these to compile the Haskell stuff)
apt-get update
apt-get install ghc cabal-install devscripts libz-dev pkg-config c2hs libgsasl7-dev libxml2-dev libgnutls-dev c2hs git debhelper ikiwiki perlmagick uuid rsync openssh-client fakeroot
##Get git-annex (either by cloning or simply moving the source into the chroot)
mkdir /src
cd /src
git clone git://git-annex.branchable.com/source.git git-annex
cd git-annex
##Install the Haskell build-dependencies from cabal
cabal update
cabal install --only-dependencies
##Optional step which doesn't work (might in the future)
If we want to run the 'make test' after build we need hothasktags, which is only available via cabal
apt-get install happy
cabal install hothasktags
export PATH=$PATH:~/.cabal/bin
But this currently fails silently inside make test->fast->tags, and if you dig a bit (manually edit the makefile to be more verbose) you see
hothasktags: ./Command/AddUnused.hs: hGetContents: invalid argument (invalid byte sequence)
##Disable the 'make test' that fails
echo >>debian/rules
echo "override_dh_auto_test:" >>debian/rules
##Remove all Debian package haskell depends (taken care of by cabal instead)
sed '/\tlibghc/d' -i debian/control
## Build!
debuild -us -uc -Igit

View file

@ -0,0 +1,45 @@
[[!comment format=mdwn
username="https://launchpad.net/~arand"
nickname="arand"
subject="comment 2"
date="2013-03-13T12:05:49Z"
content="""
Based on the hints given here I've worked on a filter to both annex and add urls via filter-branch:
[https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-filter](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-filter)
The script above is very specific but I think there are a few ideas that can be used in general, the general structure is
#!/bin/bash
# links that already exist
links=$(mktemp)
find . -type l >\"$links\"
# remove from staging area first to not block and then annex
git rm --cached --ignore-unmatch -r bin*
git annex add -c annex.alwayscommit=false bin*
# compare links before and after annexing, remove links that existed before
newlinks=$(mktemp -u)
mkfifo \"$newlinks\"
comm -13 <(sort \"$links\") <(find . -type l | sort) > \"$newlinks\" &
# rewrite links
while IFS= read -r file
do
# link is created below .git-rewrite/t/ during filter-branch, strip two parents for correct target
ln -sf \"$(readlink \"$file\" | sed -e 's%^\.\./\.\./%%')\" \"$file\"
done < \"$newlinks\"
git annex merge
which would be run using
git filter-branch --tree-filter path/annex-filter --tag-filter cat -- --all
or similar.
* I'm using `find` to make sure the only rewritten symlinks are for the newly annexed files, this way it is possible to annex an unknown set of filenames
* If doing several git annex commands using `-c annex.alwayscommit=false` and doing a `git annex merge` at the end instead might be faster.
"""]]