Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2013-08-16 15:56:56 +02:00
commit 6b4b2a6515
11 changed files with 308 additions and 15 deletions

View file

@ -0,0 +1,52 @@
### Please describe the problem.
(small) identical files fail to unannex, leaving broken symlinks, except for the first copy.
### What steps will reproduce the problem?
* Have multiple identical files. For example, run this, which creates four 6-byte files:
> echo Hello>file1.txt && cp file1.txt file2.txt && cp file1.txt file3.txt && cp file1.txt file4.txt
* Run this (git init needs credentials to have been specified though)
> git init && git-annex init && git-annex add
Now there are 4 symlinks, pointing to the same object:
> lrwxrwxrwx 1 186 Aug 16 15:54 file1.txt -> .git/annex/objects/31/XV/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt
* Optionally run "git commit -a". It doesn't affect the outcome.
* Run git-annex unannex
> $ git annex unannex
> unannex file1.txt ok
> (Recording state in git...)
> $
Now file1.txt is a normal 6-byte file again, but 2, 3, and 4 are broken symlinks:
-rw-r----- 1 6 Aug 16 15:54 file1.txt
lrwxrwxrwx 1 186 Aug 16 15:54 file2.txt -> .git/annex/objects/31/XV/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt
lrwxrwxrwx 1 186 Aug 16 15:54 file3.txt -> .git/annex/objects/31/XV/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt
lrwxrwxrwx 1 186 Aug 16 15:54 file4.txt -> .git/annex/objects/31/XV/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt/SHA256E-s6--66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18.txt
$ git-annex fsck
fsck file2.txt
** No known copies exist of file2.txt
failed
fsck file3.txt
** No known copies exist of file3.txt
failed
fsck file4.txt
** No known copies exist of file4.txt
failed
git-annex: fsck: 3 failed
### What version of git-annex are you using? On what operating system?
git-annex 4.20130802 package
on Debian GNU/Linux jessie/sid (testing), amd64.

View file

@ -0,0 +1,20 @@
Using `git-annex version: 4.20130802` on Debian unstable, when trying to add the feed at <http://www.ndr.de/fernsehen/sendungen/extra_3/videos/zum_mitnehmen/extradrei196_version-hq.xml>, I get:
[[!format sh """
importfeed http://www.ndr.de/fernsehen/sendungen/extra_3/videos/zum_mitnehmen/extradrei196_version-hq.xml
--2013-08-16 09:14:13-- http://www.ndr.de/fernsehen/sendungen/extra_3/videos/zum_mitnehmen/extradrei196_version-hq.xml
Auflösen des Hostnamen »www.ndr.de (www.ndr.de)«... 212.201.100.171, 212.201.100.187
Verbindungsaufbau zu www.ndr.de (www.ndr.de)|212.201.100.171|:80... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 61809 (60K) [application/xml]
In »»/tmp/feed4404«« speichern.
100%[============================================>] 61.809 --.-K/s in 0,03s
2013-08-16 09:14:13 (2,20 MB/s) - »»/tmp/feed4404«« gespeichert [61809/61809]
failed
git-annex: importfeed: 1 failed
"""]]
(Oh, and using `format` with nono-ASCII seems to break down., at least in the preview.)

View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="guilhem"
ip="129.16.20.209"
subject="comment 4"
date="2013-08-16T07:14:12Z"
content="""
The [[OpenPGP standard|https://tools.ietf.org/html/rfc4880]] specifies that revoked keys/subkeys \"are not to be used\". AFIK GnuPG, as any RFC-compliant implementation, will not let you encrypt to a revoked key no matter what. An extremely dirty workaround is to set up your system clock prior to the revocation date (but that might put your whole system at risk since other applications may rely synced clocks to work properly).
That said, what you really wanted to do was to revoke access to K1 and add K2 instead. That seems to be a perfectly valid use-case, and it shouldn't be hard to add to git-annex; stay tunned ;-)
Tobias: Not sure what you meant by \"revoke access to my annex\", but if you were thinking of the key owner, note that with the current [[encryption design|http://git-annex.branchable.com/design/encryption]], since that person may simply grab from the git repo and then at any time decrypt the passphrase for the symmetric cipher, it makes little sense to revoke access for that person unless you change that passphrase, and reencrypt all annexed files on the remote, which of course needs to be done locally for the encryption to make sense at all.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawliqfHEW134uawIUPwyKiyOdoF-oI5TxnQ"
nickname="Ethan"
subject="Doh. /usr/bin/ssh-askpass needs to be on *local* machine."
date="2013-08-15T20:13:31Z"
content="""
My mistake; the problem was the I was missing /usr/bin/ssh-askpass on my local machine, not the rsync.net host.
I still think this is a bug. I'm on a debian machine and installed git-annex from its debian package, so seems like the ssh-askpass package should be listed as a dependency. But that's a debian packaging problem, not a git-annex bug per se, so I'll go file it elsewhere.
"""]]

View file

@ -0,0 +1,137 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlmRpGORNKWimtzqItvwm4I6cn16vx8OvU"
nickname="hayden"
subject="Many network sockets with associated fds hanging around"
date="2013-08-15T18:00:42Z"
content="""
I see something similar in logs and after roughly 10 mins the web-apps dies.
So I think I hit the same as the above user.
Sometimes I get thread deaths and restart requests but the root cause appears to match the scenario mentioned over.
Often the webapp just hangs. But always when it hits the fd ulimit... 1024 on this system.
git-annex version is 4.20130802-g1452ac3 and I used the static-linked linux tar.gz linux-binary download.
Test setup is a from scratch assistant startup on Ubuntu 12.04.
Not exactly a clean ubuntu though, so maybe difficult to duplicate troubles at your end.
I fired up the web-app with a cleaned out config. No signs of leaks until an annex is created.
On creation of an empty annex I get fd leaks a about 1 per second after a repository is created.
Strace'ing the main process only shows 8-bytes writes (see below) at the same rate as the leak.
Sometimes the fd-leak stops before the resource limit, sometimes not.
Creating a new annex on top of an existing directory tree with many files is pretty reliable trigger though.
Startup scan finishes and fds leak away until the ulimit is hit.
hayden@orca:~/gamma$ ls /proc/26319/fd
0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 6 8
1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 5 7 9
hayden@orca:~/gamma$ ls /proc/26319/fd
0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 6 8
1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 5 7 9
hayden@orca:~/gamma$ ls /proc/26319/fd
0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 5 7 9
1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 6 8
hayden@orca:~/gamma$ ls /proc/26319/fd/43
/proc/26319/fd/43
hayden@orca:~/gamma$ ls -l /proc/26319/fd/43
ls -l /proc/26319/fd/43
lrwx------ 1 hayden hayden 64 Aug 14 21:10 /proc/26319/fd/43 -> socket:[568994]
hayden@orca:~/gamma$ lsof | grep 568994
git-annex 26319 hayden 43u IPv4 568994 0t0 UDP 224.0.0.251:55556
hayden@orca:~/gamma$ uname -a
Linux orca 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
hayden@orca:~/gamma$ cat /etc/issue
Ubuntu 12.04 LTS \n \l
hayden@orca:~/gamma$ fuser 55556/udp -a
55556/udp: 4415 26319 27080 27083
hayden@orca:~/gamma$ ps aux | grep 4415
hayden 4415 0.0 0.0 66716 3036 ? S Aug12 0:03 curl -s --head -L http://127.0.0.1:38464/?auth=da9c4aba4cc2db9cf78574753f6e94d8031c6a7bdf8bfe100bde868f57b81fd751965cc9d68a9afac79f826d257a256a04ce62615a7f23cd7c925969dda1c7b8 -w %{http_code}
hayden 27458 0.0 0.0 10612 924 pts/0 S+ 21:50 0:00 grep --color=auto 4415
hayden@orca:~/gamma$ ps aux | grep 26319
hayden 26319 0.1 0.7 497188 28560 pts/5 Sl 21:09 0:04 git-annex webapp
hayden 26338 3.4 3.4 1035572 137864 pts/5 Sl 21:09 1:25 /usr/lib/firefox/firefox /tmp/webapp26319.html
hayden 27460 0.0 0.0 10612 920 pts/0 S+ 21:50 0:00 grep --color=auto 26319
hayden@orca:~/gamma$ ps aux | grep 27080
hayden 27080 0.0 0.0 16476 1288 pts/5 S 21:33 0:00 git --git-dir=/home/hayden/boo/.git --work-tree=/home/hayden/boo cat-file --batch
hayden 27462 0.0 0.0 10612 924 pts/0 S+ 21:50 0:00 grep --color=auto 27080
hayden@orca:~/gamma$ ps aux | grep 27083
hayden 27083 0.0 0.0 16476 1060 pts/5 S 21:33 0:00 git --git-dir=/home/hayden/boo/.git --work-tree=/home/hayden/boo check-attr -z --stdin annex.backend annex.numcopies --
hayden 27464 0.0 0.0 10612 920 pts/0 S+ 21:51 0:00 grep --color=auto 27083
----> has 579 open fds at this point but this number holds stable over 10 min
(copy in new tree to provoke)
(no change)
----> restart daemon in gui to provoke
(new process has open fds slowly climbing after startup scan)
straces look like this repeating every second. (clipped down)
futex(0x34f001c, FUTEX_WAIT_PRIVATE, 51, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 202
futex(0x34f001c, FUTEX_WAIT_PRIVATE, 51, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 202
futex(0x34f001c, FUTEX_WAIT_PRIVATE, 51, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 202
futex(0x34f001c, FUTEX_WAIT_PRIVATE, 51, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
write(6, \"\377\0\0\0\0\0\0\0\", 8) = 8
rt_sigreturn(0x2) = 202
futex(0x34f001c, FUTEX_WAIT_PRIVATE, 51, NULL^C <unfinished ...>
Process 29803 detached
Until after roughly 10 mins...
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1017
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1023
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1024
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1024
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1024
hayden@orca:~/boo$ ls /proc/29803/fd | wc -l
1024
hayden@orca:~/boo$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31164
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31164
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
hayden@orca:~/boo$
At this point the webapp hangs but a number of interesting crashes may occur. I've also seen the particular error in the previous users log (on a big tree).
[2013-08-14 21:56:12 CEST] main: starting assistant version 4.20130802-g1452ac3
(scanning...) [2013-08-14 21:56:12 CEST] Watcher: Performing startup scan
(started...) DaemonStatus crashed: /home/hayden/boo/.git/annex/: openTempFile: resource exhausted (Too many open files)
[2013-08-14 22:06:12 CEST] DaemonStatus: warning DaemonStatus crashed: /home/hayden/boo/.git/annex/: openTempFile: resource exhausted (Too many open files)
Is any of the above helpful? Anything else useful to kick for testing that you'd like done?
I'd guess this is something weird with my ubuntu setup that provokes this as more users would see it otherwise.
"""]]

View file

@ -6,7 +6,7 @@ locally paired systems, and remote servers with rsync.
Help me prioritize my work: What special remote would you most like
to use with the git-annex assistant?
[[!poll open=yes 16 "Amazon S3 (done)" 12 "Amazon Glacier (done)" 9 "Box.com (done)" 71 "My phone (or MP3 player)" 20 "Tahoe-LAFS" 9 "OpenStack SWIFT" 31 "Google Drive"]]
[[!poll open=yes 16 "Amazon S3 (done)" 12 "Amazon Glacier (done)" 9 "Box.com (done)" 71 "My phone (or MP3 player)" 20 "Tahoe-LAFS" 10 "OpenStack SWIFT" 31 "Google Drive"]]
This poll is ordered with the options I consider easiest to build
listed first. Mostly because git-annex already supports them and they

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
nickname="Hamza"
subject="comment 6"
date="2013-08-16T09:14:52Z"
content="""
You can either keep running 2 webapps paired using xmpp running all the time that gives you push like notifications or if you are ok with syncing every once in a while you can have a check mail script that adds files on the server commits them then calls git annex sync locally,
ssh $1 \"cd /path/to/annex/;git add .;git commit 'Update'\";git annex sync
should do the trick. The latter method you just need to annex repos no encrypted third repo. Just init your git annex repo on the server and clone on the laptop thats it.
"""]]

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
nickname="Justin"
subject="comment 1"
date="2013-08-16T03:42:00Z"
content="""
moving the annex should work fine, provided:
* You are moving it to a proper unix filesystem - NOT fat32
* The assistant is shutdown. If it is running, bad things will happen.
"""]]

View file

@ -21,23 +21,19 @@ any other git remote. Since XMPP requires a client that is continually running
to see incoming pushes, the XMPP remote cannot be used with git at the
command line.
## Hosted server support status
## XMPP server support status
[[!table data="""
Provider|Status|Type|Notes
Gmail|Working|?|Google apps users will have to configure `.git/annex/creds/xmpp` manually
?|Working|[[Prosody|http://prosody.im/]]
?|Working|[[Metronome|http://www.lightwitch.org/]]
[[Gmail|http://gmail.com]]|Working|?|Google apps users will have to configure `.git/annex/creds/xmpp` manually
[[Coderollers|http://www.coderollers.com/xmpp-server/]]|Working|[[Openfire|http://www.igniterealtime.org/projects/openfire/]]
[[jabber.org|http://jabber.org]]|?|[[Isode M-Link|http://www.isode.com/products/m-link.html]]|Please update. (jabber.org doesn't currently allow new accounts, so if someone have an account, please test. )
Facebook|Failing|?|Maybe non 2-factor will work
League Of Legends|Failing|?
?|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/]]|ejabberd|[[Authentication bug|https://support.process-one.net/browse/EJAB-1632]]: Fixed in debian unstable with version 2.1.10-5
?|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/#comment-4ce5aeabd12ca3016290b3d8255f6ef1]]|jabberd14|No further information
?|?|iChat Server/jabberd2|Please update
?|?|Tigase|Please update
[[jabber.me|http://jabber.me/]]|Working|[[Tigase|http://www.tigase.org/]]
[[xmpp.ru.net|https://www.xmpp.ru.net]]|Working|[[jabberd2|http://jabberd2.org/]]
[[jabber.org|http://jabber.org]]|Working|[[Isode M-Link|http://www.isode.com/products/m-link.html]]
-|Working|[[Prosody|http://prosody.im/]]|No providers tested.
-|Working|[[Metronome|http://www.lightwitch.org/]]|No providers tested.
-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/]]|ejabberd|[[Authentication bug|https://support.process-one.net/browse/EJAB-1632]]: Fixed in debian unstable with version 2.1.10-5
-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/#comment-4ce5aeabd12ca3016290b3d8255f6ef1]]|jabberd14|No further information
"""]]
## Free public XMPP servers
[[http://xmpp.net/]]
List of providers: [[http://xmpp.net/]]
See also: [[xmpp_protocol_design_notes|design/assistant/xmpp]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="http://www.joachim-breitner.de/"
nickname="nomeata"
subject="--fast and --relaxed"
date="2013-08-16T07:27:59Z"
content="""
Hi,
the explanations to --fast and --relaxed on this page could be extended a bit. I looked it up in the man page, but it is not yet clear to me when I would use one or the other with feeds. Also, does “Next time you run git annex addurl it will only fetch any new items.” really only apply to --relaxed, and not --fast?
Furthermore, it would be good if there were a template variable `itemnum` that I can use to ensure that `ls` prints the casts in the right order, even when the titles of the items are not helpful.
Greetings,
Joachim
"""]]

27
doc/tips/imapannex.mdwn Normal file
View file

@ -0,0 +1,27 @@
imapannex
=========
Hook program for gitannex to use imap as backend
# Requirements:
python2
# Install
Clone the git repository in your home folder.
git clone git://github.com/TobiasTheViking/imapannex.git
This should make a ~/imapannex folder
# Setup
Run the program once to set it up.
cd ~/imapannex; python2 imapannex.py
# Commands for gitannex:
git config annex.imap-hook '/usr/bin/python2 ~/imapannex/imapannex.py'
git annex initremote imap type=hook hooktype=imap encryption=shared
git annex describe imap "the imap library"
git annex content imap exclude=largerthan=30mb