mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2014-03-27 13:25:17 -04:00
commit 5a41dd23db
9 changed files with 434 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

195
doc/bugs/Race_condition_between_watch__47__assistant_and_addurl.mdwn Normal file
View file

@ -0,0 +1,195 @@
Addurl can fail due to an apparent race condition when watch or assistant is running and the repository is in direct mode. The following stress test script encounters the bug consistently on my system. I am running git-annex 5.20140320 on on Ubuntu 13.10.
[[!format sh """
#!/bin/sh
set -eu
cleanup() {
local dir
dir="$1"; shift
if [ -d "$dir" ]; then
(
set -x
fuser -k -w "$dir/annex/.git/annex/daemon.log" || :
find "$dir" -type d -exec chmod 700 '{}' '+'
find "$dir" -type f -exec chmod 600 '{}' '+'
rm -fr "$dir"
)
fi
}
go() {
local dir
dir="$(mktemp -d "${TMP:-/tmp}/stress-annex.XXXXXXXXXX")"
trap "cleanup '$dir'" 0 1 2 13 15
(
cd "$dir"
mkdir annex
cd annex
set -x
git init
git annex init
git annex direct
git annex watch
for n in $(seq 100); do
git annex addurl --file=foo http://heh.fi/robots.txt
git annex sync
rm -f foo
git annex sync
done
git annex watch --stop
git annex uninit
)
cleanup "$dir"
trap - 0 1 2 13 14
}
go
"""]]
Script output:
[[!format sh """
% ./stress-annex
+ git init
Initialized empty Git repository in /tmp/stress-annex.OKj6D8kVmV/annex/.git/
+ git annex init
init ok
(Recording state in git...)
+ git annex direct
commit
On branch master
Initial commit
nothing to commit
ok
direct ok
+ git annex watch
+ seq 100
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29-- http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’
[ <=> ] 0 --.-K/s in 0s
2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]
(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit ok
+ rm -f foo
+ git annex sync
commit (Recording state in git...)
ok
(Recording state in git...)
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29-- http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’
[ <=> ] 0 --.-K/s in 0s
2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]
(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit ok
+ rm -f foo
+ git annex sync
commit (Recording state in git...)
ok
(Recording state in git...)
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:14:29-- http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’
[ <=> ] 0 --.-K/s in 0s
2014-03-27 03:14:29 (0.00 B/s) - ‘/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]
git-annex: /tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/: openTempFile: permission denied (Permission denied)
failed
git-annex: addurl: 1 failed
+ fuser -k -w /tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/daemon.log
/tmp/stress-annex.OKj6D8kVmV/annex/.git/annex/daemon.log: 30704 30709 30735 30738 30778
+ find /tmp/stress-annex.OKj6D8kVmV -type d -exec chmod 700 {} +
+ find /tmp/stress-annex.OKj6D8kVmV -type f -exec chmod 600 {} +
+ rm -fr /tmp/stress-annex.OKj6D8kVmV
"""]]
The script also seems to encounter another issue. The output when seq 100 is changed to seq 1 and addurl happens to succeed:
[[!format sh """
+ git init
Initialized empty Git repository in /tmp/stress-annex.QEs0pNyS9z/annex/.git/
+ git annex init
init ok
(Recording state in git...)
+ git annex direct
commit
On branch master
Initial commit
nothing to commit
ok
direct ok
+ git annex watch
+ seq 1
+ git annex addurl --file=foo http://heh.fi/robots.txt
addurl foo (downloading http://heh.fi/robots.txt ...)
--2014-03-27 03:17:20-- http://heh.fi/robots.txt
Resolving heh.fi (heh.fi)... 83.145.237.222
Connecting to heh.fi (heh.fi)|83.145.237.222|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: ‘/tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’
[ <=> ] 0 --.-K/s in 0s
2014-03-27 03:17:20 (0.00 B/s) - ‘/tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/tmp/URL--http&c%%heh.fi%robots.txt’ saved [0/0]
(Recording state in git...)
ok
(Recording state in git...)
+ git annex sync
commit ok
+ rm -f foo
+ git annex sync
commit (Recording state in git...)
ok
(Recording state in git...)
+ git annex watch --stop
+ git annex uninit
git-annex: /tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.map: removeLink: permission denied (Permission denied)
+ fuser -k -w /tmp/stress-annex.QEs0pNyS9z/annex/.git/annex/daemon.log
+ :
+ find /tmp/stress-annex.QEs0pNyS9z -type d -exec chmod 700 {} +
+ find /tmp/stress-annex.QEs0pNyS9z -type f -exec chmod 600 {} +
+ rm -fr /tmp/stress-annex.QEs0pNyS9z
"""]]

2
doc/bugs/__96__git_annex_forget_--drop-dead_--force__96_____95__revives__95___repositories.mdwn
View file

@ -84,5 +84,3 @@ Debian GNU/Linux 7 \n \l
richih@apu (git)-[master] /srv/video/video.debian.net %
"""]]
> [[fixed|done]] via not removing from trust.log --[[Joey]]

90
doc/bugs/__96__git_annex_forget_--drop-dead_--force__96_____95__revives__95___repositories/comment_2_5397d488bc337cb3d7cb46ed774d0437._comment Normal file
View file

@ -0,0 +1,90 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 2"
date="2014-03-26T22:39:34Z"
content="""
Sorry, I had to remove the fixed tag.
The bug makes sense, as does your fix. I didn't even consider that this may be the cause. Still, `git remote rm` and `git annex forget --drop-dead --force` does not seem to be enough to truly get rid of the repo (and its UUID & state):
[[!format sh \"\"\"
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 4
00000000-0000-0000-0000-000000000001 -- web
070cff8a-6302-4aa7-a63c-3fdd34e598a2 -- amazon_s3_us_east--SHA512E
0bae683f-bede-43dd-a815-c4f8fb6db32d -- aws_s3_us_east--SHA512E
92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: git4392
local annex size: 884.64 gigabytes
annexed files in working tree: am4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex dead amazon_s3_us_east--SHA512E
dead amazon_s3_us_east--SHA512E ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex dead aws_s3_us_east--SHA512E
dead aws_s3_us_east--SHA512E ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git remote rm amazon_s3_us_east--SHA512E
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git remote rm aws_s3_us_east--SHA512E
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex forget --drop-dead --force
forget git-annex (Recording state in git...)
ok
(Recording state in git...)
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 2
00000000-0000-0000-0000-000000000001 -- web
92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex initremote amazon_aws_s3-us_east_1--SHA512E type=S3 encryption='none' embedcreds='no' fileprefix='SHA512E/' bucket='debian-video'
initremote amazon_aws_s3-us_east_1--SHA512E (checking bucket...) git-annex: This bucket is already in use by a different S3 special remote, with UUID: 0bae683f-bede-43dd-a815-c4f8fb6db32d
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % cat .git/annex
cat: .git/annex: Ist ein Verzeichnis
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[annex]
uuid = 92e9fac9-97ec-401f-a421-33f6b4f43e47
version = 5
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net % git annex info 537 23:33:37 Mi 26.03.2014
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 2
00000000-0000-0000-0000-000000000001 -- web
92e9fac9-97ec-401f-a421-33f6b4f43e47 -- here (apu.debconf.org/srv/video/conference_videos.annex)
untrusted repositories: 0
transfers in progress: none
available local disk space: 136.3 gigabytes (+1 megabyte reserved)
local annex keys: 4392
local annex size: 884.64 gigabytes
annexed files in working tree: 4628
size of annexed files in working tree: 885.68 gigabytes
bloom filter size: 16 mebibytes (0.9% full)
backend usage:
SHA512E: 9020
richih@apu [2] [0] (git)-[master] /srv/video/video.debian.net %
\"\"\"]]
"""]]

66
doc/bugs/problem_to_addurl_--file_with_ftp.mdwn Normal file
View file

@ -0,0 +1,66 @@
### Please describe the problem.
I want to addurl using ftp protocol.
`git annex addurl ftp://...` works fine, but `git annex addurl --file` fails with an error "failed to verify url exists".
### What steps will reproduce the problem?
setting up a new repo
% alias ga
ga=/home/applis/git-annex.linux/git-annex
% ga init
init ok
(Recording state in git...)
addurl --file works with http
% wget http://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz
[...]
2014-03-27 15:25:06 (10,1 MB/s) - ‘git-annex-standalone-amd64.tar.gz’ saved [30689438/30689438]
% ga add git-annex-standalone-amd64.tar.gz
add git-annex-standalone-amd64.tar.gz ok
(Recording state in git...)
% ga addurl http://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz --file git-annex-standalone-amd64.tar.gz
addurl git-annex-standalone-amd64.tar.gz ok
(Recording state in git...)
addurl works with ftp:
% ga addurl ftp://ftp.belnet.be/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-lxde-desktop.iso.log
addurl ftp.belnet.be_debian_cd_7.4.0_live_i386_iso_hybrid_debian_live_7.4_i386_lxde_desktop.iso.log (downloading ftp://ftp.belnet.be/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-lxde-desktop.iso.log ...)
[...]
2014-03-27 15:27:47 (11,1 MB/s) - ‘/data/annex/.git/annex/tmp/URL--ftp&c%%ftp.belnet.be%debian-cd%7.4.0-live%i386%iso-hybrid%debian-live-7.4-i386-lxde-desktop.iso.log’ saved [1235181]
ok
(Recording state in git...)
addurl --file doesn't work with ftp
% wget ftp://ftp.belnet.be/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-standard.iso.zsync
[...]
2014-03-27 15:29:32 (19,4 MB/s) - ‘debian-live-7.4-i386-standard.iso.zsync’ saved [1932014]
% ga add debian-live-7.4-i386-standard.iso.zsync
add debian-live-7.4-i386-standard.iso.zsync ok
(Recording state in git...)
% ga addurl ftp://ftp.belnet.be/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-standard.iso.zsync --file debian-live-7.4-i386-standard.iso.zsync
addurl debian-live-7.4-i386-standard.iso.zsync
failed to verify url exists: ftp://ftp.belnet.be/debian-cd/7.4.0-live/i386/iso-hybrid/debian-live-7.4-i386-standard.iso.zsync
failed
git-annex: addurl: 1 failed
### What version of git-annex are you using? On what operating system?
I am using current git-annex binary linux version on Fedora 19.
% which git ; git --version
/usr/bin/git
git version 1.8.3.1
% which ga ; ga version
ga=/home/applis/git-annex.linux/git-annex
git-annex version: 5.20140320-g63535e3
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4

41
doc/bugs/problems_with_glacier.mdwn Normal file
View file

@ -0,0 +1,41 @@
### Please describe the problem.
Annex errors when copying to glacier.
### What version of git-annex are you using? On what operating system?
OS X 10.9.2 Build 13C64
git-annex version: 5.20140318-gdcf93d0
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
> git annex initremote glacier type=glacier encryption=hybrid keyid=E9053BDA datacenter=us-west-1 ║██████████╠ ∞ ∞
initremote glacier (encryption setup) (hybrid cipher with gpg key B608B8F6E9053BDA) ok
(Recording state in git...)
> git annex copy Cobalt\ Strike\ Tradecraft --to=glacier --debug
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","show-ref","git-annex"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","show-ref","--hash","refs/heads/git-annex"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","log","refs/heads/git-annex..9f59057d857784e6ae6b3dcd6793092264375913","--oneline","-n1"]
[2014-03-27 07:27:39 PDT] chat: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","cat-file","--batch"]
[2014-03-27 07:27:39 PDT] read: git ["config","--null","--list"]
[2014-03-27 07:27:39 PDT] read: git ["--git-dir=/Users/akraut/Desktop/annexes/media/.git","--work-tree=/Users/akraut/Desktop/annexes/media","ls-files","--cached","-z","--","Cobalt Strike Tradecraft"]
copy Cobalt Strike Tradecraft/Tradecraft__1_of_9____Introduction.mp4 (gpg) [2014-03-27 07:27:39 PDT] chat: gpg ["--quiet","--trust-model","always","--decrypt"]
You need a passphrase to unlock the secret key for
user: "Andrew Mark Kraut <akraut@gmail.com>"
4096-bit ELG-E key, ID 353E49B9, created 2008-11-11 (main key ID E9053BDA)
(checking glacier...) [2014-03-27 07:27:46 PDT] read: glacier ["--region=us-west-1","archive","checkpresent","git-annex: Maybe.fromJust: Nothing
# End of transcript or log.
"""]]

12
doc/bugs/problems_with_glacier/comment_1_8d233428a16ae4276d9c69b329e8216b._comment Normal file
View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q"
nickname="Andrew"
subject="comment 1"
date="2014-03-27T14:46:19Z"
content="""
I just updated to the latest glacier-cli and boto and have confirmed that those are working properly and that the vault has been created on glacier:
[[!format sh \"\"\"
> glacier --region=us-west-1 vault list
glacier-571d1ec3-8870-46cb-977e-15830a2b474d
\"\"\"]]
"""]]

10
doc/forum/2_clients_using_an_encrypted_server_for_syncing_-_possible__63__/comment_3_f9a369a6ac69f091e6128990274d3228._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.41"
subject="comment 3"
date="2014-03-26T21:10:39Z"
content="""
What kind of encrypted remote are you using? An encrypted rsync special remote does not include the git repository, but only the content of the files, so cannot be used in this way.
If you set up a [[gcrypt special remote|special_remotes/gcrypt]], it will be encrypted and includes the full git repository, as well as the content of the files, so can be used like this.
"""]]

10
doc/forum/telehash_syncing.mdwn Normal file
View file

@ -0,0 +1,10 @@
Hi
I have read some info about telehash. It looks verry promising. I was wondering though how syncing will work. For example. I have 2 computers. Normal PC and a laptop. Mostly only one is on at a time.
* Sync messages will be sent over telehash protocoll ?
* What if I push some changes (they will be synced to a shared repository) and laptop is not online. How will git-annex know what to sync from a shared repository ?
* Do you plan to send files/commits directly to online clients ? If 2 friends are online at the same time.
* What will happen with data on a shared repository if all clients have synced content ? Will it be deleted since it is not longer needed ?
I was thinking of a model where you sync directly (if possible), and just drop shared content to repo for offline users. Whan everyone have pulled content it may be removed from shared repo.

10
doc/todo/clear_file_names_in_special_remotes/comment_2_823c279683ac3f39c921be3fcbf6bfe2._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 2"
date="2014-03-26T22:32:18Z"
content="""
In that case, we would need to export the same file name several times, just like direct mode does.
Could files be tracked via metadata? And yes, fsck would be... interesting...
"""]]