Merge branch 'master' into s3-aws

Conflicts:
	Utility/Url.hs
	debian/changelog
	git-annex.cabal
This commit is contained in:
Joey Hess 2014-09-18 14:36:20 -04:00
commit f7847ae98d
282 changed files with 6524 additions and 1207 deletions

View file

@ -13,7 +13,7 @@ can use different ones for different files.
* `SHA256` -- Does not include the file extension in the key, which can
lead to better deduplication but can confuse some programs.
* `WORM` ("Write Once, Read Many") This assumes that any file with
the same basename, size, and modification time has the same content.
the same filename, size, and modification time has the same content.
This is the least expensive backend, recommended for really large
files or slow systems.
* `SHA512`, `SHA512E` -- Best SHA-2 hash, for the very paranoid.

View file

@ -0,0 +1,295 @@
### Please describe the problem.
Installing git-annex on a new Nexus 5 with Android 4.4.4 using [Android 4.4 and 4.3 git-annex.apk](http://downloads.kitenet.net/git-annex/android/current/4.3/git-annex.apk) does not give me a working git-annex environment. It seems permission is denied to install many of the app files.
### What steps will reproduce the problem?
1. Install git-annex
2. From within `adb shell`, run: `/data/data/ga.androidterm/runshell`
3. Try one of the included programs, e.g., `git`
### What version of git-annex are you using? On what operating system?
The current (as of 2014-08-30) git-annex for Android 4.3 and up on Android 4.4.4.
### Please provide any additional information below.
Running `/data/data/ga.androidterm/runshell` from `adb shell` gives me:
[[!format txt """
shell@hammerhead:/ $ /data/data/ga.androidterm/runshell
Falling back to hardcoded app location; cannot find expected files in /data/app-lib
shell@hammerhead:/sdcard/git-annex.home $ ls
git-annex-install.log
shell@hammerhead:/sdcard/git-annex.home $ cat git-annex-install.log
Installation starting to /data/data/ga.androidterm
71c22504d777380dd59d2128b97715fde9ef6bec
mv: can't rename '/data/data/ga.androidterm/bin': Permission denied
installing busybox
ln: /data/data/ga.androidterm/bin/busybox: Permission denied
installing git-annex
ln: /data/data/ga.androidterm/bin/git-annex: Permission denied
installing git-shell
ln: /data/data/ga.androidterm/bin/git-shell: Permission denied
installing git-upload-pack
ln: /data/data/ga.androidterm/bin/git-upload-pack: Permission denied
installing git
ln: /data/data/ga.androidterm/bin/git: Permission denied
installing gpg
ln: /data/data/ga.androidterm/bin/gpg: Permission denied
installing rsync
ln: /data/data/ga.androidterm/bin/rsync: Permission denied
installing ssh
ln: /data/data/ga.androidterm/bin/ssh: Permission denied
installing ssh-keygen
ln: /data/data/ga.androidterm/bin/ssh-keygen: Permission denied
busybox: /data/data/ga.androidterm/bin/[: Permission denied
busybox: /data/data/ga.androidterm/bin/[[: Permission denied
busybox: /data/data/ga.androidterm/bin/ar: Permission denied
busybox: /data/data/ga.androidterm/bin/arp: Permission denied
busybox: /data/data/ga.androidterm/bin/ash: Permission denied
busybox: /data/data/ga.androidterm/bin/base64: Permission denied
busybox: /data/data/ga.androidterm/bin/basename: Permission denied
busybox: /data/data/ga.androidterm/bin/beep: Permission denied
busybox: /data/data/ga.androidterm/bin/blkid: Permission denied
busybox: /data/data/ga.androidterm/bin/blockdev: Permission denied
busybox: /data/data/ga.androidterm/bin/bunzip2: Permission denied
busybox: /data/data/ga.androidterm/bin/bzcat: Permission denied
busybox: /data/data/ga.androidterm/bin/bzip2: Permission denied
busybox: /data/data/ga.androidterm/bin/cal: Permission denied
busybox: /data/data/ga.androidterm/bin/cat: Permission denied
busybox: /data/data/ga.androidterm/bin/catv: Permission denied
busybox: /data/data/ga.androidterm/bin/chat: Permission denied
busybox: /data/data/ga.androidterm/bin/chattr: Permission denied
busybox: /data/data/ga.androidterm/bin/chgrp: Permission denied
busybox: /data/data/ga.androidterm/bin/chmod: Permission denied
busybox: /data/data/ga.androidterm/bin/chown: Permission denied
busybox: /data/data/ga.androidterm/bin/chpst: Permission denied
busybox: /data/data/ga.androidterm/bin/chroot: Permission denied
busybox: /data/data/ga.androidterm/bin/chrt: Permission denied
busybox: /data/data/ga.androidterm/bin/chvt: Permission denied
busybox: /data/data/ga.androidterm/bin/cksum: Permission denied
busybox: /data/data/ga.androidterm/bin/clear: Permission denied
busybox: /data/data/ga.androidterm/bin/cmp: Permission denied
busybox: /data/data/ga.androidterm/bin/comm: Permission denied
busybox: /data/data/ga.androidterm/bin/cp: Permission denied
busybox: /data/data/ga.androidterm/bin/cpio: Permission denied
busybox: /data/data/ga.androidterm/bin/cttyhack: Permission denied
busybox: /data/data/ga.androidterm/bin/cut: Permission denied
busybox: /data/data/ga.androidterm/bin/dc: Permission denied
busybox: /data/data/ga.androidterm/bin/dd: Permission denied
busybox: /data/data/ga.androidterm/bin/deallocvt: Permission denied
busybox: /data/data/ga.androidterm/bin/devmem: Permission denied
busybox: /data/data/ga.androidterm/bin/diff: Permission denied
busybox: /data/data/ga.androidterm/bin/dirname: Permission denied
busybox: /data/data/ga.androidterm/bin/dmesg: Permission denied
busybox: /data/data/ga.androidterm/bin/dnsd: Permission denied
busybox: /data/data/ga.androidterm/bin/dos2unix: Permission denied
busybox: /data/data/ga.androidterm/bin/dpkg: Permission denied
busybox: /data/data/ga.androidterm/bin/dpkg-deb: Permission denied
busybox: /data/data/ga.androidterm/bin/du: Permission denied
busybox: /data/data/ga.androidterm/bin/dumpkmap: Permission denied
busybox: /data/data/ga.androidterm/bin/echo: Permission denied
busybox: /data/data/ga.androidterm/bin/envdir: Permission denied
busybox: /data/data/ga.androidterm/bin/envuidgid: Permission denied
busybox: /data/data/ga.androidterm/bin/expand: Permission denied
busybox: /data/data/ga.androidterm/bin/fakeidentd: Permission denied
busybox: /data/data/ga.androidterm/bin/false: Permission denied
busybox: /data/data/ga.androidterm/bin/fbset: Permission denied
busybox: /data/data/ga.androidterm/bin/fbsplash: Permission denied
busybox: /data/data/ga.androidterm/bin/fdflush: Permission denied
busybox: /data/data/ga.androidterm/bin/fdformat: Permission denied
busybox: /data/data/ga.androidterm/bin/fdisk: Permission denied
busybox: /data/data/ga.androidterm/bin/fgconsole: Permission denied
busybox: /data/data/ga.androidterm/bin/find: Permission denied
busybox: /data/data/ga.androidterm/bin/findfs: Permission denied
busybox: /data/data/ga.androidterm/bin/flash_lock: Permission denied
busybox: /data/data/ga.androidterm/bin/flash_unlock: Permission denied
busybox: /data/data/ga.androidterm/bin/flashcp: Permission denied
busybox: /data/data/ga.androidterm/bin/flock: Permission denied
busybox: /data/data/ga.androidterm/bin/fold: Permission denied
busybox: /data/data/ga.androidterm/bin/freeramdisk: Permission denied
busybox: /data/data/ga.androidterm/bin/ftpd: Permission denied
busybox: /data/data/ga.androidterm/bin/ftpget: Permission denied
busybox: /data/data/ga.androidterm/bin/ftpput: Permission denied
busybox: /data/data/ga.androidterm/bin/fuser: Permission denied
busybox: /data/data/ga.androidterm/bin/getopt: Permission denied
busybox: /data/data/ga.androidterm/bin/grep: Permission denied
busybox: /data/data/ga.androidterm/bin/gunzip: Permission denied
busybox: /data/data/ga.androidterm/bin/gzip: Permission denied
busybox: /data/data/ga.androidterm/bin/hd: Permission denied
busybox: /data/data/ga.androidterm/bin/hdparm: Permission denied
busybox: /data/data/ga.androidterm/bin/head: Permission denied
busybox: /data/data/ga.androidterm/bin/hexdump: Permission denied
busybox: /data/data/ga.androidterm/bin/httpd: Permission denied
busybox: /data/data/ga.androidterm/bin/ifconfig: Permission denied
busybox: /data/data/ga.androidterm/bin/ifdown: Permission denied
busybox: /data/data/ga.androidterm/bin/ifup: Permission denied
busybox: /data/data/ga.androidterm/bin/inotifyd: Permission denied
busybox: /data/data/ga.androidterm/bin/install: Permission denied
busybox: /data/data/ga.androidterm/bin/iostat: Permission denied
busybox: /data/data/ga.androidterm/bin/ip: Permission denied
busybox: /data/data/ga.androidterm/bin/ipaddr: Permission denied
busybox: /data/data/ga.androidterm/bin/ipcalc: Permission denied
busybox: /data/data/ga.androidterm/bin/iplink: Permission denied
busybox: /data/data/ga.androidterm/bin/iproute: Permission denied
busybox: /data/data/ga.androidterm/bin/iprule: Permission denied
busybox: /data/data/ga.androidterm/bin/iptunnel: Permission denied
busybox: /data/data/ga.androidterm/bin/klogd: Permission denied
busybox: /data/data/ga.androidterm/bin/ln: Permission denied
busybox: /data/data/ga.androidterm/bin/loadkmap: Permission denied
busybox: /data/data/ga.androidterm/bin/losetup: Permission denied
busybox: /data/data/ga.androidterm/bin/lpd: Permission denied
busybox: /data/data/ga.androidterm/bin/lpq: Permission denied
busybox: /data/data/ga.androidterm/bin/lpr: Permission denied
busybox: /data/data/ga.androidterm/bin/ls: Permission denied
busybox: /data/data/ga.androidterm/bin/lsattr: Permission denied
busybox: /data/data/ga.androidterm/bin/lsof: Permission denied
busybox: /data/data/ga.androidterm/bin/lspci: Permission denied
busybox: /data/data/ga.androidterm/bin/lsusb: Permission denied
busybox: /data/data/ga.androidterm/bin/lzcat: Permission denied
busybox: /data/data/ga.androidterm/bin/lzma: Permission denied
busybox: /data/data/ga.androidterm/bin/lzop: Permission denied
busybox: /data/data/ga.androidterm/bin/lzopcat: Permission denied
busybox: /data/data/ga.androidterm/bin/makedevs: Permission denied
busybox: /data/data/ga.androidterm/bin/makemime: Permission denied
busybox: /data/data/ga.androidterm/bin/man: Permission denied
busybox: /data/data/ga.androidterm/bin/md5sum: Permission denied
busybox: /data/data/ga.androidterm/bin/mkdir: Permission denied
busybox: /data/data/ga.androidterm/bin/mkfifo: Permission denied
busybox: /data/data/ga.androidterm/bin/mknod: Permission denied
busybox: /data/data/ga.androidterm/bin/mkswap: Permission denied
busybox: /data/data/ga.androidterm/bin/mktemp: Permission denied
busybox: /data/data/ga.androidterm/bin/more: Permission denied
busybox: /data/data/ga.androidterm/bin/mpstat: Permission denied
busybox: /data/data/ga.androidterm/bin/mv: Permission denied
busybox: /data/data/ga.androidterm/bin/nbd-client: Permission denied
busybox: /data/data/ga.androidterm/bin/nc: Permission denied
busybox: /data/data/ga.androidterm/bin/netstat: Permission denied
busybox: /data/data/ga.androidterm/bin/nice: Permission denied
busybox: /data/data/ga.androidterm/bin/nmeter: Permission denied
busybox: /data/data/ga.androidterm/bin/nohup: Permission denied
busybox: /data/data/ga.androidterm/bin/od: Permission denied
busybox: /data/data/ga.androidterm/bin/openvt: Permission denied
busybox: /data/data/ga.androidterm/bin/patch: Permission denied
busybox: /data/data/ga.androidterm/bin/pidof: Permission denied
busybox: /data/data/ga.androidterm/bin/pipe_progress: Permission denied
busybox: /data/data/ga.androidterm/bin/pmap: Permission denied
busybox: /data/data/ga.androidterm/bin/popmaildir: Permission denied
busybox: /data/data/ga.androidterm/bin/printenv: Permission denied
busybox: /data/data/ga.androidterm/bin/printf: Permission denied
busybox: /data/data/ga.androidterm/bin/pscan: Permission denied
busybox: /data/data/ga.androidterm/bin/pstree: Permission denied
busybox: /data/data/ga.androidterm/bin/pwd: Permission denied
busybox: /data/data/ga.androidterm/bin/pwdx: Permission denied
busybox: /data/data/ga.androidterm/bin/raidautorun: Permission denied
busybox: /data/data/ga.androidterm/bin/rdev: Permission denied
busybox: /data/data/ga.androidterm/bin/readlink: Permission denied
busybox: /data/data/ga.androidterm/bin/readprofile: Permission denied
busybox: /data/data/ga.androidterm/bin/realpath: Permission denied
busybox: /data/data/ga.androidterm/bin/reformime: Permission denied
busybox: /data/data/ga.androidterm/bin/renice: Permission denied
busybox: /data/data/ga.androidterm/bin/reset: Permission denied
busybox: /data/data/ga.androidterm/bin/resize: Permission denied
busybox: /data/data/ga.androidterm/bin/rev: Permission denied
busybox: /data/data/ga.androidterm/bin/rm: Permission denied
busybox: /data/data/ga.androidterm/bin/rmdir: Permission denied
busybox: /data/data/ga.androidterm/bin/route: Permission denied
busybox: /data/data/ga.androidterm/bin/rpm: Permission denied
busybox: /data/data/ga.androidterm/bin/rpm2cpio: Permission denied
busybox: /data/data/ga.androidterm/bin/rtcwake: Permission denied
busybox: /data/data/ga.androidterm/bin/run-parts: Permission denied
busybox: /data/data/ga.androidterm/bin/runsv: Permission denied
busybox: /data/data/ga.androidterm/bin/runsvdir: Permission denied
busybox: /data/data/ga.androidterm/bin/rx: Permission denied
busybox: /data/data/ga.androidterm/bin/script: Permission denied
busybox: /data/data/ga.androidterm/bin/scriptreplay: Permission denied
busybox: /data/data/ga.androidterm/bin/sed: Permission denied
busybox: /data/data/ga.androidterm/bin/sendmail: Permission denied
busybox: /data/data/ga.androidterm/bin/seq: Permission denied
busybox: /data/data/ga.androidterm/bin/setconsole: Permission denied
busybox: /data/data/ga.androidterm/bin/setkeycodes: Permission denied
busybox: /data/data/ga.androidterm/bin/setlogcons: Permission denied
busybox: /data/data/ga.androidterm/bin/setserial: Permission denied
busybox: /data/data/ga.androidterm/bin/setsid: Permission denied
busybox: /data/data/ga.androidterm/bin/setuidgid: Permission denied
busybox: /data/data/ga.androidterm/bin/sh: Permission denied
busybox: /data/data/ga.androidterm/bin/sha1sum: Permission denied
busybox: /data/data/ga.androidterm/bin/sha256sum: Permission denied
busybox: /data/data/ga.androidterm/bin/sha512sum: Permission denied
busybox: /data/data/ga.androidterm/bin/showkey: Permission denied
busybox: /data/data/ga.androidterm/bin/sleep: Permission denied
busybox: /data/data/ga.androidterm/bin/smemcap: Permission denied
busybox: /data/data/ga.androidterm/bin/softlimit: Permission denied
busybox: /data/data/ga.androidterm/bin/sort: Permission denied
busybox: /data/data/ga.androidterm/bin/split: Permission denied
busybox: /data/data/ga.androidterm/bin/start-stop-daemon: Permission denied
busybox: /data/data/ga.androidterm/bin/strings: Permission denied
busybox: /data/data/ga.androidterm/bin/stty: Permission denied
busybox: /data/data/ga.androidterm/bin/sum: Permission denied
busybox: /data/data/ga.androidterm/bin/sv: Permission denied
busybox: /data/data/ga.androidterm/bin/svlogd: Permission denied
busybox: /data/data/ga.androidterm/bin/sync: Permission denied
busybox: /data/data/ga.androidterm/bin/sysctl: Permission denied
busybox: /data/data/ga.androidterm/bin/tac: Permission denied
busybox: /data/data/ga.androidterm/bin/tail: Permission denied
busybox: /data/data/ga.androidterm/bin/tar: Permission denied
busybox: /data/data/ga.androidterm/bin/tcpsvd: Permission denied
busybox: /data/data/ga.androidterm/bin/tee: Permission denied
busybox: /data/data/ga.androidterm/bin/test: Permission denied
busybox: /data/data/ga.androidterm/bin/time: Permission denied
busybox: /data/data/ga.androidterm/bin/timeout: Permission denied
busybox: /data/data/ga.androidterm/bin/top: Permission denied
busybox: /data/data/ga.androidterm/bin/touch: Permission denied
busybox: /data/data/ga.androidterm/bin/tr: Permission denied
busybox: /data/data/ga.androidterm/bin/true: Permission denied
busybox: /data/data/ga.androidterm/bin/ttysize: Permission denied
busybox: /data/data/ga.androidterm/bin/tunctl: Permission denied
busybox: /data/data/ga.androidterm/bin/tune2fs: Permission denied
busybox: /data/data/ga.androidterm/bin/udhcpc: Permission denied
busybox: /data/data/ga.androidterm/bin/uname: Permission denied
busybox: /data/data/ga.androidterm/bin/uncompress: Permission denied
busybox: /data/data/ga.androidterm/bin/unexpand: Permission denied
busybox: /data/data/ga.androidterm/bin/uniq: Permission denied
busybox: /data/data/ga.androidterm/bin/unix2dos: Permission denied
busybox: /data/data/ga.androidterm/bin/unlzma: Permission denied
busybox: /data/data/ga.androidterm/bin/unlzop: Permission denied
busybox: /data/data/ga.androidterm/bin/unxz: Permission denied
busybox: /data/data/ga.androidterm/bin/unzip: Permission denied
busybox: /data/data/ga.androidterm/bin/uudecode: Permission denied
busybox: /data/data/ga.androidterm/bin/uuencode: Permission denied
busybox: /data/data/ga.androidterm/bin/vi: Permission denied
busybox: /data/data/ga.androidterm/bin/volname: Permission denied
busybox: /data/data/ga.androidterm/bin/watch: Permission denied
busybox: /data/data/ga.androidterm/bin/wc: Permission denied
busybox: /data/data/ga.androidterm/bin/wget: Permission denied
busybox: /data/data/ga.androidterm/bin/which: Permission denied
busybox: /data/data/ga.androidterm/bin/whoami: Permission denied
busybox: /data/data/ga.androidterm/bin/whois: Permission denied
busybox: /data/data/ga.androidterm/bin/xargs: Permission denied
busybox: /data/data/ga.androidterm/bin/xz: Permission denied
busybox: /data/data/ga.androidterm/bin/xzcat: Permission denied
busybox: /data/data/ga.androidterm/bin/yes: Permission denied
busybox: /data/data/ga.androidterm/bin/zcat: Permission denied
tar: can't remove old file ./links/git-shell: Permission denied
cat: can't open '/data/data/ga.androidterm/links/git': Permission denied
rm: can't stat '/data/data/ga.androidterm/links/git': Permission denied
cat: can't open '/data/data/ga.androidterm/links/git-shell': Permission denied
rm: can't stat '/data/data/ga.androidterm/links/git-shell': Permission denied
cat: can't open '/data/data/ga.androidterm/links/git-upload-pack': Permission denied
rm: can't stat '/data/data/ga.androidterm/links/git-upload-pack': Permission denied
lib/lib.runshell.so: line 133: can't create /data/data/ga.androidterm/runshell: Permission denied
lib/lib.runshell.so: line 133: can't create /data/data/ga.androidterm/runshell: Permission denied
chmod: runshell: Operation not permitted
lib/lib.runshell.so: line 133: can't create /data/data/ga.androidterm/bin/trustedkeys.gpg: Permission denied
lib/lib.runshell.so: line 133: can't create /data/data/ga.androidterm/installed-version: Permission denied
Installation complete
tar: write: Broken pipe
shell@hammerhead:/sdcard/git-annex.home $ ^D
shell@hammerhead:/ $
"""]]
Android is new to me, so it's possible I'm doing something utterly wrong.

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8"
nickname="Jon Ander"
subject="comment 14"
date="2014-09-08T07:27:46Z"
content="""
Still experiencing this bug in Debian testing (5.20140717) and Debian sid (5.20140831)
"""]]

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 1"
date="2014-09-11T17:45:31Z"
content="""
Unfortunately, the old version of git-annex you have been using is exactly the wrong version, so you ran into this horrible bug, which is fixed in newer versions.
<http://git-annex.branchable.com/bugs/bad_merge_commit_deleting_all_files/>
That page has details, including instructions on how to recover your data.
I hope that you were not using that old version because it's included in some distribution somewhere still?
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://andrew.aylett.co.uk/"
nickname="andrew"
subject="comment 2"
date="2014-09-11T19:03:07Z"
content="""
Unfortunately, that bug involves merges while I'm seeing regular commits so I don't think it's identical.
As to why I'm on that version, it appears that the updater and something in my environment conspired against me, leaving an old version in my path. I'll fix that now and let you know if I see the issue again.
"""]]

View file

@ -0,0 +1,44 @@
### Please describe the problem.
When running git-annex info I get an error when it tries to show the bloom filter size
### What steps will reproduce the problem?
git-annex info in my Photos repo
### What version of git-annex are you using? On what operating system?
[[!format sh """
$ git-annex version
git-annex version: 5.20140814-g9b89b5c
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
"""]]
### Please provide any additional information below.
[[!format sh """
$ git-annex info
repository mode: direct
trusted repositories: 2
c0e4106e-2631-11e2-9749-1bfa37a61069 -- [rose]
ca735977-973c-44bc-9257-915b2c875e39 -- synology [here]
semitrusted repositories: 3
00000000-0000-0000-0000-000000000001 -- web
7e5c0010-2634-4a5e-bc7b-6fea84b8b947 -- [glacier]
d7e01abc-d74b-40e2-8607-3d41ce8bc4bd -- seagate3
untrusted repositories: 1
c1fe5922-43f1-11e2-b146-33530f7fa6cc -- x200s
transfers in progress: none
available local disk space: 928.4 gigabytes (+1 megabyte reserved)
local annex keys: 34758
local annex size: 186.78 gigabytes
annexed files in working tree: 35300
size of annexed files in working tree: 193.76 gigabytes
bloom filter size: git-annex: Data.BloomFilter.Util.suggestSizing: capacity too large to represent
"""]]
> I've worked around this problem in the arm autobuilder (only build
> affected), so [[done]] --[[Joey]]

View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 1"
date="2014-09-12T16:03:09Z"
content="""
It seems you must have tweaked the annex.bloomcapacity and/or annex.bloomaccuracy settings, probably to some quite large values.
For example capacity of 50000000 and accuracy of 10000000000 will fail this way.
This happens when it runs out of Double floating point precision to calculate the requested bloom filter size. I think that a bloom filter can be built that has this capacity/accuracy, it's just that Data.BloomFilter.Easy.safeSuggestSizing falls over trying to find the bloom filter size. Also, such a bloom filter may use rather a lot of memory..
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 2"
date="2014-09-12T16:34:56Z"
content="""
However, in Greg's case he had no such configuration. Instead, I think something is broken with the use of floating point or bit math that bloomfilter uses, on the NAS where he's using git-annex.
I have made git-annex not crash when this happens, just show a warning and fall back to a reasonable default bloom filter size. If the problem is with the bit math, then the bloom filter may not work either, which would probably show up as false negatives, so `git annex unused` not finding things that are unused.
I need to update the armel build with this so Greg can test it..
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 3"
date="2014-09-12T16:38:47Z"
content="""
I have reproduced the bug, using the standalone build on an arm box (turtle).
On the same box, the debian git-annex build works ok.
Suggests to me the problem is related to the cross-compiling method used for the standalone arm build.
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="turns out to be an upstream bug already filed"
date="2014-09-12T17:46:23Z"
content="""
It seems that this is a bug on bloomfilter 2.0.0.0 on armel generally. It's also preventing this newer version from building on armel currently:
<http://bugs.debian.org/756801>
The git-annex standalone arm autobuilder installed it with cabal, so ended up with the newer, broken version.
"""]]

View file

@ -21,3 +21,4 @@ Instead of just using the basename, WORM keys could be kept stable by
using the relative path and anchoring it to the root of the
repository.
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,21 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 2"
date="2014-08-16T11:42:22Z"
content="""
Hm, I dont quite follow the remark on having everything in a single
directory. Rather than saying that the relative path adds additional
entropy, what I was aiming at is the file-system cannot have two
alternate versions of one file name at the same path with the same
mtime, and thats why it occurred to me that encoding both path and
mtime within the key doesnt just increase the odds, but effectively
_guarantees_ that there wont be any collisions. Does this seem to
hold up, or am I missing something? (Of course one can fudge the
mtimes, but thats something under the users control.)
While a large repo with many files very likely has lots of distinct
files with identical basename, mtime (in s.) and size, all these files
with the same mtime must necessarily be located at different paths.
"""]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 3"
date="2014-08-16T13:58:28Z"
content="""
One scenario where the above guarantee would be violated is when one
moves a new file of identical size, basename, and mtime, into a path
where a key-colliding file has been kept before. Still, Id consider
this a scenario one could reasonably control for (especially in the
archive usecase); plus, even without manual control such a
move-induced collision would be much more unlikely than a collision of
basenames only.
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.7"
subject="comment 4"
date="2014-08-18T18:39:33Z"
content="""
> Rather than saying that the relative path adds additional entropy, what I was aiming at is the file-system cannot have two alternate versions of one file name at the same path with the same mtime
True of a single filesystem, but not of a set of connected git repositories. :)
So there are multiple scenarios when encoding the file path in the key doesn't help. The probabilities of these seem low, but perhaps not as low as the probability that there will be two differing files with the same name+size+mtime in the first place. It's not clear to me that it adds more than a false sense of security to change from basename to git filename.
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 5"
date="2014-08-18T20:54:10Z"
content="""
> True of a single filesystem, but not of a set of connected git repositories.
Thats a good point. Might depend on the use case, though.
> The probabilities of these seem low, but perhaps not as low as the probability that there will be two differing files with the same name+size+mtime in the first place.
This one Im not completely sure about. E.g., I have an annex with web pages mirrored from the web. Due to the crawler implementation, there are lots of «index.html» or «favicon.ico» with the same mtime (in particular when mtime is read with a 1 sec. precision). Files like favicon are often bitmaps of the same resolution and often have the same size due to this. Because there are file-formats where both size and basename are semantically pre-determined, there is zero entropy from these sources alone (also cf. «readme.txt»). The entropy of mtime alone is not really large, I suppose, and in some use-cases will also approach zero (think «initializing a repo by cp -r on a fast disk without preserving mtime). The relative path could make a huge difference there. I believe this argument is actually what worried me the most. Does it seem valid?
Apart from entropy, theres the non-probabilistic advantage we discussed (granted, with some limiting constraints which one has to assure for oneself). Granted, one might argue a hash would be the better way, but this is not always practical in every setup.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 6"
date="2014-09-11T18:41:45Z"
content="""
Ok, those are good examples. I personally think it would be insane to use WORM in a repository in either of those cases, or really in almost any case where you do not have a strong degree of confidence that unique file contents have unique file names. If people are going to abuse WORM like that, it might be best to simply remove it. (Except I have quite a lot of WORMy disks.)
I suppose I'll add the extra data, although I remain unconvinced that it is going to help anyone who should actually be using WORM.
"""]]

View file

@ -0,0 +1,387 @@
### Please describe the problem.
Using the assistant on two computers to setup a shared encrypted repository (while sharing the same pgp key) on a third computer leads to files not propagating between one and two.
The first and second computer does not get changes done on the other. If new files are added on the first computer it appears as if everything works (no error messages) but the files never reach the second computer (and vice versa).
### What steps will reproduce the problem?
Three computers needed.
* Computer A: Use the assistant to create a repository
* Computer A: Use the assitant to setup a remote repository on Computer C (Add another repository - Remote server - Encrypt with GnuPG key/Encript repository with a new encryption key - Save changes)
[At this point files propagate from A to C]
* Computer A: Export the private and public gpg keys to files
* Computer B: Import these private and public gpg files, fix trust to ultimate
* Computer B: Use the assistant to create a repository
* Computer B: Use the assitant to connect with the remote repository on Computer C (Add another repository - Remote server - Combine the repositories)
[Files created on A before adding B now appear on B]
[New files created on A do not appear on B, new files created on B do not appear on A. Files from A and B seem to propagate to C (the number of files/directories in the object sub directory on C goes up after adding files on A or B)]
### What version of git-annex are you using? On what operating system?
Computer A:
[[!format sh """
dirk@A:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
dirk@A:~$ git-annex version
git-annex version: 5.20140818-g10bf03a
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
dirk@A:~$
dirk@A:~$ gpg --list-keys --list-options show-uid-validity
/home/dirk/.gnupg/pubring.gpg
-----------------------------
pub 4096R/0A7AA2A4 2014-08-23
uid [ultimate] dirk's git-annex encryption key
dirk@A:~$ gpg --list-secret-keys --list-options show-uid-validity
/home/dirk/.gnupg/secring.gpg
-----------------------------
sec 4096R/0A7AA2A4 2014-08-23
uid dirk's git-annex encryption key
dirk@A:~$
"""]]
Computer B:
[[!format sh """
dirk@B:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
dirk@B:~$ git-annex version
git-annex version: 5.20140818-g10bf03a
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
dirk@B:~$
dirk@B:~$ gpg --list-keys --list-options show-uid-validity
/home/dirk/.gnupg/pubring.gpg
-----------------------------
pub 4096R/0A7AA2A4 2014-08-23
uid [ultimate] dirk's git-annex encryption key
dirk@B:~$ gpg --list-secret-keys --list-options show-uid-validity
/home/dirk/.gnupg/secring.gpg
-----------------------------
sec 4096R/0A7AA2A4 2014-08-23
uid dirk's git-annex encryption key
dirk@B:~$
"""]]
Computer C:
[[!format sh """
dirk@C:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.6 (wheezy)
Release: 7.6
Codename: wheezy
dirk@C:~$ git-annex version
git-annex version: 5.20140717~bpo70+1
build flags: Assistant Webapp Pairing S3 Inotify DBus XMPP Feeds Quvi TDFA
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
remote types: git gcrypt S3 bup directory rsync web tahoe glacier ddar hook external
dirk@C:~$
"""]]
### Please provide any additional information below.
.git/annex/daemon.log - Computer A
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2014-08-23 15:15:01 CEST] main: starting assistant version 5.20140818-g10bf03a
[2014-08-23 15:15:01 CEST] Cronner: You should enable consistency checking to protect your data.
(scanning...) [2014-08-23 15:15:01 CEST] Watcher: Performing startup scan
(started...)
gpg: new configuration file `/home/dirk/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/dirk/.gnupg/gpg.conf' are not yet active during this run
Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 235 more bytes)
....+++++
Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 196 more bytes)
.......+++++
gpg: /home/dirk/.gnupg/trustdb.gpg: trustdb created
gpg: key 0A7AA2A4 marked as ultimately trusted
Generating public/private rsa key pair.
Your identification has been saved in /tmp/git-annex-keygen.0/key.
Your public key has been saved in /tmp/git-annex-keygen.0/key.pub.
The key fingerprint is:
7d:02:34:56:d4:86:b6:e5:82:b0:d9:4f:3b:51:b3:c7 dirk@A
The key's randomart image is:
+--[ RSA 2048]----+
| +ooo |
| .o .o * |
| =.o * + |
| o oo= o E |
| Soo+.. |
| +o |
| . |
| |
| |
+-----------------+
(encryption setup) (hybrid cipher with gpg key 7815EA570A7AA2A4) gcrypt: Development version -- Repository format MAY CHANGE
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gcrypt: Repository not found: ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Repository not found: ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
gcrypt: Setting up new repository
gcrypt: Remote ID is :id:00RaA3cNQu+nZDMERYMM
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
* [new branch] git-annex -> git-annex
ok
[2014-08-23 15:25:46 CEST] main: Syncing with C_annex
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:45 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:45 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
* [new branch] git-annex -> synced/git-annex
* [new branch] annex/direct/master -> synced/master
[2014-08-23 15:26:46 CEST] Pusher: Syncing with C_annex
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
Everything up-to-date
[2014-08-23 15:34:01 CEST] Committer: Adding hhhhn.txt
add hhhhn.txt ok
add hhhhn.txt ok
[2014-08-23 15:34:01 CEST] Committer: Committing changes to git
(Recording state in git...)
[2014-08-23 15:34:01 CEST] Pusher: Syncing with C_annex
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
(gpg)
GPGHMACSHA1--7a46226ea53e4043cb45e8df6a2382ac2696164e
74 100% 0.00kB/s 0:00:00
74 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
[2014-08-23 15:34:01 CEST] Transferrer: Uploaded hhhhn.txt
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:33:27 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: WARNING:
gcrypt: WARNING: Remote ID has changed!
gcrypt: WARNING: from :id:00RaA3cNQu+nZDMERYMM
gcrypt: WARNING: to :id:h/BFJbR+mE8CEkASZ/tx
gcrypt: WARNING:
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
85b70d6..e1d6871 annex/direct/master -> synced/master
+ 99dc810...a7a89ff git-annex -> synced/git-annex (forced update)
[2014-08-23 15:34:07 CEST] Pusher: Syncing with C_annex
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:34:04 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
a7a89ff..e68b5a9 git-annex -> synced/git-annex
[2014-08-23 15:48:30 CEST] main: warning git-annex has been shut down
# End of transcript or log.
"""]]
.git/annex/daemon.log - Computer B
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2014-08-23 15:30:11 CEST] main: starting assistant version 5.20140818-g10bf03a
[2014-08-23 15:30:11 CEST] Cronner: You should enable consistency checking to protect your data.
dbus failed; falling back to mtab polling (ClientError {clientErrorMessage = "runClient: unable to determine DBUS address", clientErrorFatal = True})
No known network monitor available through dbus; falling back to polling
(scanning...) [2014-08-23 15:30:11 CEST] Watcher: Performing startup scan
(started...)
Generating public/private rsa key pair.
Your identification has been saved in /tmp/git-annex-keygen.0/key.
Your public key has been saved in /tmp/git-annex-keygen.0/key.pub.
The key fingerprint is:
b5:c3:6b:af:fc:fe:82:f2:a6:f3:42:e9:50:4b:63:9e dirk@A
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . |
| =o . |
| =S=+ |
| . E o |
| + o. |
| =oo.. |
| .O=++o. |
+-----------------+
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Remote ID is :id:00RaA3cNQu+nZDMERYMM
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Remote ID is :id:00RaA3cNQu+nZDMERYMM
Receiving objects: 14% (1/7)
Receiving objects: 28% (2/7)
Receiving objects: 42% (3/7)
Receiving objects: 57% (4/7)
Receiving objects: 71% (5/7)
Receiving objects: 85% (6/7)
Receiving objects: 100% (7/7)
Receiving objects: 100% (7/7), done.
Receiving objects: 12% (1/8)
Receiving objects: 25% (2/8)
Receiving objects: 37% (3/8)
Receiving objects: 50% (4/8)
Receiving objects: 62% (5/8)
Receiving objects: 75% (6/8)
Receiving objects: 87% (7/8)
Receiving objects: 100% (8/8)
Receiving objects: 100% (8/8), done.
From gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex
* [new branch] git-annex -> tmpgcryptremote/git-annex
* [new branch] synced/git-annex -> tmpgcryptremote/synced/git-annex
* [new branch] synced/master -> tmpgcryptremote/synced/master
* [new branch] master -> tmpgcryptremote/master
(merging tmpgcryptremote/git-annex tmpgcryptremote/synced/git-annex into git-annex...)
(Recording state in git...)
(encryption update) (hybrid cipher with gpg key 7815EA570A7AA2A4) gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Remote ID is :id:00RaA3cNQu+nZDMERYMM
From gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex
* [new branch] git-annex -> C_annex/git-annex
* [new branch] synced/git-annex -> C_annex/synced/git-annex
* [new branch] synced/master -> C_annex/synced/master
* [new branch] master -> C_annex/master
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
remote: error: denying non-fast-forward refs/heads/master (you should pull first)
To ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
! [remote rejected] refs/gcrypt/gitception+ -> master (non-fast-forward)
error: failed to push some refs to 'ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/'
error: failed to push some refs to 'gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/'
ok
[2014-08-23 15:31:36 CEST] main: Syncing with C_annex
Automatic merge went well; stopped before committing as requested
Already up-to-date!
gcrypt: Development version -- Repository format MAY CHANGE
[2014-08-23 15:31:37 CEST] Pusher: Syncing with C_annex
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gcrypt: Repository not found: ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
gcrypt: Setting up new repository
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
fatal: ambiguous argument 'refs/heads/synced/master..refs/remotes/C_annex/synced/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Remote ID is :id:h/BFJbR+mE8CEkASZ/tx
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:25:49 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
* [new branch] git-annex -> synced/git-annex
* [new branch] annex/direct/master -> synced/master
fatal: Not a valid object name refs/gcrypt/gitception+
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
5d2eb63..e4763b8 git-annex -> synced/git-annex
da18915..3068bad annex/direct/master -> synced/master
[2014-08-23 15:32:37 CEST] Pusher: Syncing with C_annex
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:31:43 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: WARNING:
gcrypt: WARNING: Remote ID has changed!
gcrypt: WARNING: from :id:00RaA3cNQu+nZDMERYMM
gcrypt: WARNING: to :id:h/BFJbR+mE8CEkASZ/tx
gcrypt: WARNING:
Everything up-to-date
[2014-08-23 15:33:17 CEST] Committer: Adding fmksmxxs.txt
add fmksmxxs.txt ok
add fmksmxxs.txt ok
[2014-08-23 15:33:18 CEST] Committer: Committing changes to git
(Recording state in git...)
[2014-08-23 15:33:18 CEST] Pusher: Syncing with C_annex
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
(gpg) gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:31:43 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
GPGHMACSHA1--f605f108429ffba3058a2fcf0bc006a1fbe600be
70 100% 0.00kB/s 0:00:00
70 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
[2014-08-23 15:33:20 CEST] Transferrer: Uploaded fmksmxxs.txt
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
e4763b8..85dbfc5 git-annex -> synced/git-annex
3068bad..85b70d6 annex/direct/master -> synced/master
[2014-08-23 15:33:25 CEST] Pusher: Syncing with C_annex
(Recording state in git...)
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:33:22 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from "dirk's git-annex encryption key"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
85dbfc5..99dc810 git-annex -> synced/git-annex
[2014-08-23 15:48:39 CEST] main: warning git-annex has been shut down
# End of transcript or log.
"""]]

View file

@ -0,0 +1,35 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawk7iPiqWr3BVPLWEDvJhSSvcOqheLEbLNo"
nickname="Dirk"
subject="comment 1"
date="2014-08-23T18:13:06Z"
content="""
Restarting the two git-annex instances actually now leads to an error message on computer B.
[[!format sh \"\"\"
[2014-08-23 20:02:00 CEST] main: starting assistant version 5.20140818-g10bf03a
[2014-08-23 20:02:00 CEST] Cronner: You should enable consistency checking to protect your data.
dbus failed; falling back to mtab polling (ClientError {clientErrorMessage = \"runClient: unable to determine DBUS address\", clientErrorFatal = True})
[2014-08-23 20:02:00 CEST] TransferScanner: Syncing with C_annex
No known network monitor available through dbus; falling back to polling
(scanning...) [2014-08-23 20:02:00 CEST] Watcher: Performing startup scan
gcrypt: Development version -- Repository format MAY CHANGE
(started...)
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:34:08 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from \"dirk's git-annex encryption key\"
gcrypt: Packfile 59a8d97d3d252effb044625e020f9dc8621804649186a5c33c4e47f9e961cc1a does not match digest!
fatal: early EOF
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gpg: Signature made Sat 23 Aug 2014 03:34:08 PM CEST using RSA key ID 0A7AA2A4
gpg: Good signature from \"dirk's git-annex encryption key\"
gcrypt: Encrypting to: -r 7815EA570A7AA2A4
gcrypt: Requesting manifest signature
To gcrypt::ssh://dirk@git-annex-C-dirk_1022_annex/~/annex/
e1d6871..85b70d6 annex/direct/master -> synced/master
+ e68b5a9...99dc810 git-annex -> synced/git-annex (forced update)
\"\"\"]]
"""]]

View file

@ -0,0 +1,57 @@
### Please describe the problem.
Uploading a 21GB file to an S3 special remote fails. It will generally fail somewhere at about 3-15%. I am using the new chunking feature, with chunks set to 25MiB.
### What steps will reproduce the problem?
$ git annex copy my-big-file.tar.bz --to s3
copy my-big-file.tar.bz (gpg) (checking s3...) (to s3...)
13% 863.8KB/s 6h0m
ErrorClosed
failed
git-annex: copy: 1 failed
### What version of git-annex are you using? On what operating system?
Running on Arch Linux.
git-annex version: 5.20140818-g10bf03a
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
### Please provide any additional information below.
If I fire up the web app and open the log, the end looks like this:
[[!format sh """
...
3% 857.3KB/s 6h46m
3% 857.3KB/s 6h46m
3% 857.3KB/s 6h46m
3% 857.4KB/s 6h46m
3% 857.4KB/s 6h46m
3% 857.5KB/s 6h46m
3% 857.5KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.7KB/s 6h46m
3% 857.7KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.9KB/s 6h46m
3% 857.9KB/s 6h46m
3% 858.0KB/s 6h46m
3% 858.0KB/s 6h46m
3% 858.1KB/s 6h46m
3% 858.1KB/s 6h45m
3% 858.1KB/s 6h45mmux_client_request_session: read from master failed: Broken pipe
"""]]

View file

@ -0,0 +1,14 @@
### Please describe the problem.
When pairing with xmpp buddies, the well does not expand to fit the whole buddy list
### What steps will reproduce the problem?
Go to the pairing menu
### What version of git-annex are you using? On what operating system?
5.20140717 from the homebrew bottle
### Please provide any additional information below.
![image of bug](http://i.imgur.com/fZe1ERD.png)
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,39 @@
### Please describe the problem.
The windows build seems to be hardcoded to finding git at c:\program files\Git\
I have git in another directory. Git-annex does not find it.
### What steps will reproduce the problem?
Install git-annex. Run the webapp.
Get error "Internal Server Error
You need to install git in order to use git-annex!"
### What version of git-annex are you using? On what operating system?
5.20140817-g71c2250.
Windows XP.
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
[[!meta title="git-annex on windows does not find msgit if user does not let msysgit add itsselt to PATH"]]
> I don't think it's any better for git-annex's installer to prompt for the
> path to git, than it is for msysgit's installer to prompt for adding it
> to the system path.
>
> The best fix would be to bundle msysgit into the git-annex installer
> along with all the other stuff. But, that adds build-time complications
> I would rather avoid.
>
> For now, I am going to treat this as a documentation problem;
> I've updated the install page to be clear that msysgit needs to be
> installed into PATH. [[done]] --[[Joey]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="24.159.78.125"
subject="comment 1"
date="2014-08-20T14:37:30Z"
content="""
git-annex does not hardcode any paths, and certianly not the path to git. Your system probably does not have the location you installed git added to the PATH. In that case, git-annex may do what windows programs do and look for git.exe in the same directory it was installed into.
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="Hans_Ryding"
ip="81.229.194.7"
subject="Quite right"
date="2014-08-21T08:54:50Z"
content="""
Incorrect assumption from my part.
I reinstalled git into the expected path (C:\program files\Git)
and the problem is still there.
Running git-annex from command-line works.
(I tried running git-annex test. It had 23 failed tests,
most of them because of inability to access the remote: origin.
But it ran just fine.)
Running the web-app gives the error listed above.
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="Hans_Ryding"
ip="81.229.194.7"
subject="Change the name of the bug"
date="2014-08-21T09:14:16Z"
content="""
I can't seem to change the name of the bug to something more appropriate.
Maybe you can?
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBmv0HhwTFxkpxlf8ifTlMOHnIwHCHTYs"
nickname="y"
subject="path on windows"
date="2014-08-23T22:02:07Z"
content="""
I think I have a related problem on win7 sp1.
When first installing msys git, there's a screen asking for how to set the PATH variable. I chose the option not to update the windows PATH variable, which is the default. Then I installed git annex. Then launching the git-annex-autostart.vbs as well as the webapp one gets an object not found error (on line 2). launching git-annex from git bash with full path yielded an error about not finding git.
Then I proceeded and reinstalled git on top of itself and picked the option to only add git and bash to windows path and it worked.
"""]]

View file

@ -0,0 +1,22 @@
[[!comment format=mdwn
username="Hans_Ryding"
ip="81.229.194.7"
subject="Relying on path is not best practice in a Windows environment"
date="2014-08-25T16:16:33Z"
content="""
Unlike under POSIX environments
generally applications under windows don't add themselves to path,
or to a directory already in path.
Generally applications announce their location using the registry.
Under either HKEY_LOCAL_MACHINE\SOFTWARE,
or in case of software installed for one particular user only
under HKEY_CURRENT_USER\SOFTWARE.
Git however AFAIK does not.
Most likely the best thing to do is to prompt the user when installing git-annex
where git is, and store this variable.
Note that in both my installs I installed git-annex into the git directory,
and the git-annex webapp still couldn't find it.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBmv0HhwTFxkpxlf8ifTlMOHnIwHCHTYs"
nickname="y"
subject="path on windows"
date="2014-08-26T12:18:39Z"
content="""
To add to my comment I also installed git-annex in the same directory as the msys git distrib in both cases.
"""]]

View file

@ -0,0 +1,22 @@
### Please describe the problem.
After having added new content (SHA1E backend), when trying to commit, git commit fails with the following error:
[[!format sh """
(Recording state in git...)
error: invalid object 100644 5d471129a031f0f493de3736eaea6f2f4056aeee for '000/091/WORM-s1493-m1321288671--scrapbook%data%20111114173520%horiz-menu-tab-r_001.png.log'
fatal: git-write-tree: error building trees
git-annex: failed to read sha from git write-tree
"""]]
The commit subsequently fails and the index is left as is. When I did git-annex add, I got the same error, but the additions seem to have been staged, at least.
Whats curious about this is that I migrated all keys to SHA1E earlier and dropped all WORM keys. git annex info also says that all my keys are SHA1E.
Can this be related to your changes to the WORM backend? I upgraded to git-annex 5.20140818 today. Rolling back to 5.20140716 didnt allow me to commit, either, though.
Any way I could resolve this? I dont want to git reset for now, since this will leave the added objects in the annex store.
### What version of git-annex are you using? On what operating system?
git-annex 5.20140818
Linux 3.16.1

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="CandyAngel"
ip="81.111.193.130"
subject="comment 10"
date="2014-09-08T08:08:50Z"
content="""
Removing .git/annex/index is safe, it is a step in getting git-annex to [forget a commit entirely](http://git-annex.branchable.com/forum/How_to_get_git-annex_to_forget_a_commit__63__).
"""]]

View file

@ -0,0 +1,35 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 1"
date="2014-08-22T09:27:34Z"
content="""
git fsck only shows a few dangling blobs from a branch I did earlier and left behind, but otherwise reports no errors.
git annex fsck --fast ultimately fails with the original error message at some point:
[[!format sh \"\"\"
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# nx fsck --fast|egrep -v 'ok$'
[2014-08-22 11:14:43 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"ls-files\",\"--cached\",\"-z\",\"--\"]
[2014-08-22 11:14:43 CEST] chat: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
[2014-08-22 11:14:43 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"git-annex\"]
[2014-08-22 11:14:43 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-08-22 11:14:43 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"log\",\"refs/heads/git-annex..dda9b068ac5c075e79ab63a531770ad772ae8491\",\"-n1\",\"--pretty=%H\"]
[2014-08-22 11:14:43 CEST] chat: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"cat-file\",\"--batch\"]
[2014-08-22 11:25:24 CEST] chat: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
[2014-08-22 11:25:24 CEST] feed: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"update-index\",\"-z\",\"--index-info\"]
[2014-08-22 11:25:24 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2014-08-22 11:25:24 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"write-tree\"]
error: invalid object 100644 5d471129a031f0f493de3736eaea6f2f4056aeee for '000/091/WORM-s1493-m1321288671--scrapbook%data%20111114173520%horiz-menu-tab-r_001.png.log'
fatal: git-write-tree: error building trees
git-annex: failed to read sha from git write-tree
(Recording state in git...)
# End of transcript or log.
\"\"\"]]
"""]]

View file

@ -0,0 +1,30 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 2"
date="2014-08-22T09:38:03Z"
content="""
git commit with git-annex debug output enabled:
[[!format sh \"\"\"
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"diff\",\"--cached\",\"--name-only\",\"-z\",\"--diff-filter=ACMRT\",\"--\",\".\"]
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"diff\",\"--name-only\",\"--diff-filter=T\",\"-z\",\"--cached\",\"--\",\".\"]
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"symbolic-ref\",\"HEAD\"]
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"refs/heads/master\"]
[2014-08-22 11:36:46 CEST] chat: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
[2014-08-22 11:36:46 CEST] feed: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"update-index\",\"-z\",\"--index-info\"]
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
(Recording state in git...)
[2014-08-22 11:36:46 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"write-tree\"]
error: invalid object 100644 5d471129a031f0f493de3736eaea6f2f4056aeee for '000/091/WORM-s1493-m1321288671--scrapbook%data%20111114173520%horiz-menu-tab-r_001.png.log'
fatal: git-write-tree: error building trees
git-annex: failed to read sha from git write-tree
# End of transcript or log.
\"\"\"]]
"""]]

View file

@ -0,0 +1,27 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 3"
date="2014-08-22T09:58:05Z"
content="""
Doing a git annex fsck on a new clone of the repository succeded; the problem must somehow with the .git/annex/index then, I presume?
I did a git reset to restore to the sane state state before adding, but the problem is that I cannot unannex the files I added. :(
[[!format sh \"\"\"
nx unannex scrapbook/data/20140822101558/1.jpg
[2014-08-22 11:56:16 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"--head\"]
[2014-08-22 11:56:16 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"diff-index\",\"-z\",\"--raw\",\"--no-renames\",\"-l0\",\"--cached\",\"HEAD\"]
[2014-08-22 11:56:16 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"scrapbook/data/20140822101558/1.jpg\"]
[2014-08-22 11:56:16 CEST] call: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"commit\",\"-q\",\"--allow-empty\",\"--no-verify\",\"-m\",\"content removed from git annex\"]
[2014-08-22 11:56:16 CEST] chat: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
[2014-08-22 11:56:16 CEST] feed: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"update-index\",\"-z\",\"--index-info\"]
[2014-08-22 11:56:16 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
(Recording state in git...)
[2014-08-22 11:56:16 CEST] read: git [\"--git-dir=/home/seb/Webmirror/.git\",\"--work-tree=/home/seb/Webmirror\",\"write-tree\"]
error: invalid object 100644 5d471129a031f0f493de3736eaea6f2f4056aeee for '000/091/WORM-s1493-m1321288671--scrapbook%data%20111114173520%horiz-menu-tab-r_001.png.log'
fatal: git-write-tree: error building trees
git-annex: failed to read sha from git write-tree
\"\"\"]]
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 4"
date="2014-08-22T10:15:51Z"
content="""
The file referred to in the error message seems to be in good shape:
[[!format sh \"\"\"
git --no-pager show git-annex:000/091/WORM-s1493-m1321288671--scrapbook%data%20111114173520%horiz-menu-tab-r_001.png.log
1408605730.57892s 0 b25f42de-f4be-4d31-84d1-ab0b71dfec01
1408562938.526946s 0 e148ea91-0eb6-4f47-86e9-db2136a15279
\"\"\"]]
Strangely, the SHA1 of the blob is different from the one reported in the write-tree error.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 5"
date="2014-08-22T13:07:34Z"
content="""
I remembered I keep an hourly snapshot regimen and was able to get back the repository from before doing the «add» this morning. Both git fsck and git annex fsck return no errors, and yet, whenever anything is done to the git-annex branch (I tried add and forget), I get the above error.
"""]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 6"
date="2014-08-22T13:15:06Z"
content="""
I tried git annex repair on the repo (before doing any adds). It reports no fsck errors, but the repair then dies from a stack overflow.
[[!format sh \"\"\"
Running git fsck ...
No problems found.
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
\"\"\"]]
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 7"
date="2014-08-22T14:00:42Z"
content="""
I experimented on my snapshot a bit and found out something odd: When I reset the git-annex branch from dda9b06 to git-annex~1 (4246f73) my local file additions succeed, even though git-annex will fast-forward the branch to dda9b06 again before adding (when merging from origin/git-annex). dda9b06 is a large commit in which I dropped many unused WORM keys from another remote.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 8"
date="2014-08-22T18:57:37Z"
content="""
I just checked my other large git annex repo and noticed that here too I could no longer add files to the annex. The same observations as above apply. Here too on the tip of the git-anenx branch I had one huge commit in which I dropped the last of the unused WORM keys from another remote. Resetting the git-annex branch to git-annex~1 allowed me to make additions again, even though the reset tip was subsequently merged in again from the remote tracking branch.
"""]]

View file

@ -0,0 +1,22 @@
[[!comment format=mdwn
username="zardoz"
ip="78.48.163.229"
subject="comment 9"
date="2014-09-07T14:04:51Z"
content="""
Any ideas? I noticed one alternative way (cf. the reset workaround
above) to make «git annex add» work again is by deleting
.git/annex/index*. Is this safe?
In both repos, I had not even staged annex additions before the index
was corrupted; the corruption must somehow have been left-over from
earlier actions, altough all previous additions succeeded at the time,
before both repositories mysteriously stopped working (in the context
of backend-migration).
I still have the original snapshots around if youd like to debug
this. As noted, «git fsck» succeeds, and all the block-level checksums
check out, so the problem cant be on the block device or file-system
level.
"""]]

View file

@ -0,0 +1,31 @@
~~~~
$ git annex version
git-annex version: 5.20140818-g10bf03a
~~~~
When repository was initially created, it used "old" hashing from http://git-annex.branchable.com/internals/hashing/ . After some operations, annex was upgraded to "new" format. However, symlinks are still in "old" format and dangling. "git annex fsck", "git annex repair", "git annex pre-commit" - none helps.
~~~~
$ ls -l pics
lrwxrwxrwx 1 pfalcon pfalcon 199 Jan 22 2012 IMG_3776.JPG -> ../.git/annex/objects/KM/j6/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG
lrwxrwxrwx 1 pfalcon pfalcon 199 Jan 22 2012 renamed2.jpg -> ../.git/annex/objects/7F/z3/SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG/SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG
lrwxrwxrwx 1 pfalcon pfalcon 199 Jan 22 2012 renamed.jpg -> ../.git/annex/objects/W1/vK/SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG/SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG
$ find .git/annex/objects/
.git/annex/objects/
.git/annex/objects/219
.git/annex/objects/219/741
.git/annex/objects/219/741/SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG
.git/annex/objects/219/741/SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG/SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG
.git/annex/objects/7a6
.git/annex/objects/7a6/632
.git/annex/objects/7a6/632/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG
.git/annex/objects/7a6/632/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG
.git/annex/objects/df3
.git/annex/objects/df3/9a8
.git/annex/objects/df3/9a8/SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG
.git/annex/objects/df3/9a8/SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG/SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG
~~~~
> Unforunately, I cannot see through the attitude problem to a clear bug
> report. Lacking time/energy to try to coax one out. [[done]] --[[Joey]]

View file

@ -0,0 +1,35 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawld54zdyk6b0W4jXnssSO_j2Nn3W1uVsUE"
nickname="Paul"
subject="comment 1"
date="2014-08-24T20:27:08Z"
content="""
Aha, so local repo is created with old hash format. But when you add remote (special rsync remote in my case), and copy --to it, it uses new hashes:
~~~~
copy 20120122 Routing doorbell/IMG_3776.JPG (checking nas-rsync...) (to nas-rsync...)
sending incremental file list
7a6/
7a6/632/
7a6/632/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG/
7a6/632/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG/SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG
~~~~
This explains this nonsense:
~~~~
$ git annex unused --from=nas-rsync
unused nas-rsync (checking for unused data...) (checking master...)
Some annexed data on nas-rsync is not used by any files:
NUMBER KEY
1 SHA256E-s585398--005fe0534d6cc17a3536c1817b091d00249834c338f289ec6569e9f262889251.JPG
2 SHA256E-s688630--5bc2e8beb7a57f6fbcd7d9321cd5283f04448ea475099dac07ae38f002208040.JPG
3 SHA256E-s676047--3cd28892ee54aba13e074f230709b2c3b87915ff36efd9be3ddfc603e92ecdda.JPG
(To see where data was previously used, try: git log --stat -S'KEY')
To remove unwanted data: git-annex dropunused --from nas-rsync NUMBER
ok
~~~~
"""]]

View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawld54zdyk6b0W4jXnssSO_j2Nn3W1uVsUE"
nickname="Paul"
subject="comment 2"
date="2014-08-24T22:26:47Z"
content="""
Ok, I see, http://git-annex.branchable.com/internals/hashing/ says that old vs new hash mess is deliberate, to make user experience better. (One might ask why one hash was replaced with another equivalent, but nobody would. Oh wait, it's a filesystem case sensitivity issue of course. But it's too secret to be mentioned on \"hashing\" page.)
\"unused --from=\" issue comes and goes, don't see it now. That initial issue of completely broken symlinks happened after running testremote, then breaking it (because it should say it takes hour(s) to complete). So, many users probably won't be affected (nevermind that those who will, will essentially have data loss).
Last issue I faced that somehow my local working copy gets \"bare = true\" each time I sync against remote SSH repo (which is bare of course, as remote repo should be).
"""]]

View file

@ -0,0 +1,30 @@
### Please describe the problem.
When attempting to 'git annex get' a file that does not exist in the git repository, git-annex correctly reports "not found". But it still returns exit code 0, incorrectly indicating success. This is problematic for scripting.
### What steps will reproduce the problem?
See transcript
### What version of git-annex are you using? On what operating system?
git-annex 5.20140517.4 as supplied by 'git-annex' aptitude package on Ubuntu 12.04.4 LTS (32-bit)
### Please provide any additional information below.
[[!format sh """
henry@commsbox:~/work/tmp$ git init test
Initialized empty Git repository in /home/henry/work/tmp/test/.git/
henry@commsbox:~/work/tmp$ cd test
henry@commsbox:~/work/tmp/test$ git annex init
init ok
(Recording state in git...)
henry@commsbox:~/work/tmp/test$ git annex get nonexistent.file
git-annex: nonexistent.file not found
henry@commsbox:~/work/tmp/test$ echo $?
0
"""]]
> Ok, I can find no reason why it was implemented as a warning in
> 5f3661238de9f31e6fed0be74fca9d5f1659278c in the bug report associated
> with that commit. So, promoted to error. [[done]] --[[Joey]]

View file

@ -0,0 +1,30 @@
### Please describe the problem.
I have an rsync remote with the following option in config:
annex-rsync-transport = ssh -i /path/to/private/key
and git annex copy --to <remote> still asks for the remote's password. I've checked and I can ssh into the remote using that key with no problems, here's the --debug output:
copy SHA256E-s152396--56bcf5e3f72daa1a194b16e42330fe82806cc1dbc6f3bb52888ff5e5c57b8d08.log (gpg) (checking <remote>...) [2014-09-05 13:12:41 CEST] read: rsync ["<user>@<remote>:<remote path>/81c/b5e/'GPGHMACSHA1--7d9470e207a5669e2d9120538b68815274dbf16d/GPGHMACSHA1--7d9470e207a5669e2d9120538b68815274dbf16d'"]
<user>@<remote>'s password:
Once it has asked for the password twice, it starts using the rsync options correctly to copy and the password is no longer needed, until the next "checking remote…". Maybe the "checking remote…" part ignores the configuration?
### What steps will reproduce the problem?
Use an rsync remote with annex-rsync-transport config set.
### What version of git-annex are you using? On what operating system?
git annex version gives:
git-annex version: 5.20140814-g9b89b5c
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
which is the latest available linux-armel version.
> Also removing didn't use the configured transport. Both [[fixed|done]]
> --[[Joey]]

View file

@ -0,0 +1,45 @@
### Please describe the problem.
See the logs. git-annex-shell tries to use not existing runshell
### What steps will reproduce the problem?
I am on Debian testing and have, some month ago, tried the tarball distribution.
I have returned to deb packages later and deleted the tarball installation.
Seems that there some traces left.
I have tried to find the runshell configuration, but failed to do so.
I have destroyed the repo completely, has not helped.
### What version of git-annex are you using? On what operating system?
ii git-annex 5.20140831 amd64
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2014-09-07 17:15:04 CEST] main: starting assistant version 5.20140831
[2014-09-07 17:15:04 CEST] Cronner: Consistency check in progress
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
(scanning...) [2014-09-07 17:16:47 CEST] Watcher: Performing startup scan
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.2013121/
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
/home/<user>/.ssh/git-annex-shell: 4: exec: /home/<user>/git-annex.linux.5.20131213/runshell: not found
# End of transcript or log.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 1"
date="2014-09-11T17:55:28Z"
content="""
~/.ssh/git-annex-shell is a wrapper script that gets installed when you use the standalone build. You can delete it and your problem will be fixed.
It would probably be good if the standalone build came with an uninstallation script.
"""]]

33
doc/bugs/box.com.mdwn Normal file
View file

@ -0,0 +1,33 @@
### Please describe the problem.
I´m trying to use an box.com acount as special remote repository for transfering data amoung my clients. Adding the box.com is possible. On the box.com website i see that git-annex has create an folder, but there is no syncing.
When i try to enable the box.com Repository on an another client i get an Internal Error Server: bad creds.
### What version of git-annex are you using? On what operating system?
Git-Annex Version: git-annex version 5.20140830-g3c96b79
Mac OS X 10.9.4
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
03/Sep/2014:20:41:31 +0200 [Error#yesod-core] bad creds @(yesod-core-1.2.19:Yesod.Core.Class.Yesod ./Yesod/Core/Class/Yesod.hs:503:5)
[2014-09-03 20:41:47 CEST] main: Syncing with Box.com
bad creds
bad creds
bad creds
bad creds
# End of transcript or log.
"""]]

View file

@ -0,0 +1,77 @@
### Please describe the problem.
While the assistant is synchronizing (to local USB backup disks or remote repositories) I am continually prompted for GPG passphrase when it is most definitely already in the gpg-agent.
### What steps will reproduce the problem?
Set up some remote gcrypt repositores using an existing GPG key, add some files, use the system, you are prompted for the passphrase far more often than the timeout of the passphrase in the agent (every few minutes). The number of times you are prompted also seems to increase linearly with the number of repositories - I am guessing they all exhibit the same need for the passphrase and all request at once, resulting in a string of 10+ pinentry popups.
I am wondering if there is something specific in my gpg setup that git annex isn't expecting. Always encrypt to self? Signing subkeys? Either way, standard tools manage to call gpg to encrypt/decrypt using gpg-agent and not prompting for the passphrase, so git annex should be able to as well.
### What version of git-annex are you using? On what operating system?
Arch Linux, 5.20140831-g62e6ad8
### Please provide any additional information below.
I have read the various discussions about using -R or -r for the recipients, and I can see in the logs that -r is being used, but there is definitely something not working correctly. At the same time that git annex is making a gpg-agent request that results in a passphrase request, I can encrypt and decrypt whatever files I want manually with no prompting.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Development version -- Repository format MAY CHANGE
gcrypt: Decrypting manifest
gcrypt: Decrypting manifest
gpg: anonymous recipient; trying secret key 7426266D ...
gpg: anonymous recipient; trying secret key 7426266D ...
gpg: okay, we are the anonymous recipient.
gpg: Signature made Thu 11 Sep 2014 06:21:58 BST using RSA key ID AC305414
gpg: Good signature from "user <XXXXXX>" [ultimate]
gpg: aka "[jpeg image of size 2004]" [ultimate]
gpg: okay, we are the anonymous recipient.
gpg: Signature made Thu 11 Sep 2014 06:21:58 BST using RSA key ID AC305414
gpg: Good signature from "user <XXXXXX>" [ultimate]
gpg: aka "[jpeg image of size 2004]" [ultimate]
gcrypt: WARNING:
gcrypt: WARNING: Remote ID has changed!
gcrypt: WARNING: from :id:QydYJR8dPq7y7kMUQDG1
gcrypt: WARNING: to :id:gU3sc34/rhmta4xfSm3O
gcrypt: WARNING:
gcrypt: Encrypting to: -r 49AFD42BB9E8CD9D
gcrypt: Requesting manifest signature
gcrypt: Encrypting to: -r 49AFD42BB9E8CD9D
gcrypt: Requesting manifest signature
gcrypt: Decrypting manifest
gpg: anonymous recipient; trying secret key 7426266D ...
gpg: okay, we are the anonymous recipient.
gpg: Signature made Thu 11 Sep 2014 06:22:58 BST using RSA key ID AC305414
gpg: Good signature from "user <XXXXXXX>" [ultimate]
gpg: aka "[jpeg image of size 2004]" [ultimate]
gcrypt: Encrypting to: -r 49AFD42BB9E8CD9D
gcrypt: Requesting manifest signature
gpg: cancelled by user
gpg: skipped "49AFD42BB9E8CD9D": Operation cancelled
gpg: [stdin]: sign+encrypt failed: Operation cancelled
error: failed to push some refs to 'gcrypt::/autofs/ext/wdpassport0/annexes/user/docs'
[2014-09-11 06:36:18 BST] read: git ["--git-dir=/home/user/docs/.git","--work-tree=/home/user/docs","-c","core.bare=false","push","wdpassport0","master"]
...
# End of transcript or log.
"""]]
The section above I clicked cancel on the pinentry dialog as can be seen. The question is, why was it asking me anyway?
> gpg: anonymous recipient; trying secret key 7426266D ...
>
> That means that you have git-remote-gcrypt configured to use anonymous
> recipients. This causes gpg to try a bunch of gpg keys until it finds one
> that works, rather than immediately trying the right key.
>
> I modified git-remote-gcrypt in July so you can configure
> gcrypt.publish-participants to avoid this problem.
>
> There may also be a local confguration problem; I don't know. In any
> case, it's not git-annex responsible, but git-remote-gcrypt. [[done]] --[[Joey]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="131.252.200.111"
subject="comment 3"
date="2014-08-31T22:29:44Z"
content="""
Occurs to me that your repo may not have a pre-commit hook; if not then `git commit -a` would not behave as I described..
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://andrew.aylett.co.uk/"
nickname="andrew"
subject="comment 4"
date="2014-09-07T18:41:28Z"
content="""
I, too, have seen this issue -- took me a while to recover from it. I do (now, at least) have a pre-commit hook that calls git annex pre-commit; I didn't set that up myself.
"""]]

View file

@ -0,0 +1,32 @@
[[!comment format=mdwn
username="Francois"
ip="2001:788:5:1:29c2:de49:9811:51c8"
subject="comment 3"
date="2014-08-15T20:45:38Z"
content="""
I've been experiencing the exact same problem and searching for **recovery from race** lead me to this bug report. Thanks for reporting it!
For a few months, a repo storing ~19'000 files (mostly immutable pictures) started to launch memory hungry \"git log\" processes. For example:
4797 francois 20 0 8118296 7.719g 2032 D 22.3 50.2 0:11.61 git
4797 pts/1 D+ 0:12 git --git-dir=~/Pictures/.git --work-tree=~/Pictures -c core.bare=false log refs/heads/git-annex..52e44b967ad5d316d832562be02c5555c1f6d2a4 --oneline -n1
Thanks to the hints found in this report, I was able to find many huge commit messages such as this one:
$ git show 6357b208
commit 6357b2081e7c85dfe1ccc10824b75f3e212e6386
Author: Francois Deppierraz <francois@ctrlaltdel.ch>
Date: Sat Jun 14 10:38:46 2014 +0200
update (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) [...]
$ git show 6357b208 | wc
5 444026 3108236
There were probably many new files added on Jun 14th and looking for a way to increase to sync speed, especially to a S3-like remote, I found the solution on this wiki for [multiple concurrent transfers](https://git-annex.branchable.com/forum/Feature_request:_Multiple_concurrent_transfers/).
This looks like a likely culprit for generating race conditions. What do you think?
git-annex version: 5.20140412ubuntu1
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Francois"
ip="2001:788:5:1:29c2:de49:9811:51c8"
subject="comment 4"
date="2014-08-15T20:51:12Z"
content="""
Bonus question: is there any way to bring this repo back into a working state (for instance with git-filter-branch?) ?
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="jg123h12jh3y12g3y"
ip="86.62.100.131"
subject="Log with --debug"
date="2014-08-20T05:49:02Z"
content="""
https://mega.co.nz/#!HYZmwSIb!gCd9jvVIyYye_bpsUq_vuEed4g7NTlEl2xDRheE1Lx4
This is the log of git annex repair --debug.
"""]]

View file

@ -0,0 +1,29 @@
### Please describe the problem.
When i try to sync to my server (path in .git/config is "[fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e]:/home/thomas/git/musik") the url gets messed up by annex and i get the error "git-annex: bad url ssh://[fcb8/~/b10:1cb8:c94:58d0:2522:89f9:c89e]:/home/thomas/git/musik".
### What steps will reproduce the problem?
1. init git & annex
2. add files
3. add a IPv6 address remote
4. push git branches
5. git annex sync
### What version of git-annex are you using? On what operating system?
```
git-annex version: 5.20140412ubuntu1
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
```
> [[Fixed|done]] by adding support for ipv6 addresses when git-annex
> converts a git remote loction into an url. BTW, the
> simple workaround is to give it a valid url from the beginning
> `ssh://[fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e]/home/thomas/git/musik"`
>
> As to any problems using an ipv6 remote once it's set up, I've used them
> with no problems.
> --[[Joey]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
nickname="Justin"
subject="comment 1"
date="2014-08-28T20:46:29Z"
content="""
Try using ~/.ssh/config as a workaround
Host myserver
Hostname fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e
then just tell git-annex to use myserver
"""]]

View file

@ -0,0 +1,29 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlog_5wIICaMcrKTexlFNA6IO6UTp323aE"
nickname="Torkaly"
subject="comment 2"
date="2014-09-01T11:01:30Z"
content="""
thank you. I workaround this by using the DNS hostname instead the IPv6 address directly. But this is not a nice solution, like any workaround. But now i have problems with `git annex get` over IPv6-only:
```
get ***.mp3 (not available)
Try making some of these repositories available:
5636aefa-c509-4ea0-bebe-f5b96d8eb15a -- hserver
failed
<snip>
```
but i can ping it:
```
thomas@alus:~/Musik$ ping6 hserver.h.b128.net
PING hserver.h.b128.net(fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e) 56 data bytes
64 bytes from fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e: icmp_seq=1 ttl=42 time=453 ms
64 bytes from fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e: icmp_seq=2 ttl=42 time=441 ms
64 bytes from fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e: icmp_seq=3 ttl=42 time=425 ms
64 bytes from fcb8:b10:1cb8:c94:58d0:2522:89f9:c89e: icmp_seq=5 ttl=42 time=413 ms
```
(the high pings are caused by a download from an other source. Also i have no problems with rsync over IPv6)
PS: the markdown for code blocks is not working too :)
"""]]

View file

@ -0,0 +1,27 @@
### Please describe the problem.
The tahoe-lafs remote has no built-in way to perform the repair operation.
This results to data loss if expiration is enabled on the Tahoe grid.
For the current tahoe-lafs release (1.10.0), the only way storage space is freed
is via garbage collection. Garbage collection removes shares whose lease has expired.
Data loss will occur if leases are not periodically renewed via
"tahoe repair --add-lease WRITECAP".
The current implementation of the Tahoe remote in git-annex does not offer a way to
run lease renewal, and cannot be used on grids where GC is enabled. (GC is not enabled
in the default configuration, but on private grids it is a sensible option.)
One way renewal could be made easier to do is to add the uploaded files to a directory
in Tahoe, so that the leases could be easily updated if the directory writecap is known,
without needing to go through the full list of writecaps for each file stored.
### What steps will reproduce the problem?
1. Use tahoe remote on a tahoe grid where GC is enabled.
2. After GC expiration period, data loss ensues.
### What version of git-annex are you using? On what operating system?
Seems to affect current git master (as of 2014-08-24).

View file

@ -0,0 +1,42 @@
### Please describe the problem.
error message:
copy somefile.jpg (checking myserver...) (to myserver...)
git-annex: runInteractiveProcess: pipe: Too many open files
rsync failed -- run git annex again to resume file transfer
failed
### What steps will reproduce the problem?
1. Start a `git annex copy` with lots of files in the queue.
2. Start a second `git annex copy` on the same set of files.
The intention is to minimize the amount of silent time on the wire due to administrative work between actual file transfers. These two processes will trip over each other and see that transfer X is already going, and skip to the next file Y, so in the end they upload about half of the files each.
3. Expect all files to be uploaded. Actually observe the above error message for at least one of the processes.
### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140420-ga25b8bb
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
Darwin mymacbook 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
### Please provide any additional information below.
[[!format sh """
lsof -p <my annex process>
... some .app/** files, tty etc ...
... some unnamed pipes ...
.../.git/annex/ssh/myserver.lock
.../.git/annex/transfer/upload/b4d67c4f-8cca-423c-9363-f3063b7fe3e4/lck.SHA256E-s10448418--4f61fab4... ~200 different files.
"""]]
> Thanks for a very clear bug report! Was easy from that to find
> where the lock file was not being closed in this situation.
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://id.clacke.se/"
nickname="Claes"
subject="5.20140830"
date="2014-09-07T19:24:49Z"
content="""
Will verify if this is still valid for 5.20140830.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://id.clacke.se/"
nickname="Claes"
subject="yep"
date="2014-09-07T19:42:04Z"
content="""
Still valid for `git-annex version: 5.20140830-g3c96b79`
"""]]

View file

@ -0,0 +1,152 @@
### Please describe the problem.
I can change the settings in one repo and sync it everywhere. Just to be surprised that one repo starts syncing to the transfer, every time it turns out that this repo lost its vicfg settings. Especially the Repository preferred contents are all back on standard. It was even once that it had the current settings and after the change and sync it goes back to some older state instead of the new one.
### What steps will reproduce the problem?
Well that is very hard. I have 8 repos and it happens randomly to some of them. I recreated all of them recently because I thought they are corrupt, that didn't help, just took me one week of time. It is also very hard to find a way to reproduce this because every vicfg causes a merge which takes minutes to hours.
### What version of git-annex are you using? On what operating system?
Linux: git-annex version: 5.20140412ubuntu1
Mac OS: git-annex version: 5.20140717
### Please provide any additional information below.
Layout:
transfer on rsync.net, conntented to that:
- Two OS X Clients
- Two Linux Archives
My settings:
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# git-annex configuration
#
# Changes saved to this file will be recorded in the git-annex branch.
#
# Lines in this file have the format:
# setting field = value
# Repository trust configuration
# (Valid trust levels: trusted semitrusted untrusted dead)
# (for Music bei Pirmin)
trust 0734498b-817c-419f-a0c0-660854dc7cbe = trusted
# (for Music bei Jean (Willikins) [willikins])
trust 20e9d2e5-9563-4507-82d5-bf8e23be29a5 = trusted
# (for Music bei Jean (Willikins Clone))
trust 6e3431e9-8ec2-404a-9c35-b967db63147d = trusted
# (for Music bei Jean (Watson))
trust a6febfa0-9fe5-4a65-95bb-dc255d87c2e2 = trusted
# (for )
trust dafe9a64-2480-40e2-9688-9f783577ef72 = dead
# (for web)
#trust 00000000-0000-0000-0000-000000000001 = semitrusted
# (for music transfer via rsync.net [music_rsync])
#trust 83c42610-42ad-459d-92a4-1aca2dfb97e1 = semitrusted
# Repository groups
# (Standard groups: client transfer backup incrementalbackup smallarchive archive source manual public unwanted)
# (Separate group names with spaces)
# (for Music bei Jean (Willikins) [willikins])
group 20e9d2e5-9563-4507-82d5-bf8e23be29a5 = archive
# (for Music bei Jean (Willikins Clone))
group 6e3431e9-8ec2-404a-9c35-b967db63147d = archive
# (for )
group 26d38f31-cb6c-412c-84ef-597d7959a680 = backup
# (for Music bei Pirmin)
group 0734498b-817c-419f-a0c0-660854dc7cbe = client
# (for Music bei Jean (Watson))
group a6febfa0-9fe5-4a65-95bb-dc255d87c2e2 = client
# (for music transfer via rsync.net [music_rsync])
group 83c42610-42ad-459d-92a4-1aca2dfb97e1 = transfer
# (for )
group dafe9a64-2480-40e2-9688-9f783577ef72 = unwanted
# (for web)
#group 00000000-0000-0000-0000-000000000001 =
# Repository preferred contents
# (Set to "standard" to use a repository's group's preferred contents)
# (for Music bei Jean (Willikins) [willikins])
wanted 20e9d2e5-9563-4507-82d5-bf8e23be29a5 = (not (copies=archive:2 or copies=smallarchive:2)) or approxlackingcopies=2
# (for Music bei Jean (Willikins Clone))
wanted 6e3431e9-8ec2-404a-9c35-b967db63147d = (not (copies=archive:2 or copies=smallarchive:2)) or approxlackingcopies=2
# (for music transfer via rsync.net [music_rsync])
wanted 83c42610-42ad-459d-92a4-1aca2dfb97e1 = not (inallgroup=client and copies=archive:2 and copies=client:2) and ((((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and not unused) or approxlackingcopies=1)
# (for Music bei Pirmin)
wanted 0734498b-817c-419f-a0c0-660854dc7cbe = standard
# (for )
wanted 26d38f31-cb6c-412c-84ef-597d7959a680 = standard
# (for )
wanted dafe9a64-2480-40e2-9688-9f783577ef72 = standard
# (for web)
#wanted 00000000-0000-0000-0000-000000000001 =
# (for Music bei Jean (Watson))
wanted a6febfa0-9fe5-4a65-95bb-dc255d87c2e2 = standard
# Group preferred contents
# (Used by repositories with "groupwanted" in their preferred contents)
#groupwanted archive =
#groupwanted backup =
#groupwanted client =
#groupwanted incrementalbackup =
#groupwanted manual =
#groupwanted public =
#groupwanted smallarchive =
#groupwanted source =
#groupwanted transfer =
#groupwanted unwanted =
# Standard preferred contents
# (Used by wanted or groupwanted expressions containing "standard")
# (For reference only; built-in and cannot be changed!)
# standard client = (((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and not unused) or approxlackingcopies=1
# standard transfer = (not (inallgroup=client and copies=client:2) and ((((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and not unused) or approxlackingcopies=1)) or approxlackingcopies=1
# standard backup = include=* or unused
# standard incrementalbackup = ((include=* or unused) and (not copies=incrementalbackup:1)) or approxlackingcopies=1
# standard smallarchive = ((include=*/archive/* or include=archive/*) and ((not (copies=archive:1 or copies=smallarchive:1)) or approxlackingcopies=1)) or approxlackingcopies=1
# standard archive = (not (copies=archive:1 or copies=smallarchive:1)) or approxlackingcopies=1
# standard source = not (copies=1)
# standard manual = present and ((((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and not unused) or approxlackingcopies=1)
# standard public = inpreferreddir
# standard unwanted = exclude=*
# Repository required contents
# (for web)
#required 00000000-0000-0000-0000-000000000001 =
# (for Music bei Pirmin)
#required 0734498b-817c-419f-a0c0-660854dc7cbe =
# (for Music bei Jean (Willikins) [willikins])
#required 20e9d2e5-9563-4507-82d5-bf8e23be29a5 =
# (for Music bei Jean (Willikins Clone))
#required 6e3431e9-8ec2-404a-9c35-b967db63147d =
# (for music transfer via rsync.net [music_rsync])
#required 83c42610-42ad-459d-92a4-1aca2dfb97e1 =
# (for Music bei Jean (Watson))
#required a6febfa0-9fe5-4a65-95bb-dc255d87c2e2 =
# Scheduled activities
# (Separate multiple activities with "; ")
# (for web)
#schedule 00000000-0000-0000-0000-000000000001 =
# (for Music bei Pirmin)
#schedule 0734498b-817c-419f-a0c0-660854dc7cbe =
# (for Music bei Jean (Willikins) [willikins])
#schedule 20e9d2e5-9563-4507-82d5-bf8e23be29a5 =
# (for Music bei Jean (Willikins Clone))
#schedule 6e3431e9-8ec2-404a-9c35-b967db63147d =
# (for music transfer via rsync.net [music_rsync])
#schedule 83c42610-42ad-459d-92a4-1aca2dfb97e1 =
# (for Music bei Jean (Watson))
#schedule a6febfa0-9fe5-4a65-95bb-dc255d87c2e2 =
# End of transcript or log.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 1"
date="2014-09-11T18:11:43Z"
content="""
vicfg makes changes to files in the git-annex branch, so you can use regular git commands to examine those changes, and look through the history to find any changes that are contrary to the settings you want, and see when and where they were committed. That should provide a strong pointer to what is causing this to happen.
"""]]

View file

@ -0,0 +1,21 @@
### Please describe the problem.
I am unable to run the webapp on redhat6.5
### What steps will reproduce the problem?
yum install git-annex
### What version of git-annex are you using? On what operating system?
I am using git-annex version 3.20120523 and on redhat 6.5
### Please provide any additional information below.
I am seeing the following error when running git annex webapp:
git-annex: unknown command webapp
> git-annex can be built without the webapp, and this is
> often done if a distribution does not have the full haskell stack
> packaged yet. The solution is to contact the distribution and let them
> know they need to improve it, and/or use the standalone build from this
> website.
>
> So, please file a bug on redhat. [[done]] --[[Joey]]

View file

@ -0,0 +1,76 @@
### Please describe the problem.
Cant add Server repo
### What steps will reproduce the problem?
install linux prebuild torball on server (readynas pro 600)
un-tar and run with ./git-annex
(git-annex test pass without error)
install windows client on win7 64 bit and start webapp
try add server repo as second repo will led to an internal server error about gpg
followed the assistend video
### What version of git-annex are you using? On what operating system?
Readynas (Linux):
annex@readynas-pro:~/git-annex.linux$ ./git-annex version
git-annex version: 5.20140831-g62e6ad8
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
Windows:
Version: 5.20140914-gb169612
Build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
Git on Windows:
C:\Users\Xaver>git --version
git version 1.9.4.msysgit.1
GPG on Windows:
C:\Users\Xaver>gpg --version
gpg (GnuPG) 2.0.26 (Gpg4win 2.2.2)
libgcrypt 1.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Home: C:/Users/Xaver/AppData/Roaming/gnupg
Unterstützte Verfahren:
Öff. Schlüssel: RSA, ELG, DSA
Verschlü.: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Komprimierung: nicht komprimiert, ZIP, ZLIB, BZIP2
### Please provide any additional information below.
Internal Server Error
user error (gpg ["--quiet","--trust-model","always","--with-colons","--list-secret-keys","--fixed-list-mode"] exited 2)
gpg: fatal: can't create directory `/home/Xaver/.gnupg': No such file or directory
makes no sense on windows machine
must be like "C:\Users\Xaver\AppData\Roaming\gnupg" or %Username%\AppData\Roaming\gnupg I guess.
gpg itself works fine I use it with thunderbird
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/documentation/faqs.html for more information
gpg: fatal: can't create directory `/home/Xaver/.gnupg': No such file or directory
secmem usage: 0/0 bytes in 0/0 blocks of pool 0/65536
14/Sep/2014:23:17:08 +0200 [Error#yesod-core] user error (gpg ["--quiet","--trust-model","always","--with-colons","--list-secret-keys","--fixed-list-mode"] exited 2) @(yesod-core-1.2.19:Yesod.Core.Class.Yesod .\Yesod\Core\Class\Yesod.hs:503:5)
# End of transcript or log.
"""]]
> Thanks for reporting this problem. I've fixed it to not crash when gpg
> fails to list secret keys.
>
> That doesn't fix the problem that the cygnus build of gpg does not find
> the user's home directory properly. But that's only needed for the
> encrypted repository (gcrypt) support, which is listed in
> [[windows_support]] as not yet available for Windows.
>
> So, not leaving this bug report open. [[done]] --[[Joey]]

View file

@ -1,6 +1,6 @@
### Please describe the problem.
`git annex whereis` says that there are no copies of any of the files that have been added in repositories running in direct mode.
`git annex whereis` says that there are no copies of any of the files that have been added in repositories running in direct mode when `annex.alwayscommit` is set to `false`.
In other words, if I add a file from PC1 in direct mode, `whereis` in PC2 will fail. Instead, if I add the same file from PC1 in indirect mode, `whereis` in PC2 will work correctly and will report that the file is present in PC1.
@ -20,7 +20,10 @@ The following script (available at <https://gist.github.com/gioele/dde462df89edf
set -e ; set -u
export LC_ALL=C
# alwayscommit must be set globally to affects whereis and sync
git config --global annex.alwayscommit false
direct=true # set to false to make the problem disappear
h=${h:-localhost}
@ -84,3 +87,5 @@ echo "Why isn't location info available even after sync? (press Enter)"
### What version of git-annex are you using? On what operating system?
git-annex version: 5.20140716-g8c14ba8
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://svario.it/gioele"
nickname="gioele"
subject="comment 5"
date="2014-08-31T10:15:30Z"
content="""
I have found out that there is a connection between this problem and the _global_ configuration of `annex.alwayscommit`. This problem will appear only if `annex.alwayscommit` is globally set to `false`. What is very strange is that setting `annex.alwayscommit` locally does not make this bug appear; only a globally set `annex.alwayscommit` will trigger this problem.
I fixed the test script to set `annex.alwayscommit` globally.
Now I see why I could reproduce this bug on different machines but Joey could not: all my machines have the same `~/.gitconfig`.
"""]]

View file

@ -0,0 +1,49 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.132"
subject="comment 6"
date="2014-09-11T18:34:02Z"
content="""
It looks like there might be some minor inconsistency in when git-annex syncs when in indirect mode vs direct mode. This results in the location tracking information not being committed until after the git-annex sync in the pc1/Docs repository has pushed the git-annex branch to origin. Since that is the only
time that pc1/Docs syncs with origin, the location tracking info never reaches origin, and the rest of the behavior follows.
Here is the direct mode sync, showing the git-annex branch commit occurring after the push. Specifically, when the sync merges the git-annex branch, it also commits any deferred changes at that point:
<pre>
commit ok
pull origin
ok
push origin
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 574 bytes | 0 bytes/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To localhost:/tmp/annex/Docs.git
* [new branch] git-annex -> synced/git-annex
* [new branch] annex/direct/master -> synced/master
ok
(Recording state in git...)
</pre>
And here is the indirect mode sync, showing a \"commit\" which includes committing deferred changes to the git-annex branch:
<pre>
commit (Recording state in git...)
ok
pull origin
ok
push origin
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (15/15), 1.32 KiB | 0 bytes/s, done.
Total 15 (delta 0), reused 0 (delta 0)
To localhost:/tmp/annex/Docs.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
</pre>
It seems that [[!commit 2cfda59174b9cbc02e87c069982096d44601cd40]] and some subsequent changes accidentially removed the Annex.Branch.commit from the direct mode code path within the first part of `sync`. So, easily fixed.
"""]]

View file

@ -40,5 +40,8 @@
<h2>OSX Mavericks</h2>
<iframe width=1024 scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuild/x86_64-apple-mavericks/">
</iframe>
<h2>Windows</h2>
<h2><a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">Windows</a></h2>
<a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">here</a>
<h2><a href="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid">Debian</a></h2>
<iframe width=1024 scrolling=no height=500px frameborder=0 marginheight=0 marginwidth=0 src="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid">
</iframe>

View file

@ -24,7 +24,8 @@ you'll make Joey more productive!
## code contributions
[[download]] the source code and send patches!
[[download]] the source code, [[build|install/fromsource]] it
and send patches!
If you know Haskell, git-annex has lots of Haskell code that
could be improved. See the [[coding_style]] and have at it.

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.22"
subject="comment 5"
date="2014-09-16T18:36:18Z"
content="""
Note that --listen=address:port had to be removed.
OTOH, the webapp can be run with a https certificate now, which makes remote access much more secure.
The webapp will use HTTPS if it finds
a .git/annex/privkey.pem and .git/annex/certificate.pem. Here's
one way to generate those files, using a self-signed certificate:
openssl genrsa -out .git/annex/privkey.pem 4096
openssl req -new -x509 -key .git/annex/privkey.pem > .git/annex/certificate.pem
"""]]

View file

@ -4,4 +4,4 @@ Same as the desktop webapp, users will be able to enter a directory they
want the first time they run it, but to save typing on android, anything
that gets enough votes will be included in a list of choices as well.
[[!poll open=yes expandable=yes 68 "/sdcard/annex" 6 "Whole /sdcard" 7 "DCIM directory (photos and videos only)" 2 "Same as for regular git-annex. ~/annex/"]]
[[!poll open=yes expandable=yes 69 "/sdcard/annex" 6 "Whole /sdcard" 7 "DCIM directory (photos and videos only)" 2 "Same as for regular git-annex. ~/annex/"]]

View file

@ -53,7 +53,7 @@ once, and can be left alone when refining a view.
## automatically added metadata
When annex.genmetadata is set, git annex add automatically attaches
When `annex.genmetadata` is set, git annex add automatically attaches
some metadata to a file. Currently year and month fields, from its mtime.
There's also a post-commit-annex hook script.

View file

@ -1,6 +1,6 @@
## roadmap
Now in the
Just finished the
[sustaining git-annex development](https://campaign.joeyh.name/) year
(starting September 2013).
@ -15,7 +15,7 @@ Now in the
* Month 9 Brazil!, [[!traillink assistant/sshpassword]]
* Month 10 polish [[assistant/Windows]] port
* Month 11 [[!traillink assistant/chunks]]
* **Month 12** user-driven features and polishing
* Month 12 user-driven features and polishing
Deferred until later:

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="209.250.56.7"
subject="comment 2"
date="2014-08-15T19:45:28Z"
content="""
That was a typo. Of course, spaces are no problem. Newlines in filenames, on the other hand..
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="EskildHustvedt"
ip="80.202.103.55"
subject="comment 3"
date="2014-08-16T15:22:35Z"
content="""
Ah, well then, that sounds a lot more reasonable. Though legal, I have yet to hear of a sane reason for using newlines in filenames.
"""]]

View file

@ -0,0 +1,10 @@
Over the past couple days, got the arm autobuilder working again. It had
been down since June with several problems. cabal install tended to crash;
apparenty this has something to do with threading in user-mode qemu,
because -j1 avoids that. And strange invalid character problems were fixed
by downgrading file-embed. Also, with Yury's help I got the Windows
autobuilder upgraded to the new Haskell Platform and working again.
Today a last few finishing touches, including getting rid of the last
dependency on the old haskell HTTP library, since http-conduit is being
used now. Ready for the release!

View file

@ -0,0 +1,24 @@
Plan is to be on vacation and/or low activity this week before DebConf.
However, today I got involved in fixing a bug that caused the assistant to
keep files open after syncing with repositories on removable media.
Part of that bug involved lock files not being opend close-on-exec, and
while fixing that I noticed again that the locking code was scattered all
around and rather repetitive. That led to a lot of refactoring, which is
always fun when it involves scary locking code. Thanks goodness for
referential transparency.
Now there's a Utility.LockFile that works on both POSIX and Windows.
Howver, that module actually exports very different functions for the two.
While it might be tempting to try to do a portability layer, the
two locking models are really very different, and there are lots of gotchas
such a portability layer would face. The only API that's completely the
same between the two is dropLock.
This refactoring process and the cleaner, more expressive
code it led to helped me spot a couple of bugs involving locking. See
[[!commit e386e26ef207db742da6d406183ab851571047ff]]
and [[!commit 0a4d301051e4933661b7b0a0791afa95bfe9a1d3]]
Neither bug has ever seemed to cause
a problem, but it's nice to be able to spot and fix such bugs before they
do.

View file

@ -0,0 +1,109 @@
Yesterday and today were the first good solid days working on git-annex in a
while. There's a big backlog, currently of 133 messages, so I have been
concentrating on bug reports first. Happily, not many new bugs have been
reported lately, and I've made good progress on them, fixing 5 bugs today,
including a file descriptor leak.
## catching up
In this end of summer rush, I've been too busy to blog for the past 20 days,
but not entirely too busy to work on git-annex. Two releases have been made
in that time, and a fair amount of improvements worked on.
Including a new feature: When a local git repository is cloned with `git
clone --shared`, git-annex detects this and defaults to a special mode
where file contents get hard linked into the clone. It also makes the cloned
repository be untrusted, to avoid confusing numcopies counting with the
hard links. This can be useful for temporary working repositories without
the overhead of lots of copies of files.
## looking back
I want to look back further, over the crowdfunded year of work covered
by this devblog. There were a lot of things I wanted to accomplish this
past year, and I managed to get to most of them. As well as a few surprises.
* Windows support improved more than I guessed in my wildest dreams.
git-annex went from working not too well on the command line to
being pretty solid there, as well as having a working
and almost polished webapp on Windows.
There are still warts -- it's Windows after all!
* Android didn't get many improvements. Most of the time I had budgeted to
Android porting ended up being used on Windows porting instead. I did,
however, get the Android build environment cleaned up a lot from the initial
hacked together one, and generally kept it building and working on Android.
* The [direct mode guard](http://git-annex.branchable.com/devblog/day_48__direct_mode_guard_design/)
was not planned, but the need for it became clear, and
it's dramatically reduced the amount of command-line foot-shooting
that goes on in direct mode.
* Repository repair was planned, and I've very proud of [git-repair](http://git-repair.branchable.com/).
Also pleased with the webapp's UI for scheduling repository consistency
checks.
Always room for improvement in this kind of thing, but this brings a new
capability to both git and git-annex.
* The [[external_special_remote_interface|special_remotes/external]] came
together beautifully. External special remotes are now just as well
supported as built-in ones, except the webapp cannot be used to configure
them.
* Using git-remote-gcrypt for fully encrypted git repositories, including
support in the webapp for setting them (and gpg keys if necessary),
happened. Still needs testing/more use/improvements. Avoided doing
much in the area of gpg key management, which is probably good to avoid when
possible, but is probably needed to make this a really suitable option for
end users.
* Telehash is still being built, and it's not clear if they've gotten it
to work at all yet. The v2 telehash has recently been superseded by a
a new v3. So I am not pleased that I didn't get git-annex working with
telehash, but it was outside my control. This is a problem that needs to get
solved outside git-annex first, either by telehash or something else.
The plan is to keep an eye on everything in this space, including for example,
Maidsafe.
* In the meantime, the new notifychanges support in git-annex-shell
makes XMPP/telehash/whatever unnecessary in a lot of configurations.
git-annex's remotedaemon architecture supports that and is designed
to support other notification methods later. And the webapp has a lot of
improvements in the area of setting up ssh remotes, so fewer users will
be stuck with XMPP.
* I didn't quite get to [[design/assistant/deltas]], but the final month
of work on chunking provides a lot of new features and hopefully a
foundation that will get to deltas eventually. There is a new haskell
library that's being developed with the goal of being used for git-annex
deltas.
* I hadn't planned to make git-annex be able to upgrade itself, when installed
from this website. But there was a need for that, and so it happened.
Even got a gpg key trust path for the distribution of git-annex.
* Metadata driven views was an entirely unplanned feature. The current
prototype is very exciting, it opens up entire new use cases.
I had to hold myself back to not work on it too much,
especially as it shaded into adding a caching database to git-annex.
Had too much other stuff planned to do all I wanted.
Clearly this is an area I want to spend more time on!
Those are most of the big features and changes, but probably half
of my work on git-annex this past year was in smaller things, and general
maintenance. Lots of others have contributed, some with
code (like the large effort to switch to bootstrap3),
and others with documentation, bug reports, etc.
Perhaps it's best to turn to `git diff --stat` to sum up the activity
and see just how much both the crowdfunding campaign and
the previous kickstarter have pushed git-annex into high gear:
campaign: 5410 files changed, 124159 insertions(+), 79395 deletions(-)
kickstarter: 4411 files changed, 123262 insertions(+), 13935 deletions(-)
year before: 1281 files changed, 7263 insertions(+), 55831 deletions(-)
What's next? The hope is, no more crowdfunded campaigns where I have
to promise the moon anytime soon. Instead, the goal is to move to a more
mature and sustainable funding model, and continue to grow the git-annex
community, and the spaces where it's useful.

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkRGMQkg9ck_pr47JXZV_C2DJQXrO8LgpI"
nickname="Michael"
subject="Hard linking on local clone"
date="2014-09-13T06:28:01Z"
content="""
Thanks for this feature. It will save a lot of space when working on one-off projects with big scientific datasets.
Unfortunately, there is probably no easy solution to achieve similar savings across file systems. On our shared cluster individual labs have their data in separate ZFS volumes (to ease individual backup handling), but data is often shared (i.e. copied) across volumes when cloning an annex. We need expensive de-duplication on the backup-server to, at least, prevent this kind of waste to hit the backups -- but the master file server still suffers (de-duplication ratio sometimes approaching a factor of 2.0).
"""]]

View file

@ -0,0 +1,19 @@
[[!comment format=mdwn
username="https://id.koumbit.net/anarcat"
ip="72.0.72.144"
subject="comment 2"
date="2014-09-14T17:38:34Z"
content="""
thanks so much for your work on git-annex, joeyh. it's hard to imagine that just 4 years ago, we didn't have anything even close to this tool and how far it went since then.
~~~~
anarcat@marcos:git-annex$ pepper age
Loading cache index... done
git-annex is 3 years old
git-annex's birthday is in about 1 month (October 19th)
~~~~
birthday is coming soon! :)
it's also quite impressive how much work can be done in a single year with some (fairly minimal) funding to dedicate a full dev on a project. very inspiring - keep up the good work! -- [[anarcat]]
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="anarcat"
ip="70.83.139.100"
subject="camlistore"
date="2014-09-17T20:18:56Z"
content="""
have you looked at [camlistore](http://camlistore.org/) at all? it's a fairly new project, but it seems like it could be an interesting backend, or at least inspiration for git-annex's design. --[[anarcat]]
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmH7o6q2l99M-PQolOfbR3_i5B_jtTIcAE"
nickname="Giovanni"
subject="Camlistore"
date="2014-09-17T20:36:43Z"
content="""
anarcat, have you used it? I tried, but it is buggy. They seem to be at [\"the Archivist\"](http://git-annex.branchable.com/) group of people, but if you don't have a hard drive to store the things, everything breaks up. I paid a lot of money to Amazon because I believed I could use Camlistore to organize data stored at S3, but apparently S3 is \"just for backup\", and if it is your only storage, then Camlistore will keep fetching data over and over \"to index it\" and in the end you pay.
Yes, it keeps working, so you need some server online at all times, with Camlistore running.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="anarcat"
ip="70.83.139.100"
subject="comment 5"
date="2014-09-17T20:53:01Z"
content="""
i haven't tried it at all - only looked at [this one demo video](https://www.youtube.com/watch?v=yxSzQIwXM1k) that reminded me of git-annex a lot...
"""]]

View file

@ -0,0 +1,14 @@
Made a release yesterday, which was all bugfixes.
Today, a few more bug fixes. Looked into making the webapp
create non-bare repositories on removable drives, but before I got too far
into the code, I noticed [there's a big problem with that idea](http://git-annex.branchable.com/forum/usability:_creating_an_archive_on_a_new_external_drive/).
Rest of day was spent getting caught up on forum posts etc. I'm happy to
read lots of good answers that have been posted while I've been away.
Here's an excellent example: <http://git-annex.branchable.com/install/fromsource/#comment-5f8ceb060643ae71cd2adc72f0fca3f0>
That led to rewriting the docs for building git-annex from source.
New page: [[install/fromsource]].
Backlog is now down to 117.

View file

@ -26,13 +26,10 @@ The git repository has some branches:
(merge it into master if you need it)
* `no-bloom` avoids using bloom filters. (merge it into master if you need it)
* `no-s3` avoids using the S3 library (merge it into master if you need it)
* `debian-stable` contains the latest backport of git-annex to Debian
stable.
* `debian-*-backport` contains the latest backport of git-annex.
* `tweak-fetch` adds support for the git tweak-fetch hook, which has
been proposed and implemented but not yet accepted into git.
* `setup` contains configuration for this website
* `pristine-tar` contains [pristine-tar](http://kitenet.net/~joey/code/pristine-tar)
data to create tarballs of any past git-annex release.
----

View file

@ -17,7 +17,7 @@ remote.
You should decide whether to use encryption with a special remote before
any data is stored in it. So, `git annex initremote` requires you
to specify "encryption=none" when first setting up a remote in order
to disable encryption. To use encryption, you run
to disable encryption. To use encryption, you run
`git-annex initremote` in one of these ways:
* `git annex initremote newremote type=... encryption=hybrid keyid=KEYID ...`
@ -29,10 +29,10 @@ to disable encryption. To use encryption, you run
The [[hybrid_key_design|design/encryption]] allows additional
encryption keys to be added on to a special remote later. Due to this
flexibility, it is the default and recommended encryption scheme.
git annex initremote newremote type=... [encryption=hybrid] keyid=KEYID ...
Here the KEYID(s) are passed to `gpg` to find encryption keys.
Here the KEYID(s) are passed to `gpg` to find encryption keys.
Typically, you will say "keyid=2512E3C7" to use a specific gpg key.
Or, you might say "keyid=joey@kitenet.net" to search for matching keys.
@ -58,8 +58,8 @@ risks associated with encryption.
Alternatively, you can configure git-annex to use a shared cipher to
encrypt data stored in a remote. This shared cipher is stored,
**unencrypted** in the git repository. So it's shared among every
clone of the git repository.
clone of the git repository.
git annex initremote newremote type=... encryption=shared
The advantage is you don't need to set up gpg keys. The disadvantage is
@ -74,10 +74,10 @@ and since it's exactly the way everyone else uses gpg.
git annex initremote newremote type=.... encryption=pubkey keyid=KEYID ...
A disavantage is that it is not easy to later add additional public keys
A disadvantage is that it is not easy to later add additional public keys
to the special remote. While the `enableremote` parameters `keyid+=` and
`keyid-=` can be used, they have **no effect** on files that are already
present on the remote. Probably the only use for these parameters is
present on the remote. Probably the only use for these parameters is
to replace a revoked key:
git annex enableremote myremote keyid-=2512E3C7 keyid+=788A3F4C

View file

@ -0,0 +1,21 @@
Hello,
I am using the latest git-annex with the webui having two local folders (one over nfs) connected as a full backup group.
On every reboot I get a jumping ball icon with the text:
"Attempting to repair [tr2]"
And the later the text:
"failed to sync to tr2"
The debug log is filled with entries like this, where the number of deltas is increasing:
[2014-08-26 20:34:50 CEST] PushRetrier: Syncing with tr2
fatal: pack has 15 unresolved deltas
error: unpack failed: index-pack abnormal exit
To /nfs/backup
! [remote rejected] git-annex -> synced/git-annex (n/a (unpacker error))
! [remote rejected] annex/direct/master -> synced/master (n/a (unpacker error))
error: failed to push some refs to '/nfs/backup''

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="anders7788"
ip="212.247.195.173"
subject="comment 1"
date="2014-08-27T06:59:08Z"
content="""
My version is:
assistant version 5.20140517.4
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="anders7788"
ip="89.160.15.173"
subject="[Solved]"
date="2014-09-04T15:47:24Z"
content="""
[Solved]
The following of the steps at: http://git-annex.branchable.com/tips/what_to_do_when_a_repository_is_corrupted/ provided some clues and I discovered that the disk was broken. But a big thanks to git-annex which made it possible to discover this issue early!
"""]]

View file

@ -0,0 +1,7 @@
I was wondering if there was a way to make shared encryption more secure. Here is my suggestion:
The shared repository is encrypted using a key for the whole repository, just the way normal encryption would work.
The server additionally keeps a copy of every user's public key.
When a user is authorized, their repository is initialized and they receive the common key, encrypted by their public key.
The only issue would be storage of the common key, which would have to be restricted to repository on a trusted machine.
Not sure if this would be easy for you to implement, but I figured I'd submit a post detailing it, to see if maybe it was doable.

View file

@ -0,0 +1 @@
Is it possible to use bitbucket as a remote?

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="108.236.230.124"
subject="comment 1"
date="2014-09-18T17:43:44Z"
content="""
You can use any git repository as a git remote in git-annex, since git-annex uses plain old git repos.
You will need to add some kind of [[special_remote|special_remotes]] to hold the content of the files stored in git-annex however.
"""]]

View file

@ -0,0 +1,13 @@
I have a local repo on a computer with some folder on the path
/computer1/annex/
Then I tried to run git annex on my phone and added a remote ssh repo syncing to
/computer1/annex/mobile/ from my mobile's picture folder.
But the files ending up there are:
ls /computer1/annex/mobile/
annex branches config description HEAD hooks info objects refs
I chose "full backup"
Why dont I see my pictures there, even if they are hiding in the metadata ?

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Petter_petterson"
ip="89.160.15.173"
subject="addition"
date="2014-09-13T07:54:58Z"
content="""
I understand that the copy of the cellphones' photos are stored on the server too, when typing git annex whereis <photo> I see that it exists on the server, but I need to be able to, at will copy out the jpg files for editing and using in other places.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
nickname="Justin"
subject="comment 2"
date="2014-09-14T18:57:12Z"
content="""
If `/computer1/annex/` was already an annex repository you should have synced the phone to that, not to a new bare repository at `/computer1/annex/mobile/`
"""]]

View file

@ -0,0 +1,36 @@
[[!comment format=mdwn
username="Petter_petterson"
ip="89.160.15.173"
subject="comment 3"
date="2014-09-16T08:13:59Z"
content="""
Thanks Justin, but that wont work. Even pointing out a normal, non-bare repo and then adding it as a ssh remote will convert it into a bare repo. I confirmed that, and then I read this post:
http://git-annex.branchable.com/forum/Local_and_remote_in_direct_mode/
that states that
> The \"Remote server using ssh\" option in the webapp is intended to set up a bare git repository on a server, not a non-bare git repository on a client.\"
I even tried to do
git remote add B ssh://machineB:/~/annex
but to no avail, the created annex on machine B becomes a bare repo.
The only way to do it for me was to do the following,
Assume my cellphone is device A, and my desktop is device B:
On machine B:
cd ~/DCIM
git init
git annex init \"B\"
git annex direct
echo '*/5 * * * * * cd ~/DCIM; git annex sync' > crontab
On machine A:
git clone ssh://user@machineB:/home/user/DCIM
git annex sync
git annex webapp
now pictures are synced to the computer in direct, non-bare format every 5 minutes. I have spent literally days on this and now I finally nailed it in a crude but working fashion.
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="108.236.230.124"
subject="comment 4"
date="2014-09-18T18:00:42Z"
content="""
I have double-checked, and when I already have an existing, non-bare repository, pointing the webapp at it over ssh keeps it as a non-bare repository. As I would expect.
> I even tried to do git remote add B ssh://machineB:/~/annex but to no avail, the created annex on machine B becomes a bare repo.
I didn't try this because it's such a violation of the way git actually works that I just can't believe it could happen. If it does, you've found the git bug of the year.
But, I think you just got confused about whether the repository existed before, or gave the wrong path to the existing repository which would result in a new, bare repository being created in the location you told it.
If you really think this happens, show a transcript with enough details for me, or the git developers, to reproduce the problem.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="108.236.230.124"
subject="comment 5"
date="2014-09-18T18:05:33Z"
content="""
It occurs to me that another way you could have gotten confused is, if ssh://machineB:/~/annex was eg, created in the first place by running the git-annex webapp on machineB, then it would be a direct mode repo. In this case, yes core.bare=true, but so does annex.direct=true. And that repository will not be a bare repo really; it will contain the same file tree as you have on your mobile.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Petter_petterson"
ip="89.160.15.173"
subject="comment 2"
date="2014-09-16T08:15:05Z"
content="""
Doing git remote add B ssh://machineB:/~/annex still makes the repository on machineB a bare one, just try it and check git config -l | grep core.bare...
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="108.236.230.124"
subject="comment 3"
date="2014-09-18T17:55:21Z"
content="""
Petter_petterson, I think you're mistaken about that. If you were right, that would be a massive bug in git -- nothing git-annex specific about that command after all.
"""]]

Some files were not shown because too many files have changed in this diff Show more