Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2013-09-12 12:21:50 -04:00
commit 525c762690
18 changed files with 363 additions and 12 deletions

View file

@ -0,0 +1,33 @@
> git annex status
supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
supported remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 8
00000000-0000-0000-0000-000000000001 -- web
44AF00F1-511F-4902-8235-DFF741B09400 -- here
44af00f1-511f-4902-8235-dff741b09400 -- chrissy
53499200-CA18-4B51-B6B3-651C18208349 -- stevedave
56C56658-0995-4613-8A1B-B2FA534A834C -- olaf
8FE9B19F-4FC8-4CFA-AD89-4B70EB432EDC -- passport
AFC75641-B34A-4644-B566-C8D3127823F7 -- glacier
B3238A12-D81B-40EA-BE89-3BDB318AE2B7 -- brodie
untrusted repositories: 0
transfers in progress: none
available local disk space: 78.8 gigabytes (+1 gigabyte reserved)
local annex keys: 3915
local annex size: 81.37 gigabytes
known annex keys: 5728
known annex size: 641.36 gigabytes
bloom filter size: 16 mebibytes (0.8% full)
backend usage:
SHA256E: 8716
URL: 927
> git annex version
git-annex version: 4.20130909
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi
local repository version: 3
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2

View file

@ -0,0 +1,17 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 1"
date="2013-09-12T09:44:54Z"
content="""
There does not seem to be an actual bug description in here. Can you provide more info?
To answer the actual question: RFC 4122 clearly answers no:
Each field is treated as an integer and has its value printed as a
zero-filled hexadecimal digit string with the most significant
digit first. The hexadecimal values \"a\" through \"f\" are output as
lower case characters and are case insensitive on input.
Yes, that means that git-annex prints them wrongly as that's upper case.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 2"
date="2013-09-12T09:49:56Z"
content="""
What OS are you using?
versions 4.20130827 and 4.20130911 on Debian Sid print UUIDs which are lower case, as should be expected.
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ"
nickname="Nicolas"
subject="comment 3"
date="2013-09-11T23:19:26Z"
content="""
This behaviour is indeed very confusing when one attempts to use git-annex with the assistant only, without thinking about the underlying machinery.
Having the assistant automatically put a .gitignore file in empty directories is a possible solution, but maybe printing a warning in the log when the watcher sees an empty directory could be an unobtrusive and helpful solution? (At least it would have helped me)
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ"
nickname="Nicolas"
subject="comment 4"
date="2013-09-11T23:24:58Z"
content="""
Another problem with the current behaviour is that when deleting a directory and its contents on a repository, the contents are deleted on the other repositories but empty directories are left behind...
"""]]

View file

@ -0,0 +1,33 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlEhzszkzOIy8-Rx8b2mcr75QcnIc6O_OA"
nickname="Rachel"
subject="I'm afraid I'm getting this too"
date="2013-09-12T12:48:04Z"
content="""
Very similar setup. Initially set up on a Mac, then tried to set up the first local pairing with a Linux system. Basically seeing exactly the same thing.
bad comment in ssh public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDa09lpmgLeRVF1KZn2gX6tmBH4Jpcou/rkPbhwLoFNLtQsdNrbjDKuyc4pDbUhexWVvub1YCLItLK0vYRdMvoJWjNzy926sHs8CxXFVg6PMQXw3wOHousSn0NkVfwUMNn5aS+1vz/WDlStmg7WzXgiPg1Whn6CKIomxA63rUMe3I+2nVkyO6jGlOkliPApVP2utURJbyxBDYDB+Ys7zUxvEcCekhxJO263myuo2gwI4A3mfTLRJMzbz8frRZJz5iuUIcOHaIDy6n0qKFq18BDux4SDwNjQ34yti5yqveRBMwFndHQYN6YHk1k26h0kyhb2T6lXaaAirSqa3dwfZuKp rachel@celestia.local
# Versions and Stuff:
## Mac:
* OS X 10.8.4
* git-annex 4.20130909-ga29f960 (downloaded about half an hour ago!)
* BTW I have git installed from Homebrew, version 1.8.3.4 in my shell's $PATH, but obviously /usr/bin/git from the system is intact.
* Hostname: celestia.local
* Username: rachel
## Linux:
* Ubuntu 13.04
* git-annex 3.20121112ubuntu4 (from Ubuntu's own repos)
* I note that's later than the version you said this bug was fixed in, hence the report. :-)
* Hostname: twilight.local (mDNS domain not appended even by hostname -f but avahi is working)
* Username: rachel
IPv6 is active, but I note a comment somewhere that that's probably not relevant for git-annex.
Machines are very local to each other, in the same room, connected via a gigabit switch. They can definitely see each other. :-)
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmpBDWko1yZmngBYzm_CeBv8RjoIMXaINE"
nickname="Rachel"
subject="latest works"
date="2013-09-12T13:19:09Z"
content="""
Downloaded the latest tarball (4.20130911-g6625d0e) and that seems to be able to pair ok.
Different bug though: It objects to push.default = simple in ~/.gitconfig, but no such complaint on osx. But that's a different bug; i'll report separately when I'm a bit more up to speed.
"""]]

View file

@ -0,0 +1,9 @@
git-annex 4.20130911 on Debian Wheezy.
Using `git annex initremote type=gcrypt` as described [here](http://git-annex.branchable.com/tips/fully_encrypted_git_repositories_with_gcrypt/) creates the repository and pushes the `git-annex` branch to it. It doesn't push `master` (or more generally: the currently checked out branch), as I have verified using `git remote show`.
Manually pushing the branch makes `git annex sync` sync both branches with the gcrypt remote.
I think that it should push the current branch upon creation of the special remote, since for at least me, the whole reason for wanting `git-remote-gcrypt` is that metadata as well as data can be stored encrypted on an SSH server.
Thanks for considering this.

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnRai_qFYPVvEgC6i1nlM1bh-C__jbhqS0"
nickname="Matthew"
subject="Definitely broken Galaxy Nexus GSM 4.3"
date="2013-09-12T11:11:34Z"
content="""
Definitely broken in 4.3, stock Galaxy Nexus GSM \"maguro\" release `2013-09-09 12:46`
"""]]

View file

@ -0,0 +1,67 @@
[[!comment format=mdwn
username="gary"
ip="82.112.140.250"
subject="Thanks for the fix"
date="2013-09-12T07:20:12Z"
content="""
Hi the fix works fine, all the tests pass now. However I am having issues when trying to sync two directories. I have had this working in direct mode in fedora but not in the previously mentioned windows install
2 directories d1 and d2.
SETUP:
cd d1
mkdir annex
cd annex
cp someFile . (as get branch issue if empty)
git init
git annex init d1
cp someFile . (as get branch issue if empty)
git annex add .
git annex sync
cd ..\..\d2
git clone ..\..\d1\annex
cd annex
git annex init d2
git remote add d1 ..\..\d1\annex
cd ..\..\d1\annex
git remote add d2 ..\..\d2\annex
SYNCING
cd d1\annex
cp someFiles .
git annex add .
git annex sync
cd ..\..\d2\annex
git annex sync
git annex mirror --from d1
It looks like the files copy across but when I look at them in explorer they are still just the place holders.
any ideas on what may be going on. Does the work flow look correct?
(ps typo on all slashes but
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="spwhitton"
ip="82.36.235.9"
subject="Converting an rsync special remote to a gcrypt special remote"
date="2013-09-11T17:36:09Z"
content="""
I'm guessing there is no way to convert an rsync special remote to a gcrypt special remote? It would be cool not to have to upload 100GB across the Atlantic again!
"""]]

View file

@ -0,0 +1,85 @@
[[!comment format=mdwn
username="EmanueleAina"
ip="93.38.211.231"
subject="comment 2"
date="2013-09-11T17:55:14Z"
content="""
I seem to hit the same issue (`ResponseTimeout`) on box.com, but I don't have any log file under .git/annex.
The timeout is quite unpredictable, sometimes I'm able to transfer less than 20% of the file while sometimes it fails after reaching 51%. The file is quite big, 6.2G.
Unfortunately, after the first upload fails, `git-annex` seems to think that the file has been uploaded successfully and will refuse to copy it again. Even `whereis` will list the box.com location.
I think there are really two bugs: the one triggering the timeouts, and the fact that `git-annex` thinks that the failing upload succeeded and is unable to recover the interrupted upload.
Here's the log I got on the shell. Let me know how I can provide any additional information needed. Thanks!
$ git annex copy --verbose --debug home.tar.gz --to box.com
[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
[2013-09-11 17:42:52 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
[2013-09-11 17:42:52 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
copy home.tar.gz (gpg) [2013-09-11 17:42:52 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
(checking box.com...) (to box.com...)
[2013-09-11 17:42:57 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"11\",\"--symmetric\",\"--force-mdc\",\"--no-textmode\"]
35% 660.8KB/s 1h44mResponseTimeout
gpg: [stdout]: write error: Broken pipe
gpg: DBG: deflate: iobuf_write failed
gpg: build_packet failed: file write error
gpg: [stdout]: write error: Broken pipe
gpg: iobuf_flush failed on close: file write error
gpg: [stdout]: write error: Broken pipe
gpg: iobuf_flush failed on close: file write error
gpg: symmetric encryption of `[stdin]' failed: file write error
git-annex: fd:13: hPutBuf: resource vanished (Broken pipe)
failed
git-annex: copy: 1 failed
$ git annex copy --verbose --debug home.tar.gz --to box.com
[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
[2013-09-11 19:38:26 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
[2013-09-11 19:38:26 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
copy home.tar.gz (gpg) [2013-09-11 19:38:26 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
(checking box.com...) ok
$ git annex whereis --verbose --debug home.tar.gz
[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
[2013-09-11 19:38:57 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
[2013-09-11 19:38:57 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
whereis home.tar.gz (2 copies)
d7db543e-5463-11e2-b7dd-9f423f798cc4 -- here (em@ocracy:/data/backup)
e71fa45e-5463-11e2-a14e-93ca09c272da -- box.com
ok
$ git annex fsck --verbose --debug home.tar.gz
[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
[2013-09-11 19:39:05 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
fsck home.tar.gz [2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
[2013-09-11 19:39:05 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
[2013-09-11 19:39:05 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
(checksum...) [2013-09-11 19:39:05 CEST] read: sha256sum [\"/data/backup/.git/annex/objects/mG/kp/SHA256E-s6640317400--dcf0a535728f3f3f787db6339b740a4a6f6529e5ce1d238f28574499a8172670.tar.gz/SHA256E-s6640317400--dcf0a535728f3f3f787db6339b740a4a6f6529e5ce1d238f28574499a8172670.tar.gz\"]
ok
$ git annex fsck --verbose --debug home.tar.gz --from box.com
[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"git-annex\"]
[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1e029f92c5c82b094cbe70b55927996c31579e2e\",\"--oneline\",\"-n1\"]
[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"log\",\"refs/heads/git-annex..1090c21b121fd8b6e2ab49a772c8eab5235a3930\",\"--oneline\",\"-n1\"]
[2013-09-11 19:42:15 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"cat-file\",\"--batch\"]
[2013-09-11 19:42:15 CEST] read: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"home.tar.gz\"]
[2013-09-11 19:42:15 CEST] chat: git [\"--git-dir=/data/backup/.git\",\"--work-tree=/data/backup\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"--\"]
fsck home.tar.gz (gpg) [2013-09-11 19:42:15 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--decrypt\"]
(checking box.com...)
[2013-09-11 19:42:28 CEST] chat: gpg [\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"10\",\"--decrypt\"]
gpg: decrypt_message failed: eof
ok
"""]]

View file

@ -0,0 +1,13 @@
I'm trying to figure out if git-annex can be used for the following use case:
- 2 repos in direct mode A and B
- A is on a local filesystem, B could be a NFS mountpoint or ssh
- B contains a number of toplevel directories, A contains a subset of those
- For the subset in A all changes in those directories that happen on A or B should be synced automatically between both repos, including file removals.
- A can decide to not carry a toplevel dir any more, but it must still exist in B.
- A can copy new toplevel dirs from B and those should be synced from then on.
I've been looking at the docs and played with two test repos, but I cannot seem to make the above work. To me it looks like I would need a possibility include/exclude paths from syncing via a regex.
Thanks,
Felix

View file

@ -1,3 +1,4 @@
# Latest version 0.1.10
Hook program for gitannex to use flickr as backend.
This allows storing any type of file on flickr, not only images and movies.
@ -10,7 +11,7 @@ Credit for the flickr api interface goes to: <http://stuvel.eu/flickrapi>
Credit for the png library goes to: <https://github.com/drj11/pypng>
Credit for the png tEXt patch goes to: <https://code.google.com/p/pypng/issues/detail?id=65>
## Install
# Install
Clone the git repository in your home folder.
@ -18,7 +19,7 @@ Clone the git repository in your home folder.
This should make a ~/flickrannex folder
## Setup
# Setup
Run the program once to set it up.
@ -26,24 +27,36 @@ Run the program once to set it up.
After the setup has finished, it will print the git-annex configure lines.
## Configuring git-annex
# Configuring git-annex
git config annex.flickr-hook '/usr/bin/python2 ~/flickrannex/flickrannex.py'
git annex initremote flickr type=hook hooktype=flickr encryption=shared
git annex describe flickr "the flickr library"
## Notes
# Notes
### Unencrypted mode
## Unencrypted mode
The photo name on flickr is currently the GPGHMACSHA1 version.
The photo name on flickr is currently the [[key|backends]] used by git-annex.
Run the following command in your annex directory
git annex content flickr uuid include=*.jpg or include=*.jpeg or include=*.gif or include=*.png
### Encrypted mode
## Encrypted mode
The current version base64 encodes all the data, which results in ~35% larger filesize.
The current version base64 encodes all the data, which results in ~35%
larger filesize.
I might look into yyenc instead. I'm not sure if it will work in the tEXt field.
I might look into yyenc instead. I'm not sure if it will work in the tEXt
field.
Run the following command in your annex directory
git annex content flickr exclude=largerthan=30mb
-- Tobias
## Including directories as tags
Get get each of the directories below the top level git directory added as tags to uploads:
git config annex.flickr-hook 'GIT_TOP_LEVEL=`git rev-parse --show-toplevel` /usr/bin/python2 %s/flickrannex.py'
In this case the image:
/home/me/annex-photos/holidays/2013/Greenland/img001.jpg
would get the following tags: "holidays" "2013" "Greenland"
(assuming "/home/me/annex-photos" is the top level in the annex...)
Caveat Emptor - Tags will *always* be NULL for indirect repos - we don't (easily) know the human-readable file name.

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
nickname="develop"
subject="Version 0.1.10 pushed"
date="2013-09-11T20:31:25Z"
content="""
Since the initial release of this hook a lot of issues have been fixed, and a few features added.
I would highly suggest that everyone who is using this hook update to the latest version as i would consider one of the bugs to be fairly major.
"""]]

View file

@ -0,0 +1 @@
That would make assessing weird reports like [[bugs/Should_UUID__39__s_for_Remotes_be_case_sensitive__63__/]] easier and quicker.

View file

@ -0,0 +1,5 @@
It would be great to be able to use the pubDate of the entries with the --template option of importfeed.
Text.Feed.Query has a getItemPublishDate (and a getFeedPubDate, if we want some kind of ${feeddate}).
The best would be to allow a reformating of the date(s) with (for example) %Y-%m-%D

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="gueux"
ip="2a01:240:fe6d:0:7986:3659:a8bd:64f1"
subject="syntax"
date="2013-09-12T14:05:16Z"
content="""
use \"itemdate\" and \"feeddate\" as names?
use ${itemdate=%Y-%m-%D} syntax option?
"""]]