Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2014-12-01 19:34:45 -04:00
commit 5cc406572b
12 changed files with 282 additions and 15 deletions

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
nickname="Justin"
subject="comment 5"
date="2014-11-30T22:18:56Z"
content="""
This is getting in my way, but I'm concerned about using a mostly-untested branch (and also, compiling for ARM sounds really unpleasant). Can you elaborate on the risks here? Can I perhaps donate developer time to get this merged into mainline?
"""]]

View file

@ -0,0 +1,22 @@
### Please describe the problem.
Just feedback to you:
After I uninstalled git-annex (Windows), traditional git was not able to push changes to an origin server via ssh.
Instead I got an error "libcrypto.dll is missing".
Installing git once again after uninstalling git-annex solved the problem.
### What steps will reproduce the problem?
Installing and uninstalling git-annex.
### What version of git-annex are you using? On what operating system?
File "git-annex-installer.exe" downloaded 2014-11-09.
Windows 8.
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]

View file

@ -0,0 +1,35 @@
### Please describe the problem.
I am trying to verify that my files are where they should after having updated my remotes to use chunking. I do get the following two error messages, but then git-annex reports that everything is Ok. I find it slightly scary and thus wonder if the fsck command in all cases do a conservative enough check. All other files (except two other ones where it noticed that they where in fact not on the remote do not report any errors.)
When running the fsck command for all files I got the following slightly worring message for one file.
fsck Ekonomi/2014-06-02_Elavräkning.pdf (checking box.com5...) (checking box.com5...) (checking box.com5...)
79% 23.2KB/s 26s
Error_Packet "partial packet: expecting 16448 bytes, got: 891"
ok
Reading the message it seems that the file was not correctly downloaded, but still deemed Ok enough by git-annex.
Further while writing this report I get the following message from a remote using the external protocol:
fsck Deklaration/archive/2012/DE/Steuerklärung Formular.pdf
93% 221.9KB/s 12s
/Users/carlmod/Annex/.git/annex/tmp/GPGHMACSHA1--0e7e4951e2650ac636bc3fc17c673f1f2dfb3071: openBinaryFile: does not exist (No such file or directory)
ok
Source code for this remote is not yet published online, as i would want to build it slightly more reliable before releasing it. It however passes the remote test suite without any problem.
### What steps will reproduce the problem?
Not really sure, but rerunning the commands show that the openBinaryFile happens here for this particular file.
### What version of git-annex are you using? On what operating system?
$ git annex version
git-annex version: 5.20141125
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
built using Homebrew (i.e. in a cabal sandbox) on OS X Yosemite 10.10.1

View file

@ -1,6 +1,6 @@
### Please describe the problem.
The new ''git annex proxy'' command fails every command in the 2014-11-28 daily build for Windows.
The new ''git annex proxy'' command fails every call in the 2014-11-28 daily Windows build.
### What steps will reproduce the problem?
@ -15,7 +15,7 @@ The new ''git annex proxy'' command fails every command in the 2014-11-28 daily
### What version of git-annex are you using? On what operating system?
Daily Windows build 5.20141128-g70f997e
Daily Windows build 5.20141128-g70f997e on Windows 7 x64
### Please provide any additional information below.

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmWBvsZvSsAL8P2ye3F0OBStjFCVnOImzM"
nickname="Jarno"
subject="Not a bug"
date="2014-11-28T18:25:40Z"
content="""
This was, in fact, just an unclear error message. The command was missing \"git\", and so should have been:
$ git annex proxy -- git pull
"""]]

View file

@ -0,0 +1,45 @@
The current command fails (git-annex 5.20141125, debian sid, DAV server: radicale 0.9-1):
$ WEBDAV_USERNAME=user WEBDAV_PASSWORD=password git annex --debug initremote dav.example.com type=webdav url=https://dav.example.com encryption=none
[2014-11-28 16:53:29 CET] read: git ["--git-dir=/home/user/test-annex/.git","--work-tree=/home/user/test-annex","show-ref","git-annex"]
[2014-11-28 16:53:29 CET] read: git ["--git-dir=/home/user/test-annex/.git","--work-tree=/home/user/test-annex","show-ref","--hash","refs/heads/git-annex"]
[2014-11-28 16:53:29 CET] read: git ["--git-dir=/home/user/test-annex/.git","--work-tree=/home/user/test-annex","log","refs/heads/git-annex..f60365a0d5333e1aefe80a3fe747e2cb3d5c8162","-n1","--pretty=%H"]
[2014-11-28 16:53:29 CET] chat: git ["--git-dir=/home/user/test-annex/.git","--work-tree=/home/user/test-annex","cat-file","--batch"]
initremote dav.example.com (testing WebDAV server...)
git-annex: WebDAV test failed: StatusCodeException (Status {statusCode = 500, statusMessage = "Internal Server Error"}) [("Server","nginx/1.2.1"),("Date","Fri, 28 Nov 2014 15:53:37 GMT"),("Content-Type","text/plain"),("Content-Length","59"),("Connection","keep-alive"),("Keep-Alive","timeout=75"),("X-Response-Body-Start","A server error occurred. Please contact the administrator."),("X-Request-URL","MKCOL https://dav.example.com:443/tmp")] (CJ {expose = []}): user error
failed
git-annex: initremote: 1 failed
server log:
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:17 +0100] "MKCOL / HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:18 +0100] "PROPFIND /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:21 +0100] "MKCOL /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:32 +0100] "MKCOL / HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:34 +0100] "PROPFIND /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:53:37 +0100] "MKCOL /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:58:31 +0100] "PROPFIND / HTTP/1.1" 207 1035 "-" "cadaver/0.23.3 neon/0.30.1"
::ffff:192.168.1.6 - user [28/Nov/2014:16:59:19 +0100] "MKCOL / HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:59:21 +0100] "PROPFIND /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
::ffff:192.168.1.6 - user [28/Nov/2014:16:59:23 +0100] "MKCOL /tmp HTTP/1.1" 500 59 "-" "hDav-using application"
cadaver works:
$ cadaver https://dav.example.com
Authentication required for Radicale - Password Required on server `dav.example.com':
Username: user
Password:
dav:/> ls
Listing collection `/': succeeded.
Coll: user 0 Jan 1 1970
server log:
::ffff:192.168.1.6 - - [28/Nov/2014:17:01:31 +0100] "OPTIONS / HTTP/1.1" 200 0 "-" "cadaver/0.23.3 neon/0.30.1"
::ffff:192.168.1.6 - - [28/Nov/2014:17:01:31 +0100] "PROPFIND / HTTP/1.1" 401 0 "-" "cadaver/0.23.3 neon/0.30.1"
::ffff:192.168.1.6 - user [28/Nov/2014:17:01:42 +0100] "PROPFIND / HTTP/1.1" 207 579 "-" "cadaver/0.23.3 neon/0.30.1"
::ffff:192.168.1.6 - user [28/Nov/2014:17:01:42 +0100] "PROPFIND / HTTP/1.1" 207 579 "-" "cadaver/0.23.3 neon/0.30.1"
::ffff:192.168.1.6 - user [28/Nov/2014:17:02:01 +0100] "PROPFIND / HTTP/1.1" 207 1035 "-" "cadaver/0.23.3 neon/0.30.1"

View file

@ -0,0 +1,3 @@
My PC froze while git-annex was in the process of importing a file. It was in the final stage of updating git status when the freeze occurred. git status shows the imported file under "changes to be committed" as "new file."
I don't know how git-annex works under the hood, so I don't know how the freeze might've affected the repo. I don't want to lose my files. What do you recommend is the best way of recovering from this and making sure there are no errors?

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmWBvsZvSsAL8P2ye3F0OBStjFCVnOImzM"
nickname="Jarno"
subject="How about with the 'proxy' command?"
date="2014-11-28T18:47:17Z"
content="""
Why does this not work?
git annex proxy -- git reset HEAD
"""]]

View file

@ -1,13 +0,0 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmWBvsZvSsAL8P2ye3F0OBStjFCVnOImzM"
nickname="Jarno"
subject="How about with the 'proxy' command?"
date="2014-11-27T22:40:17Z"
content="""
With the proxy command, would the following be an appropriate way to discard all changes (mods, adds, dels) since last sync in a direct mode repo?
git annex proxy -- git clean -df
git annex proxy -- git checkout -f .
Also, is it necessary to run annex sync or other direct mode housekeeping commands afterwards?
"""]]

View file

@ -0,0 +1,54 @@
Hi,
I have run the git annex repair command on one of my repositories and it has been running for 38h hours now. I have monitored the process and its sons with htop and it seems that git annex repair is stuck.
It repeatedly launches the following command:
git --git-dir=/home/vincent/photos2/.git --work-tree=/home/vincent/photos2 ls-tree --full-tree -z -r -- 5fe5193e079631c0ceac0688ae0a6c1636491b61
I have tried to execute it by hand and it produces a lot of output. I have redirected it into some file in order to count the content produced by this command.
vincent@berlioz:~/photos2/.git$ git --git-dir=/home/vincent/photos2/.git --work-tree=/home/vincent/photos2 ls-tree --full-tree -z -r -- 5fe5193e079631c0ceac0688ae0a6c1636491b61 > /tmp/ls-tree
vincent@berlioz:/tmp$ wc ls-tree
0 232525 11611220 ls-tree
Could this cause some troubles to git-annex ? Maybe my repository is just too big and it demands some time to deal with the whole thing?
Before doing the git annex repair command I have tried to delete a distant annex (hosted on a raspberry pi, running an arm raspbian) from this one, using the git annex webapp website. Since it was kind of stuck I asked webapp to shutdown. I ran git annex repair for some time. It looked like stuck. I stopped it, then tried git fsck. It ran well. I tried git annex fsck, it took all my RAM and swap. The process was eating 14 Giga bytes of memory. I tried git gc and the same memory problem occured. I ran git annex repair again and it has been running for 38 hours now. I am lost. Please help.
vincent@berlioz:~$ LANG=C apt-cache policy git-annex
git-annex:
Installed: 5.20141125
Candidate: 5.20141125
Package pin: 5.20141125
Version table:
*** 5.20141125 950
400 http://ftp.fr.debian.org/debian/ testing/main amd64 Packages
50 http://ftp.fr.debian.org/debian/ unstable/main amd64 Packages
100 /var/lib/dpkg/status
3.20120629 950
900 http://ftp.fr.debian.org/debian/ wheezy/main amd64 Packages
vincent@berlioz:~/photos2$ du -sh .
152G .
Info gathered from another client annex:
vincent@tigrou:/mnt/mars/images/photos2$ git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 11
8<------------skipped repositories list------------->8
untrusted repositories: 1
a357bb9e-7a9d-4112-b74c-13707e2c7f85 -- vincent@rhett:~/photos [gateway]
transfers in progress: none
available local disk space: 945.91 gigabytes (+1 megabyte reserved)
bad keys size: 368 bytes (clean up with git-annex unused)
local annex keys: 76070
local annex size: 383.91 gigabytes
annexed files in working tree: 59117
size of annexed files in working tree: 319.73 gigabytes
bloom filter size: 16 mebibytes (15.2% full)
backend usage:
SHA256E: 101902
SHA256: 33285

View file

@ -0,0 +1,37 @@
[[!comment format=mdwn
username="vho"
subject="Update"
date="2014-12-01T15:47:23Z"
content="""
It seems I could sort things of.
I have run git annex sync from another annex. It took 30 minutes but could synchronize with the slow annex. Then I launched git annex sync on the slow annex (the one which seemed to had troubles with the repair command). It took 278 minutes, used all my ram and swap, but managed to end successfully.
I launched git annex sync again today, another 279 minutes:
vincent@berlioz:~/photos2$ time git annex sync
commit ok
pull tigrou
Error reading response length from authentication socket.
Compression automatique du dépôt en tâche de fond pour optimiser les performances.
Voir \"git help gc\" pour toute information sur le nettoyage manuel.
ok
pull gateway
Compression automatique du dépôt en tâche de fond pour optimiser les performances.
Voir \"git help gc\" pour toute information sur le nettoyage manuel.
ok
push gateway
fatal: The remote end hung up unexpectedly
Pushing to gateway failed.
(non-fast-forward problems can be solved by setting receive.denyNonFastforwards to false in the remote's git config)
failed
git-annex: sync: 1 failed
real 279m27.760s
user 7m15.159s
sys 3m4.340s
I will try to do some git gc.
"""]]

View file

@ -0,0 +1,55 @@
# Creating a special S3 remote to hold files shareable by URL
(In this example, I'll assume you'll be creating a bucket in S3 named **public-annex** and a special remote in git-annex, which will store its files in the previous bucket, named **public-s3**, but change these names if you are going to do the thing for real)
First, in the AWS dashboard, go to (or create) the bucket you will use at S3 and add a public get policy to it:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::public-annex/*"
}
]
}
Then set up your special [S3](http://git-annex.branchable.com/special_remotes/S3/) remote with (at least) these options:
git annex initremote public-s3 type=s3 encryption=none bucket=public-annex chunk=0
This way git-annex will upload the files to this repo, (when you call `git annex copy [FILES...] --to public-s3`) without encrypting them and without chunking them, and, because of the policy of the bucket, they will be accessible by anyone with the link.
Following the example, the files will be accessible at `http://public-annex.s3.amazonaws.com/KEY` where `KEY` is the file key created by git-annex and which you can discover running
git annex lookupkey FILEPATH
This way you can share a link to each file you have at your S3 remote.
___________________
## Sharing all links in a folder
To share all the links in a given folder, for example, you can go to that folder and run (this is an example with the _fish_ shell, but I'm sure you can do the same in _bash_, I just don't know exactly):
for filename in (ls)
echo $filename": https://public-annex.s3.amazonaws.com/"(git annex lookupkey $filename)
end
## Sharing all links matching certain metadata
The same applies to all the filters you can do with git-annex.
For example, let's share links to all the files whose _author_'s name starts with "Mario" and are, in fact, stored at your public-s3 remote.
However, instead of just a list of links we will output a markdown-formatted list of the filenames linked to their S3 files:
for filename in (git annex find --metadata "author=Mario*" --and --in public-s3)
echo "* ["$filename"](https://public-annex.s3.amazonaws.com/"(git annex lookupkey $filename)")"
end
Very useful.