Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2014-01-10 15:24:46 -04:00
commit 45555b2deb
19 changed files with 343 additions and 81 deletions

View file

@ -0,0 +1,28 @@
### Please describe the problem.
git-annex does not check for the minimum version of Git that it requires, and therefore it sometimes uses flags which my installed Git does not recognize.
This happens when converting some indirect repositories to direct ones on an older server.
### What steps will reproduce the problem?
git annex direct
### What version of git-annex are you using? On what operating system?
git-annex version: 5.20131130
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
direct iTunes/iTunes Media/iTunes U/Statistics 110_ Probability/Strategic Practice and Homework 9.pdf ok
direct iTunes/iTunes Media/iTunes U/Statistics 110_ Probability/Syllabus.pdf ok
direct error: unknown switch `B'
usage: git checkout [options] <branch>
or: git checkout [options] [<branch>] -- <file>...
# End of transcript or log.
"""]]

View file

@ -0,0 +1,54 @@
### Please describe the problem.
I was playing around thinking to try out git-annex, and bup
I sucessfully added 6GB of images in subdirectories in ~/Pictures, so decided to push over another dir full -
more than 30GB more...
sven@quad:~/Pictures$ git annex add .
....
add 2012/01/10/20120110-024457/IMG_1830.JPG
git-annex: <file descriptor: 7>: hFlush: resource vanished (Broken pipe)
failed
add 2012/01/10/20120110-024457/IMG_1831.JPG
git-annex: <file descriptor: 7>: hFlush: resource vanis
sven@quad:~/Pictures$
sven@quad:~/Pictures$ git annex add .
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
sven@quad:~/Pictures$ git git commit -a -m added
git: 'git' is not a git command. See 'git --help'.
Did you mean this?
init
sven@quad:~/Pictures$ git commit -a -m added
Bus error (core dumped)
### What steps will reproduce the problem?
see above
### What version of git-annex are you using? On what operating system?
sven@quad:~/Pictures$ git annex version
git-annex version: 3.20120406
local repository version: 3
default repository version: 3
supported repository versions: 3
upgrade supported from repository versions: 0 1 2
sven@quad:~/Pictures$ uname -a
Linux quad 3.8.0-34-generic #49~precise1-Ubuntu SMP Wed Nov 13 18:05:00 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://id.koumbit.net/anarcat"
ip="2001:1928:1:9::1"
subject="comment 7"
date="2014-01-09T23:51:02Z"
content="""
Okay, so the current situation is this: joey can't reproduce this, but I can reliably can, with the backport. So there has to be something different in our setups.
We are suspecting the kernel, so I'm trying to upgrade to the backport, we'll see.
"""]]

View file

@ -0,0 +1,56 @@
### Please describe the problem.
`git annex map` shows broken graph when direct mode is used
### What steps will reproduce the problem?
See transcript
### What version of git-annex are you using? On what operating system?
[[!format sh """
$ git annex version
git-annex version: 5.20140107
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook external
$ LANG=C apt-cache policy git-annex
git-annex:
Installed: 5.20140107
Candidate: 5.20140107
Version table:
*** 5.20140107 0
50 http://http.debian.net/debian/ sid/main amd64 Packages
100 /var/lib/dpkg/status
"""]]
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
git init a
cd a
git annex init
cd ..
git clone a b
cd b
git annex init
# OK
git annex map
cd ../a
git annex direct
cd ../b
# NOT OK
git annex map
# End of transcript or log.
"""]]

View file

@ -0,0 +1,50 @@
### Please describe the problem.
I get an error in one of my repositories.
### What steps will reproduce the problem?
I paired my macbook repository with an repository on my desktop. The repository on the desktop contains many smaller files, because it is my documents folder.
Pairing this repository with smaller files never worked for me. I posted to the forum about it:
[[forum/ssh__95__exchange__95__identification:_read:_Connection_reset_by_peer/]]
I am a little confused because I paired three other repositories between these computers sucessfully. They contain pictures, videos and music.
### What version of git-annex are you using? On what operating system?
Version: 5.20140106-gcb3351b
Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
Mac OS X Mavericks
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
Please make sure you have the correct access rights
and the repository exists.
fatal: index file smaller than expected
TransferScanner crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
[2014-01-10 16:26:00 CET] TransferScanner: warning TransferScanner crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
ssh_exchange_identification: read: Connection reset by peer
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[2014-01-10 16:56:09 CET] NetWatcherFallback: Syncing with sokrates.local_Dokumente
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: index file smaller than expected
NetWatcherFallback crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
[2014-01-10 17:01:09 CET] NetWatcherFallback: warning NetWatcherFallback crashed: user error (git ["--git-dir=/Users/juh/Share/.git","--work-tree=/Users/juh/Share","-c","core.bare=false","update-index","-z","--index-info"] exited 128)
# End of transcript or log.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89"
nickname="John"
subject="Still a problem"
date="2014-01-09T18:20:55Z"
content="""
I can confirm that this is still a problem. My iTunes Music folder contains songs with Unicode in the file names, and git-annex fails to sync it properly between Mac, ZFS and Linux.
I realize that supporting every filesystem encoding is impossible, but if you used something like `system-filepath` and expected UTF8, it might be better than the current state of affairs.
"""]]

View file

@ -0,0 +1,24 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8"
nickname="TobiasTheViking"
subject="comment 10"
date="2014-01-10T13:16:31Z"
content="""
I have just pushed version 0.2.0 of the following hooks:
googledriveannex
skydriveannex
imapannex
megaannex
flickrannex
owncloudannex
dropboxannex
"""]]

View file

@ -0,0 +1,9 @@
I'm trying to use git-annex as a way to sync fairly large folders between machines. These folders contain documents, projects, ...
There are several issues with files that just won't get added to my git-annex repository:
- files mentioned in .gitignore are not added -- this is good.
- files starting with a dot are not added unless I explicitly mention them to git add -- this is strange, but I can work around it.
- folders called .git/ are not added no matter what I try. git annex add --force doesn't help, either.
Any idea how I could solve the third issue? It might not even be specific to git-annex... maybe git just works that way... but I'd be grateful for hints.

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk"
nickname="Jonas"
subject="comment 1"
date="2014-01-09T22:58:30Z"
content="""
I just found [this thread](http://git-annex.branchable.com/forum/Tracking_a_directory_with_some_hg_and_git_repositories/) where a user was discouraged from syncing git repositories with annex.
I've used unison in the past for this. Sure, it would occasionally create a few unneeded object files; but these got removed during the next \"git gc\", and never corrupted a repository... Do people know of any better way to handle this problem?
"""]]

View file

@ -0,0 +1,11 @@
I use git everywhere i can to manage versions, but I have a usecase that is probably very common, but i could not see in the howto lists.
I have a shared fileserver(FS) that is managed by others and is full of the equivalent of binary blobs (media, word docs etc). Binary blobs of course are something that git cannot normally do anything meaningful with. That is diff/patch for managing the evolution of any given file. Binary blobs have a different lifecycle of course, and i see annex as the solution for managing that.
The remote FS is being used as an SMB server to windows and linux clients. The FS has other capabilities that i can use other than SMB mount including rsync and ssh. As it is running samba it also uses symlinks underneath the the SMB to manage shared folders, which can be seen when mounted in unix mode. This means the FS landscape is different when using samba as compared to rsync and ssh, as some paths may be symlinked in. The symlinks should not effect this use case, but I mention them because I don't know how annex would manage this.
The basic scenario is that i have a set of project files and folders in tree on the remote FS. I want to use git to pull these binary files from the FS to edit locally while also seein/capturing changes made to files in that tree by other people not using git. Then i want to push my edits back, without effecting the files that i have no responsibility for (but have write access to).
Think a team of people with various technical capabilities working on changing binary blobs in this remote directory tree. I want to manage my interactions with portions of that remote FS tree via git (annex) with a local mirror (preferably actually containing only the files i care about) so that i can manipulate them locally, clone to other work machines and generally play about with and then push back while watching the whole tree and capturing changes others make (during sync/merge/push/pull?). Because i do not manage it, or own the contents, i also need to be able to maintain the some of the permissions (guid/uid) of the remote and untouched files.
Can you give me any suggestions?

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Uic1ohv6"
ip="62.253.227.100"
subject="Are you using encfs?"
date="2014-01-09T16:30:50Z"
content="""
I've seen similar error messages when the local annex was on an encfs-mounted decrypted directory. Seemingly git-annex tries to create SSH sockets in .git, and this goes wrong at least in some such situations.
"""]]

View file

@ -1,4 +1,4 @@
dropboxannex
dropboxannex 0.2.0
=========
Hook program for gitannex to use dropbox as backend
@ -6,6 +6,7 @@ Hook program for gitannex to use dropbox as backend
# Requirements:
python2
python-pkg-resources
Credit for the Dropbox api interface goes to Dropbox.
@ -17,12 +18,12 @@ Clone the git repository in your home folder.
This should make a ~/dropboxannex folder
# Setup
Run the program once to set it up.
Make the file executable, and link it into PATH
cd ~/dropboxannex; python2 dropboxannex.py
cd ~/dropboxannex; chmod +x git-annex-remote-dropbox; sudo ln -sf `pwd`/git-annex-remote-dropbox /usr/local/bin/git-annex-remote-dropbox
# Commands for gitannex:
git config annex.dropbox-hook '/usr/bin/python2 ~/dropboxannex/dropboxannex.py'
git annex initremote dropbox type=hook hooktype=dropbox encryption=shared
git annex initremote dropbox type=external externaltype=dropbox encryption=shared folder=gitannex
git annex describe dropbox "the dropbox library"

View file

@ -1,18 +1,17 @@
# Latest version 0.1.10
Hook program for gitannex to use flickr as backend.
flickrannex
=========
This allows storing any type of file on flickr, not only images and movies.
Hook program for gitannex to use flickr as backend
# Requirements:
python2
Credit for the flickr api interface goes to: <http://stuvel.eu/flickrapi>
Credit for the png library goes to: <https://github.com/drj11/pypng>
Credit for the png tEXt patch goes to: <https://code.google.com/p/pypng/issues/detail?id=65>
Credit for the flickr api interface goes to: http://stuvel.eu/flickrapi
Credit for the png library goes to: https://github.com/drj11/pypng
Credit for the png tEXt patch goes to: https://code.google.com/p/pypng/issues/detail?id=65
# Install
Clone the git repository in your home folder.
git clone git://github.com/TobiasTheViking/flickrannex.git
@ -20,39 +19,28 @@ Clone the git repository in your home folder.
This should make a ~/flickrannex folder
# Setup
Make the file executable, and link it into PATH
Run the program once to set it up.
cd ~/flickrannex; chmod +x git-annex-remote-flickr; sudo ln -sf `pwd`/git-annex-remote-flickr /usr/local/bin/git-annex-remote-flickr
cd ~/flickrannex; python2 flickrannex.py
# Commands for gitannex:
After the setup has finished, it will print the git-annex configure lines.
USERNAME="username@provider.com" git annex initremote flickr type=external externaltype=flickr encryption=shared folder=gitannex
# Configuring git-annex
An oauth authentication link should now be launched in the default browser. The hook will wait for 30s for you to login and authenticate.
git config annex.flickr-hook '/usr/bin/python2 ~/flickrannex/flickrannex.py'
git annex initremote flickr type=hook hooktype=flickr encryption=shared
git annex describe flickr "the flickr library"
git annex describe dropbox "the flickr library"
# Notes
## Unencrypted mode
The photo name on flickr is currently the GPGHMACSHA1 version.
Run the following command in your annex directory
git annex wanted flickr uuid include=*.jpg or include=*.jpeg or include=*.gif or include=*.png
## Encrypted mode
The current version base64 encodes all the data, which results in ~35% larger filesize.
I might look into yyenc instead. I'm not sure if it will work in the tEXt field.
Run the following command in your annex directory
git annex wanted flickr exclude=largerthan=30mb
## Including directories as tags
Get get each of the directories below the top level git directory added as tags to uploads:
git config annex.flickr-hook 'GIT_TOP_LEVEL=`git rev-parse --show-toplevel` /usr/bin/python2 %s/flickrannex.py'
This feature is currently disabled, if it gets implemented again it will most likely not require user action to enable it.
In this case the image:
/home/me/annex-photos/holidays/2013/Greenland/img001.jpg

View file

@ -1,4 +1,4 @@
googledriveannex
googledriveannex 0.2.0
=========
Hook program for gitannex to use Google Drive as backend
@ -6,23 +6,27 @@ Hook program for gitannex to use Google Drive as backend
# Requirements:
python2
python-httplib2
Credit for the googledrive api interface goes to google
## Install
# Install
Clone the git repository in your home folder.
git clone git://github.com/TobiasTheViking/googledriveannex.git
This should make a ~/googledriveannex folder
## Setup
Run the program once to make an empty config file
# Setup
Make the file executable, and link it into PATH
cd ~/googledriveannex; python2 googledriveannex.py
cd ~/googledriveannex; chmod +x git-annex-remote-googledrive; sudo ln -sf `pwd`/git-annex-remote-googledrive /usr/local/bin/git-annex-remote-googledrive
## Commands for gitannex:
# Commands for gitannex:
git config annex.googledrive-hook '/usr/bin/python2 ~/googledriveannex/googledriveannex.py'
git annex initremote googledrive type=hook hooktype=googledrive encryption=shared
git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
An oauth authentication link should now be launched in the default browser. Authenticate and you will be proved with a code.
OAUTH='authentication code' git annex initremote googledrive type=external externaltype=googledrive encryption=shared folder=gitannex
git annex describe googledrive "the googledrive library"

View file

@ -1,4 +1,4 @@
imapannex
imapannex 0.2.0
=========
Hook program for gitannex to use imap as backend
@ -15,13 +15,11 @@ Clone the git repository in your home folder.
This should make a ~/imapannex folder
# Setup
Run the program once to set it up.
Make the file executable, and link it into PATH
cd ~/imapannex; python2 imapannex.py
cd ~/imapannex; chmod +x git-annex-remote-imap; sudo ln -sf `pwd`/git-annex-remote-imap /usr/local/bin/git-annex-remote-imap
# Commands for gitannex:
git config annex.imap-hook '/usr/bin/python2 ~/imapannex/imapannex.py'
git annex initremote imap type=hook hooktype=imap encryption=shared
USERNAME="username@provider.com" PASSWORD="password" git annex initremote imap type=external externaltype=imap encryption=shared folder=gitannex method="Normal password" ssl="SSL/TLS" host="imap.host.com" port="993"
git annex describe imap "the imap library"
git annex wanted imap exclude=largerthan=30mb

View file

@ -1,5 +1,7 @@
[Megaannex](https://github.com/TobiasTheViking/megaannex)
is a hook program for git-annex to use mega.co.nz as backend
megaannex 0.2.0
=========
Hook program for gitannex to use mega.co.nz as backend
# Requirements:
@ -7,35 +9,21 @@ is a hook program for git-annex to use mega.co.nz as backend
requests>=0.10
pycrypto
Credit for the mega api interface goes to:
<https://github.com/richardasaurus/mega.py>
## Install
Credit for the mega api interface goes to: https://github.com/richardasaurus/mega.py
# Install
Clone the git repository in your home folder.
git clone git://github.com/TobiasTheViking/megaannex.git
This should make a ~/megannex folder
This should make a ~/megaannex folder
## Setup
# Setup
Make the file executable, and link it into PATH
Run the program once to make an empty config file.
cd ~/megaannex; chmod +x git-annex-remote-mega; sudo ln -sf `pwd`/git-annex-remote-mega /usr/local/bin/git-annex-remote-mega
cd ~/megaannex; python2 megaannex.py
# Commands for gitannex:
Edit the megaannex.conf file. Add your mega.co.nz username, password, and folder name.
## Configuring git-annex
git config annex.mega-hook '/usr/bin/python2 ~/megaannex/megaannex.py'
git annex initremote mega type=hook hooktype=mega encryption=shared
git annex describe mega "the mega.co.nz library"
## Notes
You may need to use a different command than "python2", depending
on your python installation.
-- Tobias
USERNAME="user" PASSWORD="pword" git annex initremote mega type=external externaltype=mega encryption=shared folder=gitannex
git annex describe mega "the mega.co.nz library"

View file

@ -1,4 +1,4 @@
owncloudannex
owncloudannex 0.2.0
=========
Hook program for gitannex to use owncloud as backend
@ -6,8 +6,9 @@ Hook program for gitannex to use owncloud as backend
# Requirements:
python2
python-requests
Credit for the Owncloud api interface goes to me.
Credit for the webdav api interface goes to https://launchpad.net/python-webdav-lib
# Install
Clone the git repository in your home folder.
@ -17,12 +18,11 @@ Clone the git repository in your home folder.
This should make a ~/owncloudannex folder
# Setup
Run the program once to set it up.
Make the file executable, and link it into PATH
cd ~/owncloudannex; python2 owncloudannex.py
cd ~/owncloudannex; chmod +x git-annex-remote-owncloud; sudo ln -sf `pwd`/git-annex-remote-owncloud /usr/local/bin/git-annex-remote-owncloud
# Commands for gitannex:
git config annex.owncloud-hook '/usr/bin/python2 ~/owncloudannex/owncloudannex.py'
git annex initremote owncloud type=hook hooktype=owncloud encryption=shared
git annex describe owncloud "the owncloud library"
USERNAME="login" PASSWORD="password" git annex initremote owncloud type=external externaltype=owncloud encryption=shared folder=gitannex url="https://host/remote.php/webdav/"
git annex describe dropbox "the owncloud library"

View file

@ -1,4 +1,4 @@
skydriveannex
skydriveannex 0.2.0
=========
Hook program for gitannex to use [skydrive](http://en.wikipedia.org/wiki/SkyDrive) (previously *Windows Live SkyDrive* and *Windows Live Folders*) as backend
@ -18,12 +18,15 @@ Clone the git repository in your home folder.
This should make a ~/skydriveannex folder
# Setup
Run the program once to set it up.
Make the file executable, and link it into PATH
cd ~/skydriveannex; python2 skydriveannex.py
cd ~/skydriveannex; chmod +x git-annex-remote-skydrive; sudo ln -sf `pwd`/git-annex-remote-skydrive /usr/local/bin/git-annex-remote-skydrive
# Commands for gitannex:
git config annex.skydrive-hook '/usr/bin/python2 ~/skydriveannex/skydriveannex.py'
git annex initremote skydrive type=hook hooktype=skydrive encryption=shared
git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=gitannex
An oauth authentication link should now be launched in the default browser. Authenticate, and use the last url as OAUTH key.
OAUTH='URL after last redirect' git annex initremote skydrive type=external externaltype=skydrive encryption=shared folder=gitannex
git annex describe skydrive "the skydrive library"

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="namelessjon"
ip="193.132.159.169"
subject="inode starvation"
date="2014-01-09T15:30:16Z"
content="""
This happened to me a few times when creating new annex folders.
I use lvm to create virtual partitions and have/had several 'bulk media' logical volumes which used ext4 with -T largefile4 (i.e. one inode per 4mb) because they're storing files with 20mb+ sizes (RAW images, downloaded screencasts, FLAC soundfiles, etc). Between the extra directories git annex creates, the extra files, and the .git/object directory, I ran out of inodes on a few occasions from the profusion of small files. In some cases, I worked around this by shunting data around, or adding incrementally and then 'git gc'ing a lot to at least have a small .git/objects dir. A packed metadata would help to deal with this.
"""]]