Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
Joey Hess
commit
d95855b676
14 changed files with 427 additions and 35 deletions
30
doc/bugs/Drop_files_with_the_same_checksum..mdwn
Normal file
30
doc/bugs/Drop_files_with_the_same_checksum..mdwn
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
### Please describe the problem.
|
||||
When two identical files are annexed and one of them is dropped, both files are gone (one dangling symlink is left). This may be intentional (the checksums are the same after all), but then is there a way to drop one of the files?
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
mkdir annex
|
||||
cd annex
|
||||
git init
|
||||
git annex init
|
||||
mkdir a b
|
||||
dd if=/dev/urandom of=a/data.bin count=2048
|
||||
cp a/data.bin b
|
||||
git annex add a/data.bin b/data.bin
|
||||
git commit -m "Added raw data."
|
||||
git annex drop --force a/data.bin
|
||||
file b/data.bin
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
git-annex version: 5.20140831+b1
|
||||
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
|
||||
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository version: 5
|
||||
|
||||
Distributor ID: Debian
|
||||
Description: Debian GNU/Linux testing (jessie)
|
||||
Release: testing
|
||||
Codename: jessie
|
||||
24
doc/bugs/annex_get_fails_from_read-only_filesystem.mdwn
Normal file
24
doc/bugs/annex_get_fails_from_read-only_filesystem.mdwn
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
### Please describe the problem.
|
||||
|
||||
annex get does not work from read-only file systems...
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
$ git annex get --from=...
|
||||
error: could not lock config file /.../Annex/.git/config: Read-only file system
|
||||
get ... (from ...) error: could not lock config file .../Annex/.git/config: Read-only file system
|
||||
git [Param "config",Param "annex.version",Param "5"] failed
|
||||
failed
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
annex.version = 3 in the remote
|
||||
|
||||
$ git annex version
|
||||
git-annex version: 5.20140927
|
||||
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
|
||||
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository version: 5
|
||||
upgrade supported from repository versions: 0 1 2 4
|
||||
23
doc/bugs/git_clone_ignores_annex.mdwn
Normal file
23
doc/bugs/git_clone_ignores_annex.mdwn
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
### Please describe the problem.
|
||||
|
||||
More of a feature request than a bug. It would be nice if when creating a local clone with git clone this would run automatically:
|
||||
|
||||
ln -s ../../annex/.git/annex .git/annex
|
||||
|
||||
to hook up the annex. Just a minor thing, but I'd be nice.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawnjX_O_VChwfDjcKlcRHbc2Aah8aYQlqts"
|
||||
nickname="Ludovic"
|
||||
subject="comment 3"
|
||||
date="2014-10-05T11:01:39Z"
|
||||
content="""
|
||||
I have the same problem.
|
||||
I checked again, removed the previous annex/ directory and created a new annex repository from the webapp. No annex/.git/index file is created.
|
||||
|
||||
I then add a remore ssh server. The synchronisation fails with the error in the log: \"/storage/emulated/legacy/annex/.git/index: copyFile: does not exist (No such file or directory).\"
|
||||
|
||||
Running \"git annex sync\" solved the problem.
|
||||
"""]]
|
||||
7
doc/forum/Broken_symlinks_remain_after_drop.mdwn
Normal file
7
doc/forum/Broken_symlinks_remain_after_drop.mdwn
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
This is a newb question. I don't know whether this is a bug or the way git-annex is intended to function.
|
||||
|
||||
I have two annex repos connected to each other. My idea was to have the first repository add files, which would then be moved to the second repository for storage. After moving, repo1 would be empty again, empty and clean of any symlinks.
|
||||
|
||||
But after I 'git-annex move * --to repo2' broken symlinks remain in repo1. I don't want any broken/unused symlinks to remain in repo1 for object data it doesn't currently have (even if those files remain in the repository itself).
|
||||
|
||||
Is there a way I can clean/remove broken symlinks to object data when those objects aren't present, so the directory only contains symlinks when the repo currently has the object data for those files?
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="tdussa"
|
||||
ip="217.84.78.25"
|
||||
subject="tar -- not really"
|
||||
date="2014-10-03T20:40:45Z"
|
||||
content="""
|
||||
Hi,
|
||||
|
||||
THX for your suggestion. Unfortunately, git bundle is able to carve out particular slices of a repo, which a simple tar obviously cannot do. This functionality is much desired.
|
||||
|
||||
Cheers,
|
||||
Toby.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawldUCypSR21BObbzC0Uf8NVd1vZnjYlXZc"
|
||||
nickname="Adam"
|
||||
subject="Ok, found it :-)"
|
||||
date="2014-10-04T10:11:55Z"
|
||||
content="""
|
||||
I think I found reliable although a bit excessive way to get back to clean state:
|
||||
|
||||
1. stop all the running daemons
|
||||
2. on android, remove folder git-annex.home
|
||||
3. on linux, remove ~/.config/git-annex and ~/.ssh/git-annex
|
||||
4. on both attempted repositories I removed .git folder
|
||||
5. I removed everything I have stored on rsync.net
|
||||
|
||||
Then I re-did the steps on good wifi network an plugged in to power source on both devices :-)
|
||||
It seems that important step is to wait for the annex to share the remote repo with the other device (took ~5 minutes)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawllyDAwjTPuM6G2d3eKE481V0qGXRKtF8s"
|
||||
nickname="Pieter"
|
||||
subject="Syncing metadata only with Synology NAS and git annex"
|
||||
date="2014-10-06T12:03:35Z"
|
||||
content="""
|
||||
I am not able to get full syncing working. It says \"syncing enabled (metadata only)\".
|
||||
When I click on Actions -> Edit it says:
|
||||
|
||||
Just a git repository.
|
||||
|
||||
This repository is not currently set up as a git annex; only git metadata is synced with this repository.
|
||||
|
||||
If this repository's ssh server has git-annex installed, you can upgrade this repository to a full git annex, which will store the contents of your files, not only their metadata.
|
||||
|
||||
When I try to upgrade the repository it does not work. The log says:
|
||||
|
||||
sh: git-annex-shell: not found
|
||||
|
||||
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
|
||||
|
||||
rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.1]
|
||||
|
||||
I'm using Version: 5.20140717 on the Linux Ubuntu 14.10.
|
||||
|
||||
Ssh'ing from Ubuntu to gituser@synology works fine and shows the git-annex-shell options
|
||||
|
||||
Any ideas?
|
||||
|
||||
"""]]
|
||||
|
|
@ -20,7 +20,7 @@ Or just show the files that haven't been played yet:
|
|||
|
||||
git annex view playCount=0
|
||||
|
||||
Use `git checkout master` to reset the view. Note that the above will flatten the tree hierarchy, which you may not way. Try this in that case:
|
||||
Use `git checkout master` to reset the view. Note that the above will flatten the tree hierarchy, which you may not want. Try this in that case:
|
||||
|
||||
git annex view playCount=0 films/=*
|
||||
|
||||
|
|
|
|||
|
|
@ -1,45 +1,227 @@
|
|||
#! /usr/bin/perl -w
|
||||
|
||||
# we want to operate on relative links, so set this to the common prefix
|
||||
# to the git annex repo
|
||||
my $prefix="/home/media/video/";
|
||||
# this is the directory for the XBMC database
|
||||
my $path = '/home/video/.xbmc/userdata/Database/';
|
||||
use Getopt::Long;
|
||||
use Pod::Usage;
|
||||
|
||||
# no user-serviceable parts below
|
||||
my $help = 0;
|
||||
my $usage = 0;
|
||||
my $dryrun = 0;
|
||||
my $verbose = 0;
|
||||
my $path = '';
|
||||
my $annex = '';
|
||||
my $home = $ENV{'HOME'};
|
||||
|
||||
sub main() {
|
||||
checkargs();
|
||||
if (!$path) {
|
||||
$path = $home . '/.xbmc/userdata/Database';
|
||||
}
|
||||
print("# checking XBMC directory '$path'\n") if ($verbose);
|
||||
$dbpath = finddb($path);
|
||||
if (!$dbpath) {
|
||||
pod2usage("$0: can't find a XBMC database in '$path'.");
|
||||
}
|
||||
print("# using database '$dbpath'\n") if ($verbose);
|
||||
checkdb();
|
||||
}
|
||||
|
||||
# list videos database, find the latest one
|
||||
# modified version of
|
||||
# http://stackoverflow.com/questions/4651092/getting-the-list-of-files-sorted-by-modification-date-in-perl
|
||||
opendir my($dirh), $path or die "can't opendir $path: $!";
|
||||
my @flist = sort { -M $a <=> -M $b } # Sort by modification time
|
||||
map { "$path/$_" } # We need full paths for sorting
|
||||
grep { /^MyVideos.*\.db$/ }
|
||||
readdir $dirh;
|
||||
closedir $dirh;
|
||||
|
||||
my $dbpath=$flist[0];
|
||||
|
||||
my @lines = `echo 'SELECT playCount, path.strPath, files.strFileName FROM movie JOIN files ON files.idFile=movie.idFile JOIN path ON path.idPath=files.idPath;' | sqlite3 $dbpath`;
|
||||
for (@lines) {
|
||||
my ($count, $dir, $file) = split /\|/;
|
||||
chomp $file;
|
||||
# empty or non-numeric count is zero
|
||||
if ($count !~ /[0-9]/) {
|
||||
$count = 0;
|
||||
}
|
||||
if ($file =~ s#stack://##) {
|
||||
for (split /,/, $file) {
|
||||
s/$prefix//;
|
||||
s/^ //;
|
||||
s/ $//;
|
||||
my @cmd = (qw(git annex metadata --set), "playCount=$count", $_);
|
||||
system(@cmd);
|
||||
}
|
||||
sub finddb($) {
|
||||
my $path = shift(@_);
|
||||
opendir my($dirh), $path or die "can't opendir $path: $!";
|
||||
my @flist = sort { -M $a <=> -M $b } # Sort by modification time
|
||||
map { "$path/$_" } # We need full paths for sorting
|
||||
grep { /^MyVideos.*\.db$/ }
|
||||
readdir $dirh;
|
||||
closedir $dirh;
|
||||
if ($#flist > 0) {
|
||||
return $flist[0];
|
||||
}
|
||||
else {
|
||||
$dir =~ s/$prefix//;
|
||||
my @cmd = (qw(git annex metadata --set), "playCount=$count", "$dir$file");
|
||||
system(@cmd);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
sub checkargs() {
|
||||
pod2usage(1) if $help;
|
||||
pod2usage(-exitval => 0, -verbose => 2) if $usage;
|
||||
|
||||
GetOptions('h|?' => \$help,
|
||||
'help|usage' => \$usage,
|
||||
# we want to operate on relative links, so set this to
|
||||
# the common annex to the git annex repo
|
||||
'annex=s' => \$annex,
|
||||
'path=s' => \$path,
|
||||
'home=s' => \$home,
|
||||
'dryrun|n' => \$dryrun,
|
||||
'verbose|v' => \$verbose,
|
||||
)
|
||||
or die("Error parsing commandline\n");
|
||||
}
|
||||
|
||||
sub checkdb() {
|
||||
my @lines = `echo 'SELECT playCount, path.strPath, files.strFileName FROM movie JOIN files ON files.idFile=movie.idFile JOIN path ON path.idPath=files.idPath;' | sqlite3 $dbpath`;
|
||||
print "# finding files...\n" if $verbose;
|
||||
for (@lines) {
|
||||
my ($count, $dir, $file) = split /\|/;
|
||||
chomp $file;
|
||||
# empty or non-numeric count is zero
|
||||
if ($count !~ /[0-9]/) {
|
||||
$count = 0;
|
||||
}
|
||||
print "# $dir/$file\n" if $verbose;
|
||||
if ($file =~ s#stack://##) {
|
||||
for (split /,/, $file) {
|
||||
s/$annex//;
|
||||
s/^ //;
|
||||
s/ $//;
|
||||
my @cmd = (qw(git annex metadata --set), "playCount=$count", $_);
|
||||
if ($dryrun) {
|
||||
print join(' ', @cmd) . "\n";
|
||||
}
|
||||
else {
|
||||
system(@cmd);
|
||||
}
|
||||
}
|
||||
}
|
||||
else {
|
||||
$dir =~ s/$annex//;
|
||||
my @cmd = (qw(git annex metadata --set), "playCount=$count", "$dir$file");
|
||||
if ($dryrun) {
|
||||
print join(' ', @cmd) . "\n";
|
||||
}
|
||||
else {
|
||||
system(@cmd);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
|
||||
__END__
|
||||
=encoding utf8
|
||||
|
||||
=head1 NAME
|
||||
|
||||
git-annex-xbmc-playcount - register XBMC playcounts as git-annex metadata
|
||||
|
||||
=head1 SYNOPSIS
|
||||
|
||||
git-annex-xbmc-playcount [--path .xbmc/userdata/Database]
|
||||
|
||||
Options:
|
||||
-h short usage
|
||||
--help complete help
|
||||
--dryrun, -n do nothing and show the commands that would be ran
|
||||
--annex path to the git-annex repo
|
||||
--home the home directory where the .xbmc directory is located
|
||||
--path the location of the Database directory of XBMC, overrides --home
|
||||
--verbose show interaction details with the database
|
||||
|
||||
=head1 DESCRIPTION
|
||||
|
||||
This program will look into the XBMC database for the "playcount"
|
||||
field to register that number as metadata in the git-annex repository.
|
||||
|
||||
=head1 OPTIONS
|
||||
|
||||
=over 8
|
||||
|
||||
=item B<--dryrun>
|
||||
|
||||
Do nothing but show all the steps that would be ran. The output can be
|
||||
piped through a POSIX shell after inspection. B<-n> is an alias of
|
||||
this command. Example:
|
||||
|
||||
git-annex-xbmc-playcount -n | tee runme
|
||||
# inspect the output
|
||||
sh < runme
|
||||
|
||||
=item B<--annex>
|
||||
|
||||
This option allows the user to specify the root of the git-annex
|
||||
repository, which is then stripped off the paths found in the XBMC
|
||||
database.
|
||||
|
||||
=item B<--home>
|
||||
|
||||
Home of the user running XBMC. If not specified, defaults to the $HOME
|
||||
environment variables. The script will look into
|
||||
B<$home/.xbmc/userdata/Database> for a file matching
|
||||
B<^MyVideos.*\.db$> and will fail if none is found.
|
||||
|
||||
=item B<--path>
|
||||
|
||||
Manually specify the path to B<.xbmc/userdata/Database>. This
|
||||
overrides B<--home>.
|
||||
|
||||
Note that this doesn't point directly to the datbase itself, because
|
||||
there are usually many database files and we want to automatically
|
||||
find the latest. This may be a stupid limitation.
|
||||
|
||||
=item B<--verbose>
|
||||
|
||||
Show more information about path discovery. Doesn't obstruct
|
||||
B<--dryrun> output because lines are prefixed with C<#>.
|
||||
|
||||
=back
|
||||
|
||||
=head1 EXAMPLES
|
||||
|
||||
You have a git annex in B</srv/video> and XBMC is ran as the
|
||||
B<video> user and you want to be cautious:
|
||||
|
||||
$ ./git-annex-xbmc-playcount.pl --home /home/video/ -n --annex /srv/video/ | tee set-metadata
|
||||
git annex metadata --set playCount=0 films/Animal.Farm.1954.DVDRip.DivX-MDX.avi
|
||||
|
||||
This looks about right, set the metadata:
|
||||
|
||||
$ git annex metadata --set playCount=0 films/Animal.Farm.1954.DVDRip.DivX-MDX.avi
|
||||
metadata films/Animal.Farm.1954.DVDRip.DivX-MDX.avi
|
||||
lastchanged=2014-10-04@22-17-42
|
||||
playCount=0
|
||||
playCount-lastchanged=2014-10-04@22-17-42
|
||||
ok
|
||||
(Recording state in git...)
|
||||
|
||||
=head1 ENVIRONMENT
|
||||
|
||||
B<$HOME> is looked into to find the B<.xbmc> home directory if none of
|
||||
B<--home> or B<--path> is specified.
|
||||
|
||||
=head1 FILES
|
||||
|
||||
=over 8
|
||||
|
||||
=item B<$HOME/.xbmc/userdata/Database/MyVideos.*\.db>
|
||||
|
||||
This is where we assume the SQLite database of videos XBMC uses is
|
||||
stored.
|
||||
|
||||
=back
|
||||
|
||||
=head1 BUGS
|
||||
|
||||
If there are pipes (C<|>) in filenames, the script may fail to find
|
||||
the files properly. We would need to rewrite the database code to use
|
||||
B<DBD::SQLite>(3pm) instead of a pipe to B<sqlite3>(1).
|
||||
|
||||
=head1 LIMITATIONS
|
||||
|
||||
It took longer writing this help than writing the stupid script.
|
||||
|
||||
The script will not tag files not yet detected by XBMC.
|
||||
|
||||
The script is not incremental, so it will repeatedly add the same
|
||||
counts to files it has already found.
|
||||
|
||||
=head1 SEE ALSO
|
||||
|
||||
B<git-annex>(1), B<xbmc>(1)
|
||||
|
||||
=head1 AUTHOR
|
||||
|
||||
Written by Antoine Beaupré <anarcat@debian.org>
|
||||
|
|
|
|||
|
|
@ -0,0 +1,29 @@
|
|||
[[!comment format=mdwn
|
||||
username="hugo"
|
||||
ip="37.160.8.232"
|
||||
subject="'content-length' error"
|
||||
date="2014-10-05T19:40:23Z"
|
||||
content="""
|
||||
I moved a big PDF to Google Drive (with shared encryption).
|
||||
|
||||
Now, when I try to get it again:
|
||||
|
||||
get Documents/Guyau - The Non-Religion of the Future, nonreligionoffut00guyarich.pdf (from googledrive...) (gpg)
|
||||
Traceback (most recent call last):
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 411, in <module>
|
||||
common.startRemote()
|
||||
File \"/usr/share/googledriveannex-git/lib/CommonFunctions.py\", line 555, in startRemote
|
||||
sys.modules[\"__main__\"].transfer(line)
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 372, in transfer
|
||||
if getFile(line[2], \" \".join(line[3:]), folder):
|
||||
File \"/usr/bin/git-annex-remote-googledrive\", line 257, in getFile
|
||||
ret = common.fetchPage({\"link\": download_url, \"headers\": [(\"Authorization\", \"Bearer \" + credentials.access_token)], \"progress\": \"true\"})
|
||||
File \"/usr/share/googledriveannex-git/lib/CommonFunctions.py\", line 207, in fetchPage
|
||||
totalsize = int(con.headers['content-length'])
|
||||
File \"/usr/lib/python2.7/rfc822.py\", line 388, in __getitem__
|
||||
return self.dict[name.lower()]
|
||||
KeyError: 'content-length'
|
||||
|
||||
It works for smaller files. Is there a limit on the file size?
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="hugo"
|
||||
ip="37.160.8.232"
|
||||
subject="Re: 'content-length' error "
|
||||
date="2014-10-05T19:43:04Z"
|
||||
content="""
|
||||
The Google Drive interface tells me:
|
||||
|
||||
Size 29,776,826 bytes
|
||||
"""]]
|
||||
|
|
@ -5,3 +5,5 @@ I would expect it to at least be able to leech the files off of it.
|
|||
Otherwise, I would welcome advice on how to fix this problem without doing a `sudo chown -R` every time i plug this drive somewhere ... --[[anarcat]]
|
||||
|
||||
> Workaround: `sudo setfacl -R -m u:anarcat:rwx /media/foo/annex`
|
||||
|
||||
Note: this seems like there was at least one dupe opened about this in [[bugs/annex_get_fails_from_read-only_filesystem]].
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://id.koumbit.net/anarcat"
|
||||
ip="72.0.72.144"
|
||||
subject="this also affects ssh remotes"
|
||||
date="2014-10-05T15:26:52Z"
|
||||
content="""
|
||||
so i tried another experiment today: i tried to allow access to a remote user to my /srv/foo annex. the annex is writable by me, but not by the user, yet i was expecting the user to be able to sync with it. not push, mind you, but at least pull: in a \"git-only\" scenario, that would be perfectly possible. yet the assistant freaks out because it can't run git-annex on the repo because of a write failure on some pack files, and downgrades the repository to a \"git-only\" repo, which is also inaccurate: it can't sync the metadata either...
|
||||
|
||||
i would have expected this repository to be marked as \"readonly\" and the user be capable of fetching new changes automatically from the \"central repo\".
|
||||
|
||||
maybe that's the essence of the todo here... --[[anarcat]]
|
||||
"""]]
|
||||
Loading…
Add table
Reference in a new issue