Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2015-04-08 16:15:48 -04:00
commit a662819fa3
14 changed files with 359 additions and 0 deletions

View file

@ -0,0 +1,76 @@
### Please describe the problem.
When syncing with a remote, I get some files with "unknown backend".
Original:
$ ls -lh Pictures/2014/06/21/2014-06-21\ 13.52.34.png
lrwxrwxrwx 1 jean jean 214 Jun 21 2014 Pictures/2014/06/21/2014-06-21 13.52.34.png -> ../../../../.git/annex/objects/mx/Ff/SHA256E-s247069--d425c1049778880eb9b9fcab74a82ec86dd22882db9c026f8fc0e9cb6270d022.34.png/SHA256E-s247069--d425c1049778880eb9b9fcab74a82ec86dd22882db9c026f8fc0e9cb6270d022.34.png
This picture and symlink is fine.
Remote (broken symlink):
$ ls -lh Pictures/2014/06/21/2014-06-21\ 13.52.34.png
lrwxrwxrwx 1 jean jean 214 Apr 6 12:50 Pictures/2014/06/21/2014-06-21 13.52.34.png -> ./../../.git/annex/objects/mx/Ff/SHA256E-s247069--d425c134ac2f9b0eb9b9fcab74a82ec86dd22882db9c026f8fc0e9cb6270d022.34.pn0.png/SHE-s247069--d425c1049778880eb9b9fcab74a82ec86dd22882db9c09efe84ca9cb6270d022.34.png
In this case the paths differ: there is an extra `.pn0` component, and the final `SHA256E` has changed to `SHE`.
### What steps will reproduce the problem?
I don't know yet.
### What version of git-annex are you using? On what operating system?
$ git annex version
git-annex version: 5.20140412ubuntu1
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
Ubuntu 14.04.1 LTS
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 6
00000000-0000-0000-0000-000000000001 -- web
...
untrusted repositories: 0
transfers in progress: none
available local disk space: 168.35 gigabytes (+1 megabyte reserved)
local annex keys: 16503
local annex size: 39.15 gigabytes
annexed files in working tree:
skipping Pictures/2014/06/21/2014-06-21 13.52.34.png (unknown backend SHE)
skipping Pictures/2014/06/21/2014-06-21 13.52.45.png (unknown backend SHE)
skipping Pictures/2014/06/21/2014-06-21 13.58.03.png (unknown backend SHA256EE)
104552
size of annexed files in working tree: 124.64 gigabytes
bloom filter size: 16 mebibytes (3.3% full)
backend usage:
SHA256E: 121054
SHA256: 1
$ git annex get --from=laptop
skipping Pictures/2014/06/21/2014-06-21 13.52.34.png (unknown backend SHE)
skipping Pictures/2014/06/21/2014-06-21 13.52.45.png (unknown backend SHE)
skipping Pictures/2014/06/21/2014-06-21 13.58.03.png (unknown backend SHA256EE)
# End of transcript or log.
"""]]

View file

@ -0,0 +1,29 @@
### Please describe the problem.
The command `git annex unused --from mys3remote` shows no unused data, when I'm expecting at least one old version of a file in the current directory.
### What steps will reproduce the problem?
- Create a file, `git annex add` the file, sync with remotes and copy to an amazon S3 special remote
- Run `git annex unlock` on the file, modify it, `git annex add` the file, sync and copy to S3
- Now if you run `git annex unused` you should see the old version of the file in the local repo
- But `git annex unused --from mys3remote` shows nothing
- And I know that the old version is on S3 (at least according to the logs) because `git annex whereis --unused` shows that the file is both [here] and [mys3remote]
I also noticed that the `git annex whereis --unused` command would show nothing if it was run directly after `git annex unused --from mys3remote`, but it would correctly show the locations when run directly after `git annex unused`.
### What version of git-annex are you using? On what operating system?
git-annex version: 5.20141105-g8b19598
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
arch linux x86_64
### Please provide any additional information below.
The S3 remote is encrypted with the default "hybrid" method

View file

@ -0,0 +1,80 @@
### Please describe the problem.
I have some paths containing swedish characters (åäö ÅÄÖ).
Some paths also contains the character "@".
I use git annex assistant in autostart mode on my Documents folder.
I get the following errors in the log file:
[[!format sh """
fd:37: commitBuffer: invalid argument (invalid character)
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr<47><72><EFBFBD><EFBFBD><EFBFBD><EFBFBD>nsytem<65><6D><EFBFBD><EFBFBD><EFBFBD><EFBFBD>te/MMD_Config/SITE/LegacyNT/mmd/config/Dataintag/Fil-intag/OLDIB/SurfaceWaterTemperature.ini: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal'
fatal: Work/@Projects/archive/20140515_METOCC_Gr: pathspec magic not supported by this command: 'literal''
"""]]
I also notice a lot of zombie git processes in the ps list.
### What steps will reproduce the problem?
Upgrade to Version: 5.20150406-gb2814bc
Build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA
Pre compiled tar.gz
### What version of git-annex are you using? On what operating system?
Version: 5.20150406-gb2814bc
Arch linux
[daniel@wintermute Documents]$ uname -a
Linux wintermute 3.19.2-1-ARCH #1 SMP PREEMPT Wed Mar 18 16:21:02 CET 2015 x86_64 GNU/Linux
### Please provide any additional information below.
If I downgrade to a previous version ie. git-annex.linux.5.20150317 the described problem is gone.
I tried to remove the character "@" from my paths, but it didn't help.
Looking at the log file below, it doesn't seem to have anything to do with my swedish characters.
[[!format sh """
[2015-04-07 22:24:14 CEST] main: starting assistant version 5.20150406-gb2814bc
[2015-04-07 22:24:14 CEST] TransferScanner: Syncing with xxxxxxx.xxxxxxx.xxx_wintermute_Documents
p11-kit: couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so: cannot open shared object file: No such file or directory
(scanning...) [2015-04-07 22:24:15 CEST] Watcher: Performing startup scan
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/firewall-udp-1194-dasu-Viscosity.visc.zip: pathspec magic not supported by this command: 'literal'
git cat-file EOF: user error
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
fatal: Work/archive/Tidrapportering/Tid.md: pathspec magic not supported by this command: 'literal'
"""]]

View file

@ -0,0 +1,7 @@
Hi,
I have set up git-annex to sync a folder located on laptop A, USB Key, and laptop B. After reading the walkthrough, I don't quite understand the proper procedure to keep in sync. Should I run git-annex sync from within the sync directory on the USB Key on laptop A and then from within the sync folder on the USB Key when plugged into laptop B? Or do I have to run git-annex sync from within the sync directory on the laptops, then the USB Key? And/or do I have to run git-annex get on the sync directory on the USB key before syncing to the second laptop?
Also I should not normally be making changes on laptops A and B at the same time, but if that were the case, is there any difference in what I would run to sync the changes?
Thanks!

View file

@ -0,0 +1,31 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkutSE8_3fFAETmO_E598zja4gKwYXbb8E"
nickname="Сергей"
subject="comment 1"
date="2015-04-08T12:31:33Z"
content="""
Well, working with git annex, you should think of two things:
1. location of annex information (git commit tree), let's call it INFO
2. location of files contents, let's call it CONTENT
So, for your sneekernet scenario these general steps are required:
1. sync INFO between Laptop A and USB
2. transfer required CONTENT from LAPTOP A to USB
3. sync INFO between Laptop B and USB
4. transfer required CONTENT from USB to Laptop B
Let's describe your options on every step:
1. issue `git annex sync` on Laptop A. Either from within USB or Laptop repo, I prefer Laptop because of automatic commit.
2.
* issue `git annex copy <files> --to=<usb-remote-name>` from within Laptop A repo
* issue `git annex get <files>` from within USB repo. This is not possible if you have bare repo on USB.
3. issue `git annex sync` on Laptop B
4. same as 2, but transfer to Laptop B
Also, using `git annex sync --content`, you don't need to manually transfer CONTENT, see [--content](https://git-annex.branchable.com/git-annex-sync/). But in this case you somewhat loose control of what is transferred.
If you make changes on both laptops at the same time no extra steps required. In case of conflict both versions saved in working directory, see [automatic conflict resolution](http://git-annex.branchable.com/automatic_conflict_resolution/).
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawk_STgv_0fB-y1x_fgzUI10Bg-aajJyMyc"
nickname="猫"
subject="comment 3"
date="2015-04-08T06:30:06Z"
content="""
I'll take a closer look at myrepos, but it doesn't seem to do location tracking, which is the main feature of git-annex that I'm looking for.
"""]]

View file

@ -0,0 +1,22 @@
How do we get into the following situation?
First we complain about an ignored file (why not just ignore it?),
and then apparently we delete the ignored file.
[...]
import Pictures/2005/11/16/.IMG_0819.tmpwrite.JPG (duplicate) ok
(Recording state in git...)
The following paths are ignored by one of your .gitignore files:
Pictures/2008/11/27/.img_1315.tmpwrite.jpg
Use -f if you really want to add them.
fatal: no files added
git-annex: user error (xargs ["-0","git","--git-dir=/.../annex/.git","--work-tree=/.../annex","add","--"] exited 123)
# eek, the file that we complained about has vanished!
$ rm ../Pictures/2008/12/27/.img_1315.tmpwrite.jpg
rm: cannot remove ../Pictures/2008/11/27/.img_1315.tmpwrite.jpg: No such file or directory
Expected:
- leave ignored files untouched. Maybe report "Skipped ignored files."
Actual:
- Stop import, but delete the ignored file as side effect.

View file

@ -0,0 +1,35 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnVnsqEy82M-MuS2gLri-az83wSQ6lXSrc"
nickname="Jean"
subject="Ignored file imported, in spite of error message"
date="2015-04-08T09:08:53Z"
content="""
Today I'm seeing this:
git-annex: user error (xargs [\"-0\",\"git\",\"--git-dir=/.../annex/.git\",\"--work-tree=/.../annex\",\"add\",\"--\"] exited 123)
failed
(Recording state in git...)
The following paths are ignored by one of your .gitignore files:
btsync/DCIM (1)/Camera 1/20150101090842.jpg.tmp
Use -f if you really want to add them.
fatal: no files added
git-annex: user error (xargs [\"-0\",\"git\",\"--git-dir=/.../annex/.git\",\"--work-tree=/.../annex\",\"add\",\"--\"] exited 123)
failed
(Recording state in git...)
^C
This repeats until I kill the import.
Subsequently I see that the ignored file was in fact imported:
$ ls -lh \"btsync/DCIM (1)/Camera 1/20150101090842.jpg.tmp\"
lrwxrwxrwx 1 user user 203 Jan 11 14:11 btsync/DCIM (1)/Camera 1/20150101090842.jpg.tmp -> ../../../.git/annex/objects/0K/GX/SHA256E-s0--e3b0c44298ec1c149aebe4c8996eb92427ae41e4649b934ca495991b7852b855.jpg.tmp/SHA256E-s0--e3b0c44298ec1c149aebe4c8996eb92427ae41e4649b934ca495991b7852b855.jpg.tmp
In this case the original file was not deleted, because I used `import --duplicate`:
$ ls -lh \"../btsync/DCIM (1)/Camera 1/20150101090842.jpg.tmp\"
-rw-rw-r-- 1 user user 0 Jan 11 14:11 ../btsync/DCIM (1)/Camera 1/20150101090842.jpg.tmp
"""]]

View file

@ -0,0 +1,7 @@
What does the following mean? I would expect that if we encounter a duplicate while using the `--deduplicate` option, the duplicate file would be removed from the import location and import would continue. Does that mean that the newly encountered `img_0405.jpg` is different from the one previously seen?
Could the error message perhaps be improved?
.../usbdisk/annex(master)$ git annex import --deduplicate ../Pictures/
import Pictures/2008/10/12/img_0405.jpg git-annex: not overwriting existing Pictures/2008/10/12/img_0405.jpg (use --force to override)
Many other duplicates are being tidied away.

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkutSE8_3fFAETmO_E598zja4gKwYXbb8E"
nickname="Сергей"
subject="Slow on windows but not on linux"
date="2015-04-07T18:20:23Z"
content="""
If this is by design and `git annex status` should take more time in direct mode then what I'm experiencing is strange. On windows every 100M file adds approximately 1 second to `status` duration (on my laptop), but on linux it does not. On linux `git annex status` even in direct mode takes milliseconds. What is wrong with my setup?
"""]]

View file

@ -0,0 +1,32 @@
We have the correct symlink:
$ ls -lh Sita_Sings_the_Blues_480p_2150kbps.mp4
lrwxrwxrwx 1 user user 204 Apr 8 20:22 Sita_Sings_the_Blues_480p_2150kbps.mp4 -> .git/annex/objects/6q/Wz/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4
The object is in place:
$ ls -lh .git/annex/objects/6q/Wz/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4
-r-------- 1 user user 1,4G Dec 28 2010 .git/annex/objects/6q/Wz/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4
Now I move the symlink, which becomes broken:
$ mv Sita_Sings_the_Blues_480p_2150kbps.mp4 Videos/
I try to fix the symlink:
$ git annex sync
$ git annex fix Videos/Sita_Sings_the_Blues_480p_2150kbps.mp4
But it stays broken:
$ ls -lh Videos/Sita_Sings_the_Blues_480p_2150kbps.mp4
lrwxrwxrwx 1 user user 204 Apr 8 20:22 Videos/Sita_Sings_the_Blues_480p_2150kbps.mp4 -> .git/annex/objects/6q/Wz/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4/SHA256E-s1463013630--2a18a317a536d8e2d28e7916a45a007679b7192102a71067de488c9faa6aab45.mp4
Ah, when I try to `git mv Sita_Sings_the_Blues_480p_2150kbps.mp4 Videos/`,
I am warned that the link is not under version control. How did the symlink get
made in the first place, if not by git-annex? It should be known, right?
After I `git annex add`, I can `git mv` and then `git annex fix`.
How did I get into the state where I have a symlink pointing at an imported
object, which is not under version control?

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="CandyAngel"
subject="comment 1"
date="2015-04-08T14:29:59Z"
content="""
'git annex fix' will only fix symlinks that git is tracking (either staged, or part of the repository) because git-annex uses git-ls-files (or something like that) to get the filenames to fix.
'git annex add' adds the files to the annex and replaces them with symlinks, stages the symlinks but does not commit the symlinks.
"""]]

View file

@ -0,0 +1,7 @@
[[!comment format=mdwn
username="CandyAngel"
subject="comment 2"
date="2015-04-08T14:32:56Z"
content="""
It's also funny that this has cropped up [just after I asked for git-annex to fix untracked symlinks](https://git-annex.branchable.com/todo/__34__git-annex_fix__34___on_untracked__44___but_git-annexy_symlinks/) :P
"""]]

View file

@ -53,3 +53,11 @@ However, instead of just a list of links we will output a markdown-formatted lis
end
Very useful.
## Sharing links with time-limited URLs
By using pre-signed URLs it is possible to create limits on how long a URL is valid for retrieving an object.
To enable use a private S3 bucket for the remotes and then pre-sign actual URL with the script in [AWS-Tools](https://github.com/gdbtek/aws-tools).
Example:
key=`git annex lookupkey "$fname"`; sign_s3_url.bash --region 'eu-west-1' --bucket 'mybuck' --file-path $key --aws-access-key-id XX --aws-secret-access-key XX --method 'GET' --minute-expire 10