Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2013-08-02 11:36:39 -04:00
commit 022c3910e9
8 changed files with 407 additions and 0 deletions

View file

@ -0,0 +1,95 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 3"
date="2013-08-02T06:57:12Z"
content="""
In this particular case
* It's XFS
* It's a normal repo
* That the inode can have only one shared mtime is obvious, but the symlinks could have their own:
Random line to make MarkDown happy
richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
File: foo -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268638577 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:51:35.976674954 +0200
Modify: 2013-07-30 01:19:41.000000000 +0200
Change: 2013-07-30 01:41:43.700701516 +0200
Birth: -
File: foobar -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268774374 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:51:35.976674954 +0200
Modify: 2013-07-30 01:19:41.000000000 +0200
Change: 2013-07-30 01:41:43.700701516 +0200
Birth: -
File: .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 804h/2052d Inode: 73750 Links: 1
Access: (0444/-r--r--r--) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-07-30 01:20:11.915507749 +0200
Modify: 2013-07-30 01:19:41.131509226 +0200
Change: 2013-07-30 01:20:11.915507749 +0200
Birth: -
richih@eudyptes (git)-[master] ~/killme/target % touch foo
richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
File: foo -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268638577 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:51:35.976674954 +0200
Modify: 2013-07-30 01:19:41.000000000 +0200
Change: 2013-07-30 01:41:43.700701516 +0200
Birth: -
File: foobar -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268774374 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:51:35.976674954 +0200
Modify: 2013-07-30 01:19:41.000000000 +0200
Change: 2013-07-30 01:41:43.700701516 +0200
Birth: -
File: .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 804h/2052d Inode: 73750 Links: 1
Access: (0444/-r--r--r--) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:54:34.464668766 +0200
Modify: 2013-08-02 08:54:34.464668766 +0200
Change: 2013-08-02 08:54:34.464668766 +0200
Birth: -
richih@eudyptes (git)-[master] ~/killme/target % touch --no-dereference foo
richih@eudyptes (git)-[master] ~/killme/target % stat * .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
File: foo -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268638577 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:55:07.796667610 +0200
Modify: 2013-08-02 08:55:07.796667610 +0200
Change: 2013-08-02 08:55:07.796667610 +0200
Birth: -
File: foobar -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 178 Blocks: 8 IO Block: 4096 symbolic link
Device: 804h/2052d Inode: 268774374 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:51:35.976674954 +0200
Modify: 2013-07-30 01:19:41.000000000 +0200
Change: 2013-07-30 01:41:43.700701516 +0200
Birth: -
File: .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 804h/2052d Inode: 73750 Links: 1
Access: (0444/-r--r--r--) Uid: ( 1000/ richih) Gid: ( 1000/ richih)
Access: 2013-08-02 08:54:34.464668766 +0200
Modify: 2013-08-02 08:54:34.464668766 +0200
Change: 2013-08-02 08:54:34.464668766 +0200
Birth: -
richih@eudyptes (git)-[master] ~/killme/target %
As to where the initial time of 01:19:41 came from: Yes, those were earlier tests with other imports of empty files.
"""]]

View file

@ -0,0 +1,30 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg"
nickname="Bence"
subject="checkpresent again"
date="2013-08-01T23:18:38Z"
content="""
In the current [HEAD](https://github.com/joeyh/git-annex/commit/bb74db6ef094324062adcf26a677113ee6fd0e58) the \"checkpresent\" method in [Hook.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Hook.hs#L145) is missing a \"return\" while other hooks have a return value eg. [Directory.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Directory.hs#L241), [Rsync.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Rsync.hs#L272), ...
I noticed that if my *checkpresent* hook does not output anything (to be exact: I commented out all the lines in the block) it behaves strangely. I copied the test file to \"copyrepo\", removed it by hand (so git-annex does not know about the change) and executed a fsck.
~/annex5/test1/123 $ git annex fsck --from copyrepo --fast
fsck somefile.1 (checking copyrepo...) (fixing location log)
** Based on the location log, somefile.1
** was expected to be present, but its content is missing.
failed
running the check again
~/annex5/test1/123 $ ga fsck --from copyrepo --fast
fsck somefile.1 (checking copyrepo...) ok
and running the check again without --fast
~/annex5/test1/123 $ ga fsck --from copyrepo
fsck somefile.1 (checking copyrepo...) ok
It thinks, the file is in the repo but it is not.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="4.152.108.145"
subject="comment 9"
date="2013-08-01T23:51:48Z"
content="""
The behavior you show with `fsck --from` is that the first time it's run against the damaged remote it notices the file is not present using the checkpresent hook. It then updates the location log. The subsequent times it's run, it sees that the location log says the file is not present in the remote. It verifies this is the case by calling the checkpresent hook. Since the two data sources agree, and numcopies is still satisfied, it prints \"ok\". There does not seem to be a bug here.
(`return` in Haskell does not do what you would expect to happen in a traditional imperative language. It does not alter control flow, and any function using `return` can be mechanically converted to one that does not use `return`.)
"""]]

View file

@ -0,0 +1,46 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
nickname="Nigel"
subject="git annex get failed"
date="2013-08-02T14:29:30Z"
content="""
Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex.
I copied one file to Flickr and dropped it elsewhere (--force). I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote.
Then I find I can't get the file back, even though there is a record of it from whereis.
Can you help enlighten me as to what am I missing? I assumed whereis would only report files that exist and can be copied back. If not my error, I can raise bug or search for logs. Thanks in advance for any help.
[[!format perl \"\"\"
nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis
whereis libpeerconnection.log (3 copies)
31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
whereis test.cgi (1 copy)
31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
ok
whereis walkthrough.sh (3 copies)
31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
whereis walkthrough.sh~ (3 copies)
31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library)
3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop)
ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive
ok
nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi
get test.cgi (from flickr...)
git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory)
failed
git-annex: get: 1 failed
nrb@nrb-ThinkPad-T61:~/tmp$
\"\"\"]]
"""]]

View file

@ -0,0 +1,58 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
nickname="Nigel"
subject="re: git annex get failed"
date="2013-08-02T15:02:14Z"
content="""
Another try - this time a slightly simpler setup using my version of the walkthrough commands
[[!format bash \"\"\"
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive
drop usbdrive walkthrough.sh ok
(Recording state in git...)
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr
move walkthrough.sh (gpg) (checking flickr...) (to flickr...)
/home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
/home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
ok
(Recording state in git...)
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis
whereis walkthrough.sh (1 copy)
161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
ok
whereis walkthrough.sh~ (3 copies)
161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop)
d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive)
ok
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh
get walkthrough.sh (from flickr...) (gpg)
git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory)
gpg: decrypt_message failed: eof
Unable to access these remotes: flickr
Try making some of these repositories available:
161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library)
failed
git-annex: get: 1 failed
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr
fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log)
** Based on the location log, walkthrough.sh
** was expected to be present, but its content is missing.
** No known copies exist of walkthrough.sh
failed
fsck walkthrough.sh~ (checking flickr...) (fixing location log)
** Based on the location log, walkthrough.sh~
** was expected to be present, but its content is missing.
failed
(Recording state in git...)
git-annex: fsck: 2 failed
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$
\"\"\" ]]
"""]]

View file

@ -0,0 +1,130 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y"
nickname="Nigel"
subject="re: git annex get failed -- debug"
date="2013-08-02T15:28:41Z"
content="""
With debug turned on.
[[!format bash \"\"\"
initremote flickr (encryption setup) (shared cipher) ok
(Recording state in git...)
describe flickr ok
(Recording state in git...)
/home/nrb/repos/annex/laptop-annex
fsck walkthrough.sh (checksum...) ok
fsck walkthrough.sh~ (checksum...) ok
/home/nrb/repos/annex/laptop-annex
copy walkthrough.sh (gpg) (checking flickr...) 16:18:52 [flickrannex-0.1.5] <module> : 'START'
16:18:52 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:18:52 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:18:52 [flickrannex-0.1.5] readFile : 'Done'
16:18:52 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:18:54 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:18:54 [flickrannex-0.1.5] main : 'Trying page: 1'
16:18:55 [flickrannex-0.1.5] main : 'Error. found nothing:{'pages': '1', 'cancreate': '1', 'total': '0', 'page': '1', 'perpage': '0'}'
16:18:55 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 - u'gitannex' - '8086216@N08''
16:18:55 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists'
16:18:55 [flickrannex-0.1.5] <module> : 'STOP: 2s'
(to flickr...)
16:18:55 [flickrannex-0.1.5] <module> : 'START'
16:18:55 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P ANNEX_FILE=/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:18:55 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:18:55 [flickrannex-0.1.5] readFile : 'Done'
16:18:55 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:18:57 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:18:58 [flickrannex-0.1.5] main : 'Trying page: 1'
16:18:58 [flickrannex-0.1.5] main : 'Error. found nothing:{'pages': '1', 'cancreate': '1', 'total': '0', 'page': '1', 'perpage': '0'}'
16:18:58 [flickrannex-0.1.5] postFile : '/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 to u'gitannex' - GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0'
16:18:58 [flickrannex-0.1.5] postFile : 'pre /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 size: 1047 more than 40234050.'
16:18:58 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0' - 'rb''
16:18:58 [flickrannex-0.1.5] readFile : 'Done'
16:18:58 [flickrannex-0.1.5] postFile : 'Uploading: /home/nrb/repos/gits/flickrannex/temp/encoded-GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0'
/home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
/home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
16:19:01 [flickrannex-0.1.5] postFile : 'Done: <Element 'rsp' at 0x8726b0c>'
16:19:01 [flickrannex-0.1.5] <module> : 'STOP: 5s'
ok
copy walkthrough.sh~ (checking flickr...) 16:19:01 [flickrannex-0.1.5] <module> : 'START'
16:19:01 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:19:01 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:19:01 [flickrannex-0.1.5] readFile : 'Done'
16:19:01 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:19:03 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:19:03 [flickrannex-0.1.5] main : 'Photoset gitannex found: <Element 'photoset' at 0x98264ec>'
16:19:03 [flickrannex-0.1.5] main : 'Trying page: 1'
16:19:03 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 - 72157634897264995L - '8086216@N08''
16:19:03 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists'
16:19:03 [flickrannex-0.1.5] <module> : 'STOP: 1s'
(to flickr...)
16:19:03 [flickrannex-0.1.5] <module> : 'START'
16:19:03 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz ANNEX_FILE=/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:19:03 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:19:03 [flickrannex-0.1.5] readFile : 'Done'
16:19:03 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:19:05 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:19:05 [flickrannex-0.1.5] main : 'Photoset gitannex found: <Element 'photoset' at 0x913a54c>'
16:19:05 [flickrannex-0.1.5] main : 'Trying page: 1'
16:19:05 [flickrannex-0.1.5] postFile : '/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 to 72157634897264995L - GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9'
16:19:05 [flickrannex-0.1.5] postFile : 'pre /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 size: 1044 more than 40234050.'
16:19:05 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9' - 'rb''
16:19:05 [flickrannex-0.1.5] readFile : 'Done'
16:19:05 [flickrannex-0.1.5] postFile : 'Uploading: /home/nrb/repos/gits/flickrannex/temp/encoded-GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9'
/home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
/home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
if res:
16:19:08 [flickrannex-0.1.5] postFile : 'Done: <Element 'rsp' at 0x913f0cc>'
16:19:08 [flickrannex-0.1.5] <module> : 'STOP: 4s'
ok
(Recording state in git...)
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis
whereis walkthrough.sh (3 copies)
86491ded-899c-425d-9470-bf446cb06db1 -- flickr (the flickr library)
8e766014-7154-4f4f-a04b-9d1b3d333db1 -- here (my laptop)
eed7055b-743b-4ab6-a390-29cfd326005d -- usbdrive (portable USB drive)
ok
whereis walkthrough.sh~ (3 copies)
86491ded-899c-425d-9470-bf446cb06db1 -- flickr (the flickr library)
8e766014-7154-4f4f-a04b-9d1b3d333db1 -- here (my laptop)
eed7055b-743b-4ab6-a390-29cfd326005d -- usbdrive (portable USB drive)
ok
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr
fsck walkthrough.sh (gpg) (checking flickr...) 16:22:57 [flickrannex-0.1.5] <module> : 'START'
16:22:57 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 ANNEX_HASH_1=kQ ANNEX_HASH_2=0P /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:22:57 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:22:57 [flickrannex-0.1.5] readFile : 'Done'
16:22:57 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:22:58 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:22:59 [flickrannex-0.1.5] main : 'Photoset gitannex found: <Element 'photoset' at 0x896d92c>'
16:22:59 [flickrannex-0.1.5] main : 'Trying page: 1'
16:22:59 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--280dd2d5003ad3962b1ecaa52ba45fdd44381fd0 - 72157634897264995L - '8086216@N08''
16:22:59 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists'
16:22:59 [flickrannex-0.1.5] <module> : 'STOP: 2s'
(fixing location log)
** Based on the location log, walkthrough.sh
** was expected to be present, but its content is missing.
failed
fsck walkthrough.sh~ (checking flickr...) 16:22:59 [flickrannex-0.1.5] <module> : 'START'
16:22:59 [flickrannex-0.1.5] main : 'ARGS: 'ANNEX_ACTION=checkpresent ANNEX_KEY=GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 ANNEX_HASH_1=m5 ANNEX_HASH_2=kz /home/nrb/repos/gits/flickrannex/flickrannex.py --dbglevel 1 --stderr''
16:22:59 [flickrannex-0.1.5] readFile : ''/home/nrb/repos/gits/flickrannex/flickrannex.conf' - 'r''
16:22:59 [flickrannex-0.1.5] readFile : 'Done'
16:22:59 [flickrannex-0.1.5] login : 'nrbray@yahoo.com'
16:23:01 [flickrannex-0.1.5] login : 'Done: '72157633920418017-5c0274bd421d7bb1' - None - '8086216@N08''
16:23:01 [flickrannex-0.1.5] main : 'Photoset gitannex found: <Element 'photoset' at 0x869992c>'
16:23:01 [flickrannex-0.1.5] main : 'Trying page: 1'
16:23:01 [flickrannex-0.1.5] checkFile : 'GPGHMACSHA1--131f95d3bc932d23ef6af47cf49db3c04be4f0f9 - 72157634897264995L - '8086216@N08''
16:23:01 [flickrannex-0.1.5] checkFile : 'No set exists, thus no files exists'
16:23:01 [flickrannex-0.1.5] <module> : 'STOP: 1s'
(fixing location log)
** Based on the location log, walkthrough.sh~
** was expected to be present, but its content is missing.
failed
(Recording state in git...)
git-annex: fsck: 2 failed
\"\"\" ]]
"""]]

View file

@ -0,0 +1,26 @@
As per IRC
22:13:10 < RichiH> joeyh: btw, i have been pondering a `git annex import --lazy` or some such which basically goes through a directory and deletes everything i find in the annex it run from
22:50:39 < joeyh> not sure of the use case
23:41:06 < RichiH> joeyh: the use case is "i have important a ton of data into my annexes. now, i am going through the usual crud of cp -ax'ed, rsync'ed, and other random 'new disk, move stuff around and just put a full dump over there' file dumps and would like to delete everything that's annexed already"
23:41:33 < RichiH> joeyh: that would allow me to spend time on dealing with the files which are not yet annexed
23:41:54 < RichiH> instead of verifying file after file which has been imported already
23:43:19 < joeyh> have you tried just running git annex import in a subdirectory and then deleting the dups?
23:45:34 < joeyh> or in a separate branch for that matter, which you could then merge in, etc
23:54:08 < joeyh> Thinking anout it some more, it would need to scan the whole work tree to see what keys were there, and populate a lookup table. I prefer to avoid things that need git-annex to do such a large scan and use arbitrary amounts of memory.
00:58:11 < RichiH> joeyh: that would force everything into the annex, though
00:58:20 < RichiH> a plain import, that is
00:58:53 < RichiH> in a usual data dump directory, there's tons of stuff i will never import
00:59:00 < RichiH> i want to delete large portions of it
00:59:32 < RichiH> but getting rid of duplicates first allows me to spend my time focused on stuff humans are good at: deciding
00:59:53 < RichiH> whereas the computer can focus on stuff it's good at: mindless comparision of bits
01:00:15 < RichiH> joeyh: as you're saying this is complex, maybe i need to rephrase
01:01:40 < RichiH> what i envision is git annex import --foo to 1) decide what hashing algorithm should be used for this file 2) hash that file 3) look into the annex if that hash is annexed 3a) optionally verify numcopies within the annex 4) delete the file in the source directory
01:01:47 < RichiH> and then move on to the next file
01:02:00 < RichiH> if the hash does not exist in the annex, leave it alone
01:02:50 < RichiH> if the hash exists in annex, but numcopies is not fulfilled, just import it as a normal import would
01:03:50 < RichiH> that sounds quite easy, to me; in fact i will prolly script it if you decide not to implement it
01:04:07 < RichiH> but i think it's useful for a _lot_ of people who migrate tons of data into annexes
01:04:31 < RichiH> thus i would rather see this upstream and not hacked locally
The only failure mode I see in the above is "file has been dropped elsewhere, numcopies not fulfilled, but that info is not synched to the local repo, yet" -- This could be worked around by always importing the data.

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U"
nickname="Richard"
subject="comment 2"
date="2013-08-02T07:21:50Z"
content="""
Using symmetric keys is significantly cheaper, computation-wise.
The scheme of encrypting symmetric keys with asymmetric ones is ancient, well-proven, and generally accepted as a good approach.
Using per-key files makes access control more fine-grained and is only a real performance issue once while creating the private key and a little bit every time more than one file needs to be decrypted as more than one symmetric key needs to be taken care of.
"""]]