mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2018-11-05 11:37:39 -04:00
commit 11a37386df
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
10 changed files with 268 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

16
doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_17_c39c3b0978777987dcd8c86906aa6532._comment Normal file
View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 17"
date="2018-11-02T19:58:06Z"
content="""
> If it doesn't reply with DATA, then it's something to do with that one particular key. If it does, the problem must involve concurrency or the ssh connection somehow.
isn't the fact that it works without -j parallelization hints that it is about concurrency and not just the key?
But for the sake of completeness, I did that \"manual\" `GET 0 sub-qa/s...` snippet and indeed was bombarded with the data.
> Dealt with it in 0b053b96119c1a2d6c2d03ba7f98f032e89b8b66 which would need to be deployed to the server side to test.
ok, will build/test, thanks!
"""]]

10
doc/bugs/annex_get_-J_16_via_ssh_stalls_/comment_18_bffd59a35ae162f4141b6de7efcc7422._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="tests fail - can't build to test the change"
date="2018-11-02T21:51:16Z"
content="""
the 7.20181031-13-g4d24b712b (7.20181031+git7-g39bff661b did fine) is failing some tests, get/see build logs at http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181031+git13-g4d24b712b/
isn't there some CI running tests for git-annex or my failures are standalone build specific?
"""]]

172
doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn Normal file
View file

@ -0,0 +1,172 @@
### Please describe the problem.
If a git-annex client from Debian buster (currently 6.20180913-1) tries to talk with a server running Debian stretch (currently 6.20170101-1+deb9u2), it will fail mysteriously:
copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed
### What steps will reproduce the problem?
On the stretch machine, setup a test git-annex repository.
On the buster machine, setup a test repository, add the stretch
machine as a remote, then add a file in git-annex, and
git-annex-sync-content to the stretch machine.
### What version of git-annex are you using? On what operating system?
Debian buster and stretch standard Debian packages.
### Please provide any additional information below.
On the buster machine:
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
(test) [1265]anarcat@curie:test130$ git init test
Dépôt Git vide initialisé dans /home/anarcat/test/test/.git/
(test) [1266]anarcat@curie:test$ cd test
(test) [1267]anarcat@curie:test$ git annex init
init ok
(recording state in git...)
(test) [1268]anarcat@curie:test$ echo foo > bar
(test) [1269]anarcat@curie:test$ git annex add bar
add bar ok
(recording state in git...)
(test) [1270]anarcat@curie:test$ git remote add sal sal:test
(test) [1271]anarcat@curie:test$ git annex sync sal
commit
[master (commit racine) 05b6d24] git-annex in anarcat@curie:~/test/test
1 file changed, 1 insertion(+)
create mode 120000 bar
ok
pull sal
warning: pas de commit commun
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Dépaquetage des objets: 100% (5/5), fait.
Depuis sal:test
* [nouvelle branche] git-annex -> sal/git-annex
ok
(merging sal/git-annex into git-annex...)
(recording state in git...)
push sal
Énumération des objets: 18, fait.
Décompte des objets: 100% (18/18), fait.
Compression par delta en utilisant jusqu'à 4 fils d'exécution
Compression des objets: 100% (12/12), fait.
Écriture des objets: 100% (16/16), 1.92 KiB | 654.00 KiB/s, fait.
Total 16 (delta 1), réutilisés 0 (delta 0)
To sal:test
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
(test) [1272]anarcat@curie:test$ git annex sync --content sal
commit
Sur la branche master
rien à valider, la copie de travail est propre
ok
pull sal
ok
copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed
git-annex: sync: 1 failed
# End of transcript or log.
"""]]
The stretch repo is just `git init test && git -C test annex
init`. SSH communication is setup with a passwordless SSH key in
both. Debugging shows us the problem is with the p2pstdio command:
[[!format sh """
$ git annex sync --content sal --debug
[2018-11-03 15:25:35.018955156] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-11-03 15:25:35.022443588] process done ExitSuccess
[2018-11-03 15:25:35.022611687] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-11-03 15:25:35.02578792] process done ExitSuccess
[2018-11-03 15:25:35.026234842] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"]
[2018-11-03 15:25:35.030089742] process done ExitSuccess
[2018-11-03 15:25:35.041384604] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-11-03 15:25:35.042220799] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
commit
[2018-11-03 15:25:35.045907106] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in anarcat@curie:~/test/test"]
Sur la branche master
rien à valider, la copie de travail est propre
[2018-11-03 15:25:35.077421642] process done ExitFailure 1
ok
[2018-11-03 15:25:35.077523489] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2018-11-03 15:25:35.079008549] process done ExitSuccess
[2018-11-03 15:25:35.079088278] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2018-11-03 15:25:35.080639101] process done ExitSuccess
[2018-11-03 15:25:35.080766018] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"]
[2018-11-03 15:25:35.08225223] process done ExitSuccess
[2018-11-03 15:25:35.082343323] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"]
[2018-11-03 15:25:35.084234147] process done ExitSuccess
pull sal
[2018-11-03 15:25:35.084663095] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","sal"]
[2018-11-03 15:25:37.678014829] process done ExitSuccess
[2018-11-03 15:25:37.678131075] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"]
[2018-11-03 15:25:37.68042014] process done ExitSuccess
[2018-11-03 15:25:37.680521541] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/master"]
[2018-11-03 15:25:37.682164837] process done ExitFailure 1
[2018-11-03 15:25:37.682259432] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"]
[2018-11-03 15:25:37.683835717] process done ExitSuccess
[2018-11-03 15:25:37.683934116] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/synced/master..refs/remotes/sal/synced/master","--pretty=%H","-n1"]
[2018-11-03 15:25:37.686212308] process done ExitSuccess
ok
[2018-11-03 15:25:37.686343742] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-11-03 15:25:37.688215586] process done ExitSuccess
[2018-11-03 15:25:37.688305585] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-11-03 15:25:37.6901916] process done ExitSuccess
[2018-11-03 15:25:37.690393283] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"]
[2018-11-03 15:25:37.692726875] process done ExitSuccess
[2018-11-03 15:25:37.692906438] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--"]
copy bar [2018-11-03 15:25:37.696621566] chat: ssh ["sal","-S",".git/annex/ssh/sal","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/~/test' '--debug' 'f73778b3-17a4-4fee-9890-402bedf02456' --uuid 02ef3aa6-34d9-42b9-814e-f9ebc47c59e1"]
[2018-11-03 15:25:37.96931512] P2P > ERROR auth failed
(fd:18: hClose: resource vanished (Broken pipe)) failed
[2018-11-03 15:25:37.96973331] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2018-11-03 15:25:37.970341712] read: git ["--version"]
[2018-11-03 15:25:37.97296492] process done ExitSuccess
[2018-11-03 15:25:37.974463782] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"]
[2018-11-03 15:25:37.978567382] process done ExitSuccess
[2018-11-03 15:25:37.979111982] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"]
[2018-11-03 15:25:37.982068735] process done ExitSuccess
[2018-11-03 15:25:37.982213309] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/synced/master..refs/heads/synced/master","--pretty=%H","-n1"]
[2018-11-03 15:25:37.986086966] process done ExitSuccess
[2018-11-03 15:25:37.986239042] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/git-annex"]
[2018-11-03 15:25:37.988954838] process done ExitSuccess
[2018-11-03 15:25:37.989110109] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/git-annex..git-annex","--pretty=%H","-n1"]
[2018-11-03 15:25:37.993956367] process done ExitSuccess
[2018-11-03 15:25:37.994340874] read: ssh ["-O","stop","-S","sal","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"]
[2018-11-03 15:25:38.00909442] process done ExitSuccess
[2018-11-03 15:25:38.009887194] process done ExitSuccess
[2018-11-03 15:25:38.010571056] process done ExitSuccess
[2018-11-03 15:25:38.011368991] process done ExitSuccess
git-annex: sync: 1 failed
"""]]
A manual invocation confirms the [[design/p2p_protocol]] not supported
by the remote, probably because it's too old:
[[!format sh """
$ ssh sal git-annex-shell p2pstdio
fatal: unrecognized command 'p2pstdio'
git-annex-shell: git-shell failed
"""]]
I understand the need for this command (which has no manpage, for what
that's worth ;), but we should be able to fallback gracefully here. In
the worst case, just treat this as a remote without git-annex-shell
and revert to rsync to transfer files...
The workaround is to install the (working) backport in stretch, of course.
I wonder if this could be related to
[[bugs/new_git-annex-shell_protocol_hides_remote_error_messages/]].
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes! I'm using git-annex like a maniac and I'm using it to archive
Brazil right now. -- [[anarcat]]

8
doc/bugs/interoperability_issue_between_buster_and_stretch/comment_1_4e7d50e097418290b185f43e05ddd86b._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="spwhitton"
avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
subject="comment 1"
date="2018-11-03T19:41:32Z"
content="""
Looking at the error message alone, I am pretty sure this is fixed in recent git-annex; I am just waiting on the Haskell transition in Debian to be able to build the package.
"""]]

8
doc/bugs/interoperability_issue_between_buster_and_stretch/comment_2_768f116e98a3cef3285ef941464c0880._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="anarcat"
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
subject="oops"
date="2018-11-03T19:52:48Z"
content="""
oh. oops. sorry for the noise then... maybe i should have filed this against the BTS first? :)
"""]]

10
doc/forum/assistant_to_have_synced_folders/comment_3_49b00584ef4388e130cce63f4f33572f._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="andrew"
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
subject="comment 3"
date="2018-11-02T22:03:15Z"
content="""
Aaah OK. You don't necessarily need to add the repositories from assistant. If you create them from the command-line you need to manually add the full path to any repos you want the assistant to watch to a file located at `~/.config/git-annex/autostart`.
So on your server you want that `~/.config/git-annex/autostart` file to contain the full path to the repo that is located on the server and for the `~/.config/git-annex/autostart` file on your laptop you want it to contain the full path to your laptop repo. You also need to start the assistant on both your laptop and your server by running `git-annex-assistant --autostart` on your laptop and on your server. You'll also want to verify that this autostart command is happening automatically at boot (which `git-annex` might have done for you).
"""]]

14
doc/forum/importfeed_does_not_work_with_youtube_anymore/comment_2_58733a5065493dbac5cb8d12a029dace._comment Normal file
View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="anarcat"
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
subject="workaround with youtube-dl"
date="2018-11-03T20:52:49Z"
content="""
i hacked my way around this problem with this crazy commandline:
URL=www.youtube.com/user/WHATEVER/videos # or playlist or whatever
youtube-dl --dump-json --flat-playlist $URL | jq -r '.id' | sed 's_^_https://www.youtube.com/watch?v=_' > urls.txt
git -c annex.security.allowed-http-addresses=all annex addurl --batch < urls.txt
It's idiotic that youtube removed support for standard stuff, but that's the name of the game these days. Maybe git-annex could delegate this to youtube-dl? --[[anarcat]]
"""]]

8
doc/install/comment_4_3df3ee5c70d4d1ec6963f125c479a9c9._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="spwhitton"
avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
subject="armel+armhf available in stretch-backports"
date="2018-03-04T19:03:19Z"
content="""
@Alan: stretch-backports now includes git-annex on armel and armhf.
"""]]

4
doc/todo/Retire_Esqueleto_as_a_dependency.mdwn Normal file
View file

@ -0,0 +1,4 @@
Since Esqueleto is seemingly abandoned and it's causing grief with NixOS: [[https://git-annex.branchable.com/bugs/Old_dependencies_are_causing_pain_with_NixOS_18.09/]] I did the work to remove it as a dependency in this commit
[[https://github.com/seanparsons/git-annex/commit/62f291262f37327d9b2a864cd701aacb03766115]]
Alongside that is a bunch of nix specific files that support the building and/or development of git-annex, which is pretty much anything with "nix" in the name in the patch.

26
doc/todo/wishlist__58___global_progress_status/comment_4_60b7597af2b86ce6bcb31731f6d8c54e._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="anarcat"
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
subject="rsync can do this on its own"
date="2018-11-04T19:29:19Z"
content="""
i understand, thanks for taking the time to explain the tradeoffs! :)
i know that rsync is not used anymore, but I figured I should document this here since it's the first thing that it made me think of when i found out about it. as it turns out, rsync *does* have its own \"global status\", which has similar tradeoffs than git-annex (namely that it doesn't work in \"incremental mode\"). from #debian-til on OFTC:
[[!format txt \"\"\"
13:30:15 <juliank> TIL: rsync --partial --no-inc-recursive -ah --info=progress2
13:30:32 <lamby> explanation?
13:30:33 * juliank apologizes for the :
13:30:45 <juliank> So, --info=progress2 gives you overall progress
13:30:53 <lamby> overall?
13:30:58 <lamby> Oh, w/ multiple files?
13:31:00 <juliank> and --no-inc-recursive makes it really overall by doing scanning up front
13:31:08 <juliank> s/scanning/recursion/
13:31:09 <lamby> Oh, now that's very neat.
13:31:16 <juliank> and -h gives you human-readable sizes
13:31:16 <lamby> WTB that in debmirror :3
\"\"\"]]
So rsync allows you to switch back to the \"costly mode\" which *does* a full scan before starting. It's slower, but it allows you to get global progress info. It's a nice tradeoff and it's especially useful to be able to enable it on deman. I understand this might be complicated to implement in git-annex because there are many places where such an option would be required (and it's unclear how it would be named), but it's something that would certainly be useful for my use cases, where some repos have large files but not so many so are fairly fast to scan (e.g. i could do this on my video repo, but not the music repo). -- [[anarcat]]
"""]]