Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
commit
11a37386df
10 changed files with 268 additions and 8 deletions
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 17"
|
||||
date="2018-11-02T19:58:06Z"
|
||||
content="""
|
||||
> If it doesn't reply with DATA, then it's something to do with that one particular key. If it does, the problem must involve concurrency or the ssh connection somehow.
|
||||
|
||||
isn't the fact that it works without -j parallelization hints that it is about concurrency and not just the key?
|
||||
But for the sake of completeness, I did that \"manual\" `GET 0 sub-qa/s...` snippet and indeed was bombarded with the data.
|
||||
|
||||
> Dealt with it in 0b053b96119c1a2d6c2d03ba7f98f032e89b8b66 which would need to be deployed to the server side to test.
|
||||
|
||||
ok, will build/test, thanks!
|
||||
|
||||
"""]]
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="tests fail - can't build to test the change"
|
||||
date="2018-11-02T21:51:16Z"
|
||||
content="""
|
||||
the 7.20181031-13-g4d24b712b (7.20181031+git7-g39bff661b did fine) is failing some tests, get/see build logs at http://neuro.debian.net/_files/_buildlogs/git-annex/7.20181031+git13-g4d24b712b/
|
||||
|
||||
isn't there some CI running tests for git-annex or my failures are standalone build specific?
|
||||
"""]]
|
172
doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn
Normal file
172
doc/bugs/interoperability_issue_between_buster_and_stretch.mdwn
Normal file
|
@ -0,0 +1,172 @@
|
|||
### Please describe the problem.
|
||||
|
||||
If a git-annex client from Debian buster (currently 6.20180913-1) tries to talk with a server running Debian stretch (currently 6.20170101-1+deb9u2), it will fail mysteriously:
|
||||
|
||||
copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
On the stretch machine, setup a test git-annex repository.
|
||||
|
||||
On the buster machine, setup a test repository, add the stretch
|
||||
machine as a remote, then add a file in git-annex, and
|
||||
git-annex-sync-content to the stretch machine.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
Debian buster and stretch standard Debian packages.
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
On the buster machine:
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
(test) [1265]anarcat@curie:test130$ git init test
|
||||
Dépôt Git vide initialisé dans /home/anarcat/test/test/.git/
|
||||
(test) [1266]anarcat@curie:test$ cd test
|
||||
(test) [1267]anarcat@curie:test$ git annex init
|
||||
init ok
|
||||
(recording state in git...)
|
||||
(test) [1268]anarcat@curie:test$ echo foo > bar
|
||||
(test) [1269]anarcat@curie:test$ git annex add bar
|
||||
add bar ok
|
||||
(recording state in git...)
|
||||
(test) [1270]anarcat@curie:test$ git remote add sal sal:test
|
||||
(test) [1271]anarcat@curie:test$ git annex sync sal
|
||||
commit
|
||||
[master (commit racine) 05b6d24] git-annex in anarcat@curie:~/test/test
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 120000 bar
|
||||
ok
|
||||
pull sal
|
||||
warning: pas de commit commun
|
||||
remote: Counting objects: 5, done.
|
||||
remote: Compressing objects: 100% (3/3), done.
|
||||
remote: Total 5 (delta 0), reused 0 (delta 0)
|
||||
Dépaquetage des objets: 100% (5/5), fait.
|
||||
Depuis sal:test
|
||||
* [nouvelle branche] git-annex -> sal/git-annex
|
||||
ok
|
||||
(merging sal/git-annex into git-annex...)
|
||||
(recording state in git...)
|
||||
push sal
|
||||
Énumération des objets: 18, fait.
|
||||
Décompte des objets: 100% (18/18), fait.
|
||||
Compression par delta en utilisant jusqu'à 4 fils d'exécution
|
||||
Compression des objets: 100% (12/12), fait.
|
||||
Écriture des objets: 100% (16/16), 1.92 KiB | 654.00 KiB/s, fait.
|
||||
Total 16 (delta 1), réutilisés 0 (delta 0)
|
||||
To sal:test
|
||||
* [new branch] git-annex -> synced/git-annex
|
||||
* [new branch] master -> synced/master
|
||||
ok
|
||||
(test) [1272]anarcat@curie:test$ git annex sync --content sal
|
||||
commit
|
||||
Sur la branche master
|
||||
rien à valider, la copie de travail est propre
|
||||
ok
|
||||
pull sal
|
||||
ok
|
||||
copy bar (fd:18: hClose: resource vanished (Broken pipe)) failed
|
||||
git-annex: sync: 1 failed
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
The stretch repo is just `git init test && git -C test annex
|
||||
init`. SSH communication is setup with a passwordless SSH key in
|
||||
both. Debugging shows us the problem is with the p2pstdio command:
|
||||
|
||||
[[!format sh """
|
||||
$ git annex sync --content sal --debug
|
||||
[2018-11-03 15:25:35.018955156] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2018-11-03 15:25:35.022443588] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.022611687] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2018-11-03 15:25:35.02578792] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.026234842] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:35.030089742] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.041384604] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
|
||||
[2018-11-03 15:25:35.042220799] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
commit
|
||||
[2018-11-03 15:25:35.045907106] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in anarcat@curie:~/test/test"]
|
||||
Sur la branche master
|
||||
rien à valider, la copie de travail est propre
|
||||
[2018-11-03 15:25:35.077421642] process done ExitFailure 1
|
||||
ok
|
||||
[2018-11-03 15:25:35.077523489] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2018-11-03 15:25:35.079008549] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.079088278] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2018-11-03 15:25:35.080639101] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.080766018] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"]
|
||||
[2018-11-03 15:25:35.08225223] process done ExitSuccess
|
||||
[2018-11-03 15:25:35.082343323] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:35.084234147] process done ExitSuccess
|
||||
pull sal
|
||||
[2018-11-03 15:25:35.084663095] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","sal"]
|
||||
[2018-11-03 15:25:37.678014829] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.678131075] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"]
|
||||
[2018-11-03 15:25:37.68042014] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.680521541] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/master"]
|
||||
[2018-11-03 15:25:37.682164837] process done ExitFailure 1
|
||||
[2018-11-03 15:25:37.682259432] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"]
|
||||
[2018-11-03 15:25:37.683835717] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.683934116] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/synced/master..refs/remotes/sal/synced/master","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:37.686212308] process done ExitSuccess
|
||||
ok
|
||||
[2018-11-03 15:25:37.686343742] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2018-11-03 15:25:37.688215586] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.688305585] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2018-11-03 15:25:37.6901916] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.690393283] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..24d62bcb5e253326bde98283338a855c9bfbe983","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:37.692726875] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.692906438] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--"]
|
||||
copy bar [2018-11-03 15:25:37.696621566] chat: ssh ["sal","-S",".git/annex/ssh/sal","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/~/test' '--debug' 'f73778b3-17a4-4fee-9890-402bedf02456' --uuid 02ef3aa6-34d9-42b9-814e-f9ebc47c59e1"]
|
||||
[2018-11-03 15:25:37.96931512] P2P > ERROR auth failed
|
||||
(fd:18: hClose: resource vanished (Broken pipe)) failed
|
||||
[2018-11-03 15:25:37.96973331] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2018-11-03 15:25:37.970341712] read: git ["--version"]
|
||||
[2018-11-03 15:25:37.97296492] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.974463782] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"]
|
||||
[2018-11-03 15:25:37.978567382] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.979111982] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/synced/master"]
|
||||
[2018-11-03 15:25:37.982068735] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.982213309] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/synced/master..refs/heads/synced/master","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:37.986086966] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.986239042] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/sal/git-annex"]
|
||||
[2018-11-03 15:25:37.988954838] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.989110109] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/sal/git-annex..git-annex","--pretty=%H","-n1"]
|
||||
[2018-11-03 15:25:37.993956367] process done ExitSuccess
|
||||
[2018-11-03 15:25:37.994340874] read: ssh ["-O","stop","-S","sal","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"]
|
||||
[2018-11-03 15:25:38.00909442] process done ExitSuccess
|
||||
[2018-11-03 15:25:38.009887194] process done ExitSuccess
|
||||
[2018-11-03 15:25:38.010571056] process done ExitSuccess
|
||||
[2018-11-03 15:25:38.011368991] process done ExitSuccess
|
||||
git-annex: sync: 1 failed
|
||||
"""]]
|
||||
|
||||
A manual invocation confirms the [[design/p2p_protocol]] not supported
|
||||
by the remote, probably because it's too old:
|
||||
|
||||
[[!format sh """
|
||||
$ ssh sal git-annex-shell p2pstdio
|
||||
fatal: unrecognized command 'p2pstdio'
|
||||
git-annex-shell: git-shell failed
|
||||
"""]]
|
||||
|
||||
I understand the need for this command (which has no manpage, for what
|
||||
that's worth ;), but we should be able to fallback gracefully here. In
|
||||
the worst case, just treat this as a remote without git-annex-shell
|
||||
and revert to rsync to transfer files...
|
||||
|
||||
The workaround is to install the (working) backport in stretch, of course.
|
||||
|
||||
I wonder if this could be related to
|
||||
[[bugs/new_git-annex-shell_protocol_hides_remote_error_messages/]].
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes! I'm using git-annex like a maniac and I'm using it to archive
|
||||
Brazil right now. -- [[anarcat]]
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="spwhitton"
|
||||
avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
|
||||
subject="comment 1"
|
||||
date="2018-11-03T19:41:32Z"
|
||||
content="""
|
||||
Looking at the error message alone, I am pretty sure this is fixed in recent git-annex; I am just waiting on the Haskell transition in Debian to be able to build the package.
|
||||
"""]]
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="anarcat"
|
||||
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
|
||||
subject="oops"
|
||||
date="2018-11-03T19:52:48Z"
|
||||
content="""
|
||||
oh. oops. sorry for the noise then... maybe i should have filed this against the BTS first? :)
|
||||
"""]]
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="andrew"
|
||||
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
|
||||
subject="comment 3"
|
||||
date="2018-11-02T22:03:15Z"
|
||||
content="""
|
||||
Aaah OK. You don't necessarily need to add the repositories from assistant. If you create them from the command-line you need to manually add the full path to any repos you want the assistant to watch to a file located at `~/.config/git-annex/autostart`.
|
||||
|
||||
So on your server you want that `~/.config/git-annex/autostart` file to contain the full path to the repo that is located on the server and for the `~/.config/git-annex/autostart` file on your laptop you want it to contain the full path to your laptop repo. You also need to start the assistant on both your laptop and your server by running `git-annex-assistant --autostart` on your laptop and on your server. You'll also want to verify that this autostart command is happening automatically at boot (which `git-annex` might have done for you).
|
||||
"""]]
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="anarcat"
|
||||
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
|
||||
subject="workaround with youtube-dl"
|
||||
date="2018-11-03T20:52:49Z"
|
||||
content="""
|
||||
i hacked my way around this problem with this crazy commandline:
|
||||
|
||||
URL=www.youtube.com/user/WHATEVER/videos # or playlist or whatever
|
||||
youtube-dl --dump-json --flat-playlist $URL | jq -r '.id' | sed 's_^_https://www.youtube.com/watch?v=_' > urls.txt
|
||||
git -c annex.security.allowed-http-addresses=all annex addurl --batch < urls.txt
|
||||
|
||||
It's idiotic that youtube removed support for standard stuff, but that's the name of the game these days. Maybe git-annex could delegate this to youtube-dl? --[[anarcat]]
|
||||
"""]]
|
|
@ -1,8 +0,0 @@
|
|||
[[!comment format=mdwn
|
||||
username="spwhitton"
|
||||
avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb"
|
||||
subject="armel+armhf available in stretch-backports"
|
||||
date="2018-03-04T19:03:19Z"
|
||||
content="""
|
||||
@Alan: stretch-backports now includes git-annex on armel and armhf.
|
||||
"""]]
|
4
doc/todo/Retire_Esqueleto_as_a_dependency.mdwn
Normal file
4
doc/todo/Retire_Esqueleto_as_a_dependency.mdwn
Normal file
|
@ -0,0 +1,4 @@
|
|||
Since Esqueleto is seemingly abandoned and it's causing grief with NixOS: [[https://git-annex.branchable.com/bugs/Old_dependencies_are_causing_pain_with_NixOS_18.09/]] I did the work to remove it as a dependency in this commit
|
||||
[[https://github.com/seanparsons/git-annex/commit/62f291262f37327d9b2a864cd701aacb03766115]]
|
||||
|
||||
Alongside that is a bunch of nix specific files that support the building and/or development of git-annex, which is pretty much anything with "nix" in the name in the patch.
|
|
@ -0,0 +1,26 @@
|
|||
[[!comment format=mdwn
|
||||
username="anarcat"
|
||||
avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7"
|
||||
subject="rsync can do this on its own"
|
||||
date="2018-11-04T19:29:19Z"
|
||||
content="""
|
||||
i understand, thanks for taking the time to explain the tradeoffs! :)
|
||||
|
||||
i know that rsync is not used anymore, but I figured I should document this here since it's the first thing that it made me think of when i found out about it. as it turns out, rsync *does* have its own \"global status\", which has similar tradeoffs than git-annex (namely that it doesn't work in \"incremental mode\"). from #debian-til on OFTC:
|
||||
|
||||
[[!format txt \"\"\"
|
||||
13:30:15 <juliank> TIL: rsync --partial --no-inc-recursive -ah --info=progress2
|
||||
13:30:32 <lamby> explanation?
|
||||
13:30:33 * juliank apologizes for the :
|
||||
13:30:45 <juliank> So, --info=progress2 gives you overall progress
|
||||
13:30:53 <lamby> overall?
|
||||
13:30:58 <lamby> Oh, w/ multiple files?
|
||||
13:31:00 <juliank> and --no-inc-recursive makes it really overall by doing scanning up front
|
||||
13:31:08 <juliank> s/scanning/recursion/
|
||||
13:31:09 <lamby> Oh, now that's very neat.
|
||||
13:31:16 <juliank> and -h gives you human-readable sizes
|
||||
13:31:16 <lamby> WTB that in debmirror :3
|
||||
\"\"\"]]
|
||||
|
||||
So rsync allows you to switch back to the \"costly mode\" which *does* a full scan before starting. It's slower, but it allows you to get global progress info. It's a nice tradeoff and it's especially useful to be able to enable it on deman. I understand this might be complicated to implement in git-annex because there are many places where such an option would be required (and it's unclear how it would be named), but it's something that would certainly be useful for my use cases, where some repos have large files but not so many so are fairly fast to scan (e.g. i could do this on my video repo, but not the music repo). -- [[anarcat]]
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue