I'm looking into another SSH-based hang in the DataLad tests.  I can't
trigger the hang on my local system (Debian Buster), but in a Ubuntu
Xenial VM I can.  The hang bisects to 1f2e2d15e (async exception
safety, 2020-06-03).  It disappears on the parent of 1f2e2d15e or if I
revert that commit on top of master (currently 3b6754e2a).

I was able to reduce the hang down to a `git annex get` from an rsync
remote.  Here is a script that triggers the hang via a Xenial Docker
container.  Sorry for the length; given the system interaction, it's
the simplest reproducer I've managed to come up with.

[[!format sh """
cd "$(mktemp -d ${TMPDIR:-/tmp}/ga-XXXXXXX)"

cat >demo.sh <<'EOF'
git annex version

cd "$(mktemp -d /tmp/ga-XXXXXXX)"
remdir="$PWD/store"
mkdir "$remdir"
git init repo
(
    cd repo
    git annex init
    git annex initremote r type=rsync rsyncurl="localhost:$remdir" encryption=none
    echo 0 >f0
    git annex add f0
    git commit -m'f0'
    git annex copy --to=r f0
    git annex drop f0
)

git clone repo clone
(
    cd clone
    git annex init clone
    git annex enableremote r
    git annex get --debug f0
)
EOF

cat >Dockerfile <<'EOF'
FROM ubuntu:xenial

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y --no-install-recommends \
  curl rsync openssh-client openssh-server ca-certificates
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cd /root && curl -fSsL \
  https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz \
  | tar xz
ENV PATH="/root/git-annex.linux:$PATH"

RUN git config --system user.name "u"
RUN git config --system user.email "u@e"

RUN mkdir -p /root/.ssh
RUN mkdir -p /var/run/sshd
RUN ssh-keygen -f /root/.ssh/id_rsa -N ""
RUN cat /root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys
RUN echo "Host localhost\nStrictHostKeyChecking no\n" >>/root/.ssh/config

COPY demo.sh /root/demo.sh
CMD /usr/sbin/sshd && sh /root/demo.sh
EOF

docker build -t ga-rsync-hang:latest .
docker run -it --rm ga-rsync-hang:latest
"""]]

Output, where the last line stalled:

```
[... 50 lines ...]
git-annex version: 8.20200618-g3b6754e2a
build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
[... 32 lines ...]
[2020-07-07 14:28:55.252423613] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2020-07-07 14:28:55.256082009] process done ExitSuccess
[2020-07-07 14:28:55.256316845] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2020-07-07 14:28:55.26056207] process done ExitSuccess
[2020-07-07 14:28:55.260917245] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--cached","--","f0"]
get f0 [2020-07-07 14:28:55.26481346] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2020-07-07 14:28:55.268786541] process done ExitSuccess
[2020-07-07 14:28:55.268997933] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2020-07-07 14:28:55.272176401] process done ExitSuccess
[2020-07-07 14:28:55.272550418] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..ea06186dfc3dab39316da534144d5d6ef3d6090c","--pretty=%H","-n1"]
[2020-07-07 14:28:55.275989985] process done ExitSuccess
[2020-07-07 14:28:55.27634403] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2020-07-07 14:28:55.276782737] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2020-07-07 14:28:55.279315206] read: git ["config","--null","--list"]
[2020-07-07 14:28:55.281990769] process done ExitSuccess
(from r...) 
[2020-07-07 14:28:55.288194172] read: rsync ["-e","'ssh' '-S' '.git/annex/ssh/localhost' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T'","--progress","--inplace","localhost:/tmp/ga-075MZKs/store/0c5/66e/'SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa'",".git/annex/tmp/SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa"]
SHA256E-s2--9a271f2a916b0b6ee6cecb2426f0b3206ef074578be55d9bc94f6f3fe3ab86aa

              0   0%    0.00kB/s    0:00:00  
              2 100%    1.95kB/s    0:00:00 (xfr#1, to-chk=0/1)
^C
```

Replacing "FROM ubuntu:xenial" with "FROM ubuntu:bionic" (a later
release) resolves the hang, so perhaps there is some interaction with
an older rsync or openssh version.  Here are the versions that are
present in Xenial:

```
OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g  1 Mar 2016

rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc
```

Thinking it's an older version of openssh or rsync, I tried with an older version of Debian.  Using "FROM debian:stretch-slim" doesn't hang.  Here are the versions there:

```
OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u  20 Dec 2019

rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc
```

However, going back farther, "FROM debian:jessie-slim" does hang.  The
versions there:

```
OpenSSH_6.7p1 Debian-5+deb8u8, OpenSSL 1.0.1t  3 May 2016

rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc
```

So perhaps there's some interaction with openssh before 7.4p1 or rsync
before 3.1.2.  Those are pretty old versions, and, in the case of
Jessie, the release is now EOL.  But I figured it was at least worth
writing up given that the hang isn't triggered until a recent commit
in git-annex.

[[!meta author=kyle]]
[[!tag projects/datalad]]

> [[fixed|done]] made it cancel the threads 2 seconds after the process
> exited if still running. So there will be a bit of a slow down when using
> that broken ssh version, but it will work. --[[Joey]]
