mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

initial report on problem fetching from web within docker container

Browse source
This commit is contained in:
yarikoptic 2018-05-06 15:44:55 +00:00 • committed by admin
parent 59ae9f3f85
commit c85cb8e9f0
1 changed files with 84 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
doc/bugs/get_from_the_web_remote_fails___40__redirect__63____41___-_lack_of_further_debug_info.mdwn Normal file
View file

@ -0,0 +1,84 @@
### Please describe the problem.
Originally reported on [https://github.com/datalad/datalad/issues/2472](datalad issue 2472)
First I present full initial dump, then minimal set of commands and a few observations (e.g. seems to be related to running it within that docker environment, but cannot figure out why)
[[!format sh """
$> docker run -it --rm miykael/datalad_test
neuro@d0008523faf7:~$ cd ds000114/derivatives/fmriprep/
neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex get --debug sub-01.html
[2018-05-06 15:30:05.79497316] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","sub-01.html"]
get sub-01.html [2018-05-06 15:30:05.801191505] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-05-06 15:30:05.828819907] process done ExitSuccess
[2018-05-06 15:30:05.82896273] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-05-06 15:30:05.832716117] process done ExitSuccess
[2018-05-06 15:30:05.833138385] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..7d2ed09660cc91efbeb52b5792fb372622463813","--pretty=%H","-n1"]
[2018-05-06 15:30:05.837375012] process done ExitSuccess
[2018-05-06 15:30:05.837896891] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-05-06 15:30:05.838257537] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
(from web...)
(from web...)
Unable to access these remotes: web
Try making some of these repositories available:
00000000-0000-0000-0000-000000000001 -- web
b8b7bab3-c978-4f50-8ed4-e77900198eb9 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/fmriprep
failed
[2018-05-06 15:30:05.877509068] process done ExitSuccess
[2018-05-06 15:30:05.877986803] process done ExitSuccess
git-annex: get: 1 failed
neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex whereis sub-01.html
whereis sub-01.html (2 copies)
00000000-0000-0000-0000-000000000001 -- web
b8b7bab3-c978-4f50-8ed4-e77900198eb9 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/fmriprep
web: https://dl.dropbox.com/s/f03sxu7omhsibyc/sub-01.html?dl=0
ok
neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ curl https://dl.dropbox.com/s/f03sxu7omhsibyc/sub-01.html?dl=0
<html>
<head><title>Found</title></head>
<body>
<h1>Found</h1>
<p>The resource was found at <a href="https://dl.dropboxusercontent.com/s/f03sxu7omhsibyc/sub-01.html?dl=0">https://dl.dropboxusercontent.com/s/f03sxu7omhsibyc/sub-01.html?dl=0</a>;
you should be redirected automatically.
<!-- --></p>
<hr noshade>
<div align="right">WSGI Server</div>
</body>
</html>
neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex version
git-annex version: 6.20180416+gitg86b18966f-1~ndall+1
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.14.1 bloomfilter-2.0.1.0 cryptonite-0.20 DAV-1.3.1 feed-0.3.11.1 ghc-8.0.1 http-client-0.4.31.1 persistent-sqlite-2.6 torrent-10000.0.0 uuid-1.3.12 yesod-1.4.3
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64
"""]]
Minimal set of commands to replicate
[[!format sh """
git clone http://datasets.datalad.org/workshops/nih-2017/ds000114/derivatives/fmriprep/.git
cd fmriprep
git annex get --debug --from=web sub-01.html
"""]]
- I think the issue is somehow specific to running it within that docker -- if those commands ran on laptop, it works just fine!
- Issue persists with the `6.20180501-g0d2527e67` manually installed in the container
- Since then I also provided that file from the webserver, so if you do `git fetch`, it would start fetching that file from the web server (thus obfuscating the issue), so you would need ' --from=web ' to force an attempt to download from the web
- if before, with invoking wget/curl commands it was possible to see what is going on, ATM `--debug` of git-annex does not give any information about what is attempted to be fetched from web remote and what problem it is experiencing. I think it would be very valuable to get more troubleshooting information output in `--debug`
[[!meta author=yoh]]