This commit is contained in:
Joey Hess 2020-07-19 18:26:57 -04:00
commit 1489fbbdde
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
13 changed files with 466 additions and 0 deletions

View file

@ -0,0 +1,63 @@
Original TODO issue was [reincarnated](https://git-annex.branchable.com/todo/could_standalone___39__fixed__39___git-annex_binaries_be_prelinked__63__/#comment-0a78eeecf21ae7778436e233f9df22c2) and (thank you `git log -S`) and originally marked fixed by [6.20160613-81-gc4229be9a AKA 6.20160619~60](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=c4229be9a7a2318ef71b9ae433bc14bf604c9caf). Using standalone build of annex is notably (~30%) slower than any other. I was stracing running of a sample datalad test and looked inside to see e.g.
```
4188353 1595043333.316032 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
4188353 1595043333.316089 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
4188353 1595043333.316524 openat(AT_FDCWD, "/usr/lib/git-annex.linux/bin/git", O_RDONLY) = 3
4188353 1595043333.316818 openat(AT_FDCWD, "/usr/lib/git-annex.linux/shimmed/git/git", O_RDONLY|O_CLOEXEC) = 3
4188353 1595043333.316992 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317022 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317049 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317079 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317106 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317133 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317160 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317187 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317216 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317248 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317278 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317306 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317336 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317367 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317395 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317426 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317455 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317482 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317509 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317536 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317562 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317589 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317615 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317642 openat(AT_FDCWD, "/usr/lib/git-annex.linux//etc/ld.so.conf.d/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317670 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317697 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317724 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317750 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317778 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317805 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317832 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317858 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317889 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317917 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317943 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317970 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.317997 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318024 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318051 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318077 openat(AT_FDCWD, "/usr/lib/git-annex.linux//lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318105 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/tls/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318135 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/tls/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318162 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/tls/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318189 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/tls/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318216 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/haswell/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318243 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/haswell/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318270 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/x86_64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
4188353 1595043333.318299 openat(AT_FDCWD, "/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
```
so there is about 40 unsuccessful attempts to find/open libpcre2-8.so.0 . Other libraries might be less or more likely -- I didn't check... this one alone starts at 1595043333.316992 and finally succeeds at 1595043333.318299 -- so ~1.3 msec to just find this single library in a single invocation of git-annex (and we do many of them)... may be prelink or some other trick could be tried again? or if that is the lost effect of the original patch -- could there be some regression test added ?
git annex build: 8.20200617+git192-g5849bd634-1~ndall+1
[[!meta author=yoh]]
[[!tag projects/datalad]]

View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 1"
date="2020-07-18T03:57:20Z"
content="""
a test could be I guess as simple as
```
[ `strace -f git annex version 2>&1 1>/dev/null | grep open.*libpcre.*ENOENT | wc -l` -lt 2 ]
```
or some other number than the ideal `2` (i.e. a direct hit right away ;)) but something less than current 47
"""]]

View file

@ -0,0 +1,72 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 2"
date="2020-07-18T04:34:05Z"
content="""
<details>
<summary>also for some reason paths like `ld.so.conf.d` are consulted</summary>
```shell
$> strace -f git annex version 2>&1 1>/dev/null | grep ld.so
access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK <unfinished ...>
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5631] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5631] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5632] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5632] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5636] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5636] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5637] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5637] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5641] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5641] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5642] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5642] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5643] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5643] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d\", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libsqlite3.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libmagic.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libxml2.so.2\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libz.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
....
```
</details>
I thought that somehow may be ld.so.conf could indeed be used to actually configure paths for libraries to index at debian package installation level...
interestingly, if I do directly what you are doing in the ld.so symlink shim, I get a \"direct hit\":
```
$> ( GIT_ANNEX_PROGRAMPATH=/usr/lib/git-annex.linux/bin/git-annex strace -f /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex version 2>&1 ; ) | grep pcre
[pid 8539] openat(AT_FDCWD, \"/usr/lib/git-annex.linux/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0\", O_RDONLY|O_CLOEXEC <unfinished ...>
```
but something I guess throws it off in the \"top level\" runshell (sorry -- dunno a better word) since I do observe all those numerous attempts stracing the `/usr/lib/git-annex.linux/git-annex`
"""]]

View file

@ -0,0 +1,72 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 3"
date="2020-07-18T04:34:26Z"
content="""
<details>
<summary>also for some reason paths like `ld.so.conf.d` are consulted</summary>
```shell
$> strace -f git annex version 2>&1 1>/dev/null | grep ld.so
access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK <unfinished ...>
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5631] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5631] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5632] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5632] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5636] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5636] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5637] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5637] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5641] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5641] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5642] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5642] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5643] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5643] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3
[pid 5629] access(\"/etc/ld.so.preload\", R_OK) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/haswell\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/tls\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/haswell\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/x86_64/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d/x86_64\", 0x7ffd99c76b60) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libm.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] stat(\"/usr/lib/git-annex.linux//etc/ld.so.conf.d\", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libsqlite3.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libmagic.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libxml2.so.2\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libz.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 5629] openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
....
```
</details>
I thought that somehow may be ld.so.conf could indeed be used to actually configure paths for libraries to index at debian package installation level...
interestingly, if I do directly what you are doing in the ld.so symlink shim, I get a \"direct hit\":
```
$> ( GIT_ANNEX_PROGRAMPATH=/usr/lib/git-annex.linux/bin/git-annex strace -f /usr/lib/git-annex.linux/exe/git-annex --library-path /usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu: /usr/lib/git-annex.linux/shimmed/git-annex/git-annex version 2>&1 ; ) | grep pcre
[pid 8539] openat(AT_FDCWD, \"/usr/lib/git-annex.linux/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0\", O_RDONLY|O_CLOEXEC <unfinished ...>
```
but something I guess throws it off in the \"top level\" runshell (sorry -- dunno a better word) since I do observe all those numerous attempts stracing the `/usr/lib/git-annex.linux/git-annex`
"""]]

View file

@ -0,0 +1,38 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 4"
date="2020-07-18T04:49:49Z"
content="""
and since debian standalone is a \"static\" installation, most likely all the dance in `runshell` could be avoided upon every invocation and paths could be computed/recorded at installation time and then shim could be as lean as what I get with
```shell
$> rm -rf /tmp/fastannex; mkdir /tmp/fastannex; for f in /usr/lib/git-annex.linux/bin/git-annex{,-shell}; do echo -e \"#\!/bin/sh\nGIT_ANNEX_DIR=/usr/lib/git-annex.linux/ GIT_ANNEX_LD_LIBRARY_PATH=/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu: $f \\"\$@\\"\" > /tmp/fastannex/$(basename $f); done; chmod a+x /tmp/fastannex/*
```
and speed up on a random sample test is already obvious:
```
$> time PATH=/tmp/fastannex:$PATH python -m nose datalad/support/tests/test_annexrepo.py:test_annex_version_handling_bad_git_annex
.
----------------------------------------------------------------------
Ran 1 test in 0.280s
OK
PATH=/tmp/fastannex:$PATH python -m nose 0.78s user 0.25s system 100% cpu 1.029 total
```
vs
```
$> time python -m nose datalad/support/tests/test_annexrepo.py:test_annex_version_handling_bad_git_annex
.
----------------------------------------------------------------------
Ran 1 test in 0.498s
OK
python -m nose 0.98s user 0.29s system 101% cpu 1.254 total
```
"""]]

View file

@ -0,0 +1,38 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 5"
date="2020-07-18T04:50:12Z"
content="""
and since debian standalone is a \"static\" installation, most likely all the dance in `runshell` could be avoided upon every invocation and paths could be computed/recorded at installation time and then shim could be as lean as what I get with
```shell
$> rm -rf /tmp/fastannex; mkdir /tmp/fastannex; for f in /usr/lib/git-annex.linux/bin/git-annex{,-shell}; do echo -e \"#\!/bin/sh\nGIT_ANNEX_DIR=/usr/lib/git-annex.linux/ GIT_ANNEX_LD_LIBRARY_PATH=/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit:/usr/lib/git-annex.linux//etc/ld.so.conf.d:/usr/lib/git-annex.linux//lib64:/usr/lib/git-annex.linux//lib/x86_64-linux-gnu:/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu: $f \\"\$@\\"\" > /tmp/fastannex/$(basename $f); done; chmod a+x /tmp/fastannex/*
```
and speed up on a random sample test is already obvious:
```
$> time PATH=/tmp/fastannex:$PATH python -m nose datalad/support/tests/test_annexrepo.py:test_annex_version_handling_bad_git_annex
.
----------------------------------------------------------------------
Ran 1 test in 0.280s
OK
PATH=/tmp/fastannex:$PATH python -m nose 0.78s user 0.25s system 100% cpu 1.029 total
```
vs
```
$> time python -m nose datalad/support/tests/test_annexrepo.py:test_annex_version_handling_bad_git_annex
.
----------------------------------------------------------------------
Ran 1 test in 0.498s
OK
python -m nose 0.98s user 0.29s system 101% cpu 1.254 total
```
"""]]

View file

@ -0,0 +1,39 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 6"
date="2020-07-18T05:09:32Z"
content="""
<details>
<summary>bundled git is also a victim on many counts (bringing locale question back as well)</summary>
```shell
$> strace -f /usr/lib/git-annex.linux/git --version 2>&1 | grep open.*ENOE | nl | tail
155 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
156 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
157 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
158 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
159 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
160 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
161 openat(AT_FDCWD, \"/usr/lib/locale/locale-archive\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
162 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_MESSAGES\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
163 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_TIME\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
164 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_CTYPE\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1 47553.....................................:Sat 18 Jul 2020 01:07:51 AM EDT:.
(git)lena:~datalad/datalad-maint[maint]git
$> strace -f /usr/lib/git-annex.linux/git --version 2>&1 | grep open.*ENOE | grep -v locale | nl | tail
54 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
55 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
56 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
57 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
58 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
59 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
60 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
61 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
62 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
63 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
```
</details>
"""]]

View file

@ -0,0 +1,39 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 7"
date="2020-07-18T05:09:54Z"
content="""
<details>
<summary>bundled git is also a victim on many counts (bringing locale question back as well)</summary>
```shell
$> strace -f /usr/lib/git-annex.linux/git --version 2>&1 | grep open.*ENOE | nl | tail
155 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
156 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
157 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
158 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
159 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
160 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
161 openat(AT_FDCWD, \"/usr/lib/locale/locale-archive\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
162 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_MESSAGES\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
163 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_TIME\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
164 openat(AT_FDCWD, \"/usr/lib/locale/en_US.UTF-8/LC_CTYPE\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1 47553.....................................:Sat 18 Jul 2020 01:07:51 AM EDT:.
(git)lena:~datalad/datalad-maint[maint]git
$> strace -f /usr/lib/git-annex.linux/git --version 2>&1 | grep open.*ENOE | grep -v locale | nl | tail
54 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
55 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libpthread.so.0\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
56 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
57 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
58 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
59 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/librt.so.1\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
60 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/gconv/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
61 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//usr/lib/x86_64-linux-gnu/audit/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
62 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//etc/ld.so.conf.d/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
63 openat(AT_FDCWD, \"/usr/lib/git-annex.linux//lib64/libc.so.6\", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
```
</details>
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="Windows build of file (which includes libmagic)"
date="2020-07-16T22:01:17Z"
content="""
- I have successfully cross-built (well -- it produced binaries, didn't test if they are usable) built `file` using/as shown in https://github.com/nscaife/file-windows/pull/9
- There is https://github.com/julian-r/file-windows with a visual studio built `file`. releases provide builds using difference VS versions, not sure if it needs any VS runtime
- In conda apparently there is already `m2-file` package (a bit dated from 3 years ago)
so may be one of those ways could assist in establishing a built with MAGIC support on windows?
"""]]