mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
26521 commits 13 branches 442 tags 103 MiB
Commit graph

26521 commits

This branch
This branch
All branches
Author SHA1 Message Date
Joey Hess
07e57ca6bc
changelog for fix 2016-01-13 12:38:01 -04:00
Joey Hess
2aa3e73de1
remove now unused toCygPath 2016-01-13 12:36:53 -04:00
Joey Hess
d90895ac94
Merge remote-tracking branch 'pkitslaar/master' 2016-01-13 12:36:30 -04:00
Joey Hess
fd3d866dec
another fix for old ghc 2016-01-13 12:32:57 -04:00
Joey Hess
7c7226748d
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-13 12:24:54 -04:00
simon.parzer@f837bbade0d93f560dc574b04e835b7875c4026f
36ca768f60 2016-01-13 16:19:48 +00:00
Joey Hess
d91eb24b59
fix build with old ghc 
And add missing (unused) mappend instance
 2016-01-13 12:18:20 -04:00
git-annex@8c58bb43191e0443be5d100448ba874284c5a778
7735439ff4 Added a comment: OSX Homebrew Has Same Failure 2016-01-13 00:09:13 +00:00
emanuele.ruffaldi@56b9c9a5c1f873994b869248e9b53aa20f437d20
2245e1d940 Added a comment: Updated 2016-01-12 22:47:49 +00:00
Joey Hess
0b24c0837a
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 14:10:43 -04:00
Joey Hess
423fffcd41
change keys database to use IKey type with more efficient serialization 
This breaks any existing keys database!

IKey serializes more efficiently than SKey, although this limits the
use of its Read/Show instances.

This makes the keys database use less disk space, and so should be a win.

Updated benchmark:

benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 64.04 μs   (63.95 μs .. 64.13 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.02 μs   (63.96 μs .. 64.08 μs)
std dev              218.2 ns   (172.5 ns .. 299.3 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 52.53 μs   (52.18 μs .. 53.21 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 52.31 μs   (52.18 μs .. 52.91 μs)
std dev              734.6 ns   (206.2 ns .. 1.623 μs)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 64.60 μs   (64.46 μs .. 64.77 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.74 μs   (64.57 μs .. 65.20 μs)
std dev              900.2 ns   (389.7 ns .. 1.733 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 52.46 μs   (52.29 μs .. 52.68 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 52.63 μs   (52.35 μs .. 53.37 μs)
std dev              1.362 μs   (562.7 ns .. 2.608 μs)
variance introduced by outliers: 24% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (old)
time                 487.3 μs   (484.7 μs .. 490.1 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 490.9 μs   (487.8 μs .. 496.5 μs)
std dev              13.95 μs   (6.841 μs .. 22.03 μs)
variance introduced by outliers: 20% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 6.633 ms   (5.741 ms .. 7.751 ms)
                     0.905 R²   (0.850 R² .. 0.965 R²)
mean                 8.252 ms   (7.803 ms .. 8.602 ms)
std dev              1.126 ms   (900.3 μs .. 1.430 ms)
variance introduced by outliers: 72% (severely inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 65.36 μs   (64.71 μs .. 66.37 μs)
                     0.998 R²   (0.995 R² .. 1.000 R²)
mean                 65.28 μs   (64.72 μs .. 66.45 μs)
std dev              2.576 μs   (920.8 ns .. 4.122 μs)
variance introduced by outliers: 42% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 52.34 μs   (52.25 μs .. 52.45 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.49 μs   (52.42 μs .. 52.59 μs)
std dev              255.4 ns   (205.8 ns .. 312.9 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 64.76 μs   (64.67 μs .. 64.84 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.67 μs   (64.62 μs .. 64.72 μs)
std dev              177.3 ns   (148.1 ns .. 217.1 ns)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 52.75 μs   (52.66 μs .. 52.82 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.69 μs   (52.63 μs .. 52.75 μs)
std dev              210.6 ns   (173.7 ns .. 265.9 ns)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 489.7 μs   (488.7 μs .. 490.7 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 490.4 μs   (489.6 μs .. 492.2 μs)
std dev              3.990 μs   (2.435 μs .. 7.604 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 9.994 ms   (9.186 ms .. 10.74 ms)
                     0.959 R²   (0.928 R² .. 0.979 R²)
mean                 9.906 ms   (9.343 ms .. 10.40 ms)
std dev              1.384 ms   (1.051 ms .. 2.100 ms)
variance introduced by outliers: 69% (severely inflated)
 2016-01-12 14:01:50 -04:00
Alan
e4b503e03b Added a comment: Some additional information 2016-01-12 17:48:58 +00:00
Joey Hess
789fc32af0
devblog 2016-01-12 13:41:11 -04:00
Joey Hess
9ffd85d676
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 13:34:07 -04:00
Joey Hess
75f61df323
cleanup 2016-01-12 13:31:13 -04:00
Joey Hess
5dc7a24680
update 2016-01-12 13:24:31 -04:00
Joey Hess
015a5e485f
add benchmarks of adding an associated file 
benchmarking keys database/addAssociatedFile to 1000 (old)
time                 516.1 μs   (514.7 μs .. 517.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 514.0 μs   (512.1 μs .. 515.2 μs)
std dev              4.740 μs   (2.972 μs .. 7.068 μs)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 5.750 ms   (4.857 ms .. 6.885 ms)
                     0.815 R²   (0.698 R² .. 0.904 R²)
mean                 7.858 ms   (7.311 ms .. 8.421 ms)
std dev              1.684 ms   (1.383 ms .. 2.027 ms)
variance introduced by outliers: 88% (severely inflated)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 515.7 μs   (514.8 μs .. 516.5 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 515.4 μs   (513.7 μs .. 516.6 μs)
std dev              4.824 μs   (2.957 μs .. 7.167 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 8.934 ms   (7.779 ms .. 10.05 ms)
                     0.868 R²   (0.751 R² .. 0.934 R²)
mean                 11.51 ms   (10.66 ms .. 12.26 ms)
std dev              2.174 ms   (1.816 ms .. 2.747 ms)
variance introduced by outliers: 82% (severely inflated)
 2016-01-12 13:22:31 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
50fad302a1 Added a comment: another side-effect of largefiles not being supported for my usecase 2016-01-12 17:20:08 +00:00
Joey Hess
647b2f33af
refactor 2016-01-12 13:15:15 -04:00
Joey Hess
134496b9fb
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 13:09:54 -04:00
Joey Hess
ca2a527e93
add FileKeyIndex to Keys db to optimize getAssociatedKey 
This is a schema change so will break any existing keys databases. But,
it's not been released yet, so I'm still able to make such changes.

This speeds up the benchmark quite nicely:

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 91.65 μs   (91.48 μs .. 91.81 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 91.78 μs   (91.66 μs .. 91.94 μs)
std dev              468.3 ns   (353.1 ns .. 624.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 53.33 μs   (53.23 μs .. 53.40 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 53.43 μs   (53.36 μs .. 53.53 μs)
std dev              274.2 ns   (211.7 ns .. 361.5 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 92.99 μs   (92.74 μs .. 93.27 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 92.90 μs   (92.76 μs .. 93.16 μs)
std dev              608.7 ns   (404.1 ns .. 963.5 ns)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 53.12 μs   (52.91 μs .. 53.39 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 52.84 μs   (52.68 μs .. 53.16 μs)
std dev              715.4 ns   (400.4 ns .. 1.370 μs)
 2016-01-12 13:07:14 -04:00
Joey Hess
f9c5aa84e0
add database benchmark 
The benchmark shows that the database access is quite fast indeed!
And, it scales linearly to the number of keys, with one exception,
getAssociatedKey.

Based on this benchmark, I don't think I need worry about optimising
for cases where all files are locked and the database is mostly empty.
In those cases, database access will be misses, and according to this
benchmark, should add only 50 milliseconds to runtime.

(NB: There may be some overhead to getting the database opened and locking
the handle that this benchmark doesn't see.)

joey@darkstar:~/src/git-annex>./git-annex benchmark
setting up database with 1000
setting up database with 10000
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 62.77 μs   (62.70 μs .. 62.85 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 62.81 μs   (62.76 μs .. 62.88 μs)
std dev              201.6 ns   (157.5 ns .. 259.5 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 50.02 μs   (49.97 μs .. 50.07 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.09 μs   (50.04 μs .. 50.17 μs)
std dev              206.7 ns   (133.8 ns .. 295.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 211.2 μs   (210.5 μs .. 212.3 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 211.0 μs   (210.7 μs .. 212.0 μs)
std dev              1.685 μs   (334.4 ns .. 3.517 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 173.5 μs   (172.7 μs .. 174.2 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 173.7 μs   (173.0 μs .. 175.5 μs)
std dev              3.833 μs   (1.858 μs .. 6.617 μs)
variance introduced by outliers: 16% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 64.01 μs   (63.84 μs .. 64.18 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.85 μs   (64.34 μs .. 66.02 μs)
std dev              2.433 μs   (547.6 ns .. 4.652 μs)
variance introduced by outliers: 40% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 50.33 μs   (50.28 μs .. 50.39 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.32 μs   (50.26 μs .. 50.38 μs)
std dev              202.7 ns   (167.6 ns .. 252.0 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 1.142 ms   (1.139 ms .. 1.146 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.142 ms   (1.140 ms .. 1.144 ms)
std dev              7.142 μs   (4.994 μs .. 10.98 μs)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 1.094 ms   (1.092 ms .. 1.096 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.095 ms   (1.095 ms .. 1.097 ms)
std dev              4.277 μs   (2.591 μs .. 7.228 μs)
 2016-01-12 13:07:03 -04:00
jhannwong@c9c7a67b5632a4bbc0c959cfeb3d340e02f28565
e6e4407336 removed 2016-01-12 04:19:36 +00:00
sameerds
99fdd14f9f Added a comment: cygwin/msys-independent fix? 2016-01-12 04:07:40 +00:00
jhannwong@c9c7a67b5632a4bbc0c959cfeb3d340e02f28565
4b075509f2 Added a comment: Does not work 2016-01-12 03:52:58 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
9bc2b9ad57 Added a comment 2016-01-12 03:35:15 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
ac17961c43 initial report 2016-01-12 03:30:25 +00:00
emanuele.ruffaldi@56b9c9a5c1f873994b869248e9b53aa20f437d20
d003501fab 2016-01-12 00:32:31 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
d73469a87f 2016-01-11 22:10:57 +00:00
Joey Hess
8111eb21e6
split out raw sql interface 2016-01-11 15:52:11 -04:00
Joey Hess
2a27c77170
comment 2016-01-11 12:52:11 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
e540c3ffc5 2016-01-11 16:47:52 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
7a5969cc92 2016-01-11 16:47:04 +00:00
Joey Hess
bf776d6557
comment 2016-01-11 12:38:38 -04:00
Joey Hess
7ace93b922
comment 2016-01-11 12:34:01 -04:00
Joey Hess
c579b9800a
comment 2016-01-11 12:30:52 -04:00
Joey Hess
142803a90d
comment 2016-01-11 12:29:54 -04:00
Joey Hess
0021ae4485
commet 2016-01-11 12:21:45 -04:00
Joey Hess
dbdce142d4
comment 2016-01-11 12:20:40 -04:00
Joey Hess
1f6f9a8d34
When annex.http-headers is used to set the User-Agent header, avoid sending User-Agent: git-annex 2016-01-11 12:10:38 -04:00
pkitslaar
7cf6299888 Added a comment: Possible fix for rsync windows paths 2016-01-11 10:25:23 +00:00
Pieter Kitslaar
6cd134ade1 Added new toMSYS2Path function for use with rsync on Windows. 2016-01-11 11:18:58 +01:00
git-annex.branchable.com@69b3f2da2837a3d9de8eab0b93771008294b7d69
4caaa15f76 Added a comment 2016-01-10 12:06:53 +00:00
pkitslaar
bc1751b109 removed 2016-01-10 07:50:56 +00:00
pkitslaar
c3f0907667 Added a comment: Same here 2016-01-10 07:50:14 +00:00
pkitslaar
9e199a28f5 Added a comment: Same here 2016-01-10 07:49:52 +00:00
wsha.code+ga@b38779424f41c5701bbe5937340be43ff1474b2d
bd1ce3ef0d Added a comment 2016-01-10 03:08:24 +00:00
G.nius.ck@d885edcfde63422ee84e5ee501b7aa545e91298d
6bdbc06317 Added a comment: What do you mean by "git-annex" branch? 2016-01-09 16:04:30 +00:00
https://openid.stackexchange.com/user/e65e6d0e-58ba-41de-84cc-1f2ba54cf574
e4e76c881a Added a comment: Copy from .git/config 2016-01-09 06:44:59 +00:00
mark@6b90344cdab3158eacb94a3944460d138afc9bef
f285579729 Added a comment: Special remotes 2016-01-08 21:12:45 +00:00