mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
26548 commits 13 branches 442 tags 103 MiB
Commit graph

26548 commits

This branch
This branch
All branches
Author SHA1 Message Date
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
8728172053 Added a comment: new effect in list of "fixed" behavior to fail addurl if non-annex file exists 2016-01-15 04:45:08 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
18ada34e19 2016-01-15 04:14:36 +00:00
Joey Hess
6be9ee0d91
layout 2016-01-14 18:19:50 -04:00
Joey Hess
379004e3e7
move aside to better place 2016-01-14 18:19:00 -04:00
Joey Hess
a47480b0da
wording 2016-01-14 18:12:20 -04:00
Joey Hess
fb9727df8b
clarify 2016-01-14 18:11:09 -04:00
Joey Hess
d3ff430a04
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-14 17:57:05 -04:00
Joey Hess
6a5e241a74
throw exception if library copy fails 2016-01-14 17:56:21 -04:00
db48x
c7b6158d6a Added a comment: hexapodia! 2016-01-14 21:38:57 +00:00
anarcat
46dbbda5a4 add references to annex.thin in the upgrade docs 2016-01-14 21:21:36 +00:00
Joey Hess
196d39a813
fix format 2016-01-14 16:15:02 -04:00
Joey Hess
9240c02fd1
fix build of DistributionUpdate 2016-01-14 15:55:37 -04:00
Joey Hess
fb3d5bfedd
avoid tab warnings 2016-01-14 15:55:00 -04:00
Joey Hess
3363339c69
v6 mode is no longer experimental, downgrade warning 2016-01-14 15:33:07 -04:00
Joey Hess
3e7f419b16
add news item for git-annex 6.20160114 2016-01-14 15:30:10 -04:00
Joey Hess
d2dadeb611
comment 2016-01-14 15:29:35 -04:00
Joey Hess
9f030321cb
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-14 15:24:22 -04:00
Joey Hess
927e1a067e
fix import warnings 2016-01-14 10:30:54 -04:00
Joey Hess
6976d57f64
prep release 2016-01-14 10:18:30 -04:00
rashi.k0306@46b3566bf776802cd7adb39707f018beec8b0f26
297a744548 Added a comment: Duplicate Files Deleter 2016-01-14 11:42:10 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
78721c9afd Added a comment: "done" but only for non --fast/--relaxed 2016-01-13 23:49:30 +00:00
Joey Hess
af457d73a4
followup 2016-01-13 15:12:48 -04:00
Joey Hess
1d1cb16fe0
addurl: Refuse to overwrite any existing, non-annexed file. 2016-01-13 15:09:47 -04:00
Joey Hess
b52cf5697b
immediate queue flushing when annex.queuesize=1 
Previously, it only flushed when the queue got larger than 1.

Also, make the queue auto-flush when items are added, rather than needing
to be flushed as a separate step. This simplifies the code and make it more
efficient too, as it avoids needing to read the queue out of the state to
check if it should be flushed.
 2016-01-13 14:55:01 -04:00
Joey Hess
1d5b70db9c
addurl: Support --json, particularly useful in --batch mode. 2016-01-13 14:25:30 -04:00
Joey Hess
ecd0684bfc
avoid hard linking object from other repository when annex.thin is set 
This is simpler and less expensive than checking if the src file has a
link count >= 2, and also is unlocked.
 2016-01-13 14:19:31 -04:00
Joey Hess
2513c1dfd0
remove reundant isDirect check 
Already checked in wantHardLink
 2016-01-13 14:13:37 -04:00
Joey Hess
07e57ca6bc
changelog for fix 2016-01-13 12:38:01 -04:00
Joey Hess
2aa3e73de1
remove now unused toCygPath 2016-01-13 12:36:53 -04:00
Joey Hess
d90895ac94
Merge remote-tracking branch 'pkitslaar/master' 2016-01-13 12:36:30 -04:00
Joey Hess
fd3d866dec
another fix for old ghc 2016-01-13 12:32:57 -04:00
Joey Hess
7c7226748d
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-13 12:24:54 -04:00
simon.parzer@f837bbade0d93f560dc574b04e835b7875c4026f
36ca768f60 2016-01-13 16:19:48 +00:00
Joey Hess
d91eb24b59
fix build with old ghc 
And add missing (unused) mappend instance
 2016-01-13 12:18:20 -04:00
git-annex@8c58bb43191e0443be5d100448ba874284c5a778
7735439ff4 Added a comment: OSX Homebrew Has Same Failure 2016-01-13 00:09:13 +00:00
emanuele.ruffaldi@56b9c9a5c1f873994b869248e9b53aa20f437d20
2245e1d940 Added a comment: Updated 2016-01-12 22:47:49 +00:00
Joey Hess
0b24c0837a
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 14:10:43 -04:00
Joey Hess
423fffcd41
change keys database to use IKey type with more efficient serialization 
This breaks any existing keys database!

IKey serializes more efficiently than SKey, although this limits the
use of its Read/Show instances.

This makes the keys database use less disk space, and so should be a win.

Updated benchmark:

benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 64.04 μs   (63.95 μs .. 64.13 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.02 μs   (63.96 μs .. 64.08 μs)
std dev              218.2 ns   (172.5 ns .. 299.3 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 52.53 μs   (52.18 μs .. 53.21 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 52.31 μs   (52.18 μs .. 52.91 μs)
std dev              734.6 ns   (206.2 ns .. 1.623 μs)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 64.60 μs   (64.46 μs .. 64.77 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.74 μs   (64.57 μs .. 65.20 μs)
std dev              900.2 ns   (389.7 ns .. 1.733 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 52.46 μs   (52.29 μs .. 52.68 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 52.63 μs   (52.35 μs .. 53.37 μs)
std dev              1.362 μs   (562.7 ns .. 2.608 μs)
variance introduced by outliers: 24% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (old)
time                 487.3 μs   (484.7 μs .. 490.1 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 490.9 μs   (487.8 μs .. 496.5 μs)
std dev              13.95 μs   (6.841 μs .. 22.03 μs)
variance introduced by outliers: 20% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 6.633 ms   (5.741 ms .. 7.751 ms)
                     0.905 R²   (0.850 R² .. 0.965 R²)
mean                 8.252 ms   (7.803 ms .. 8.602 ms)
std dev              1.126 ms   (900.3 μs .. 1.430 ms)
variance introduced by outliers: 72% (severely inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 65.36 μs   (64.71 μs .. 66.37 μs)
                     0.998 R²   (0.995 R² .. 1.000 R²)
mean                 65.28 μs   (64.72 μs .. 66.45 μs)
std dev              2.576 μs   (920.8 ns .. 4.122 μs)
variance introduced by outliers: 42% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 52.34 μs   (52.25 μs .. 52.45 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.49 μs   (52.42 μs .. 52.59 μs)
std dev              255.4 ns   (205.8 ns .. 312.9 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 64.76 μs   (64.67 μs .. 64.84 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.67 μs   (64.62 μs .. 64.72 μs)
std dev              177.3 ns   (148.1 ns .. 217.1 ns)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 52.75 μs   (52.66 μs .. 52.82 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.69 μs   (52.63 μs .. 52.75 μs)
std dev              210.6 ns   (173.7 ns .. 265.9 ns)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 489.7 μs   (488.7 μs .. 490.7 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 490.4 μs   (489.6 μs .. 492.2 μs)
std dev              3.990 μs   (2.435 μs .. 7.604 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 9.994 ms   (9.186 ms .. 10.74 ms)
                     0.959 R²   (0.928 R² .. 0.979 R²)
mean                 9.906 ms   (9.343 ms .. 10.40 ms)
std dev              1.384 ms   (1.051 ms .. 2.100 ms)
variance introduced by outliers: 69% (severely inflated)
 2016-01-12 14:01:50 -04:00
Alan
e4b503e03b Added a comment: Some additional information 2016-01-12 17:48:58 +00:00
Joey Hess
789fc32af0
devblog 2016-01-12 13:41:11 -04:00
Joey Hess
9ffd85d676
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 13:34:07 -04:00
Joey Hess
75f61df323
cleanup 2016-01-12 13:31:13 -04:00
Joey Hess
5dc7a24680
update 2016-01-12 13:24:31 -04:00
Joey Hess
015a5e485f
add benchmarks of adding an associated file 
benchmarking keys database/addAssociatedFile to 1000 (old)
time                 516.1 μs   (514.7 μs .. 517.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 514.0 μs   (512.1 μs .. 515.2 μs)
std dev              4.740 μs   (2.972 μs .. 7.068 μs)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 5.750 ms   (4.857 ms .. 6.885 ms)
                     0.815 R²   (0.698 R² .. 0.904 R²)
mean                 7.858 ms   (7.311 ms .. 8.421 ms)
std dev              1.684 ms   (1.383 ms .. 2.027 ms)
variance introduced by outliers: 88% (severely inflated)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 515.7 μs   (514.8 μs .. 516.5 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 515.4 μs   (513.7 μs .. 516.6 μs)
std dev              4.824 μs   (2.957 μs .. 7.167 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 8.934 ms   (7.779 ms .. 10.05 ms)
                     0.868 R²   (0.751 R² .. 0.934 R²)
mean                 11.51 ms   (10.66 ms .. 12.26 ms)
std dev              2.174 ms   (1.816 ms .. 2.747 ms)
variance introduced by outliers: 82% (severely inflated)
 2016-01-12 13:22:31 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
50fad302a1 Added a comment: another side-effect of largefiles not being supported for my usecase 2016-01-12 17:20:08 +00:00
Joey Hess
647b2f33af
refactor 2016-01-12 13:15:15 -04:00
Joey Hess
134496b9fb
Merge branch 'master' of ssh://git-annex.branchable.com 2016-01-12 13:09:54 -04:00
Joey Hess
ca2a527e93
add FileKeyIndex to Keys db to optimize getAssociatedKey 
This is a schema change so will break any existing keys databases. But,
it's not been released yet, so I'm still able to make such changes.

This speeds up the benchmark quite nicely:

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 91.65 μs   (91.48 μs .. 91.81 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 91.78 μs   (91.66 μs .. 91.94 μs)
std dev              468.3 ns   (353.1 ns .. 624.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 53.33 μs   (53.23 μs .. 53.40 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 53.43 μs   (53.36 μs .. 53.53 μs)
std dev              274.2 ns   (211.7 ns .. 361.5 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 92.99 μs   (92.74 μs .. 93.27 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 92.90 μs   (92.76 μs .. 93.16 μs)
std dev              608.7 ns   (404.1 ns .. 963.5 ns)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 53.12 μs   (52.91 μs .. 53.39 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 52.84 μs   (52.68 μs .. 53.16 μs)
std dev              715.4 ns   (400.4 ns .. 1.370 μs)
 2016-01-12 13:07:14 -04:00
Joey Hess
f9c5aa84e0
add database benchmark 
The benchmark shows that the database access is quite fast indeed!
And, it scales linearly to the number of keys, with one exception,
getAssociatedKey.

Based on this benchmark, I don't think I need worry about optimising
for cases where all files are locked and the database is mostly empty.
In those cases, database access will be misses, and according to this
benchmark, should add only 50 milliseconds to runtime.

(NB: There may be some overhead to getting the database opened and locking
the handle that this benchmark doesn't see.)

joey@darkstar:~/src/git-annex>./git-annex benchmark
setting up database with 1000
setting up database with 10000
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 62.77 μs   (62.70 μs .. 62.85 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 62.81 μs   (62.76 μs .. 62.88 μs)
std dev              201.6 ns   (157.5 ns .. 259.5 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 50.02 μs   (49.97 μs .. 50.07 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.09 μs   (50.04 μs .. 50.17 μs)
std dev              206.7 ns   (133.8 ns .. 295.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 211.2 μs   (210.5 μs .. 212.3 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 211.0 μs   (210.7 μs .. 212.0 μs)
std dev              1.685 μs   (334.4 ns .. 3.517 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 173.5 μs   (172.7 μs .. 174.2 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 173.7 μs   (173.0 μs .. 175.5 μs)
std dev              3.833 μs   (1.858 μs .. 6.617 μs)
variance introduced by outliers: 16% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 64.01 μs   (63.84 μs .. 64.18 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.85 μs   (64.34 μs .. 66.02 μs)
std dev              2.433 μs   (547.6 ns .. 4.652 μs)
variance introduced by outliers: 40% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 50.33 μs   (50.28 μs .. 50.39 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.32 μs   (50.26 μs .. 50.38 μs)
std dev              202.7 ns   (167.6 ns .. 252.0 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 1.142 ms   (1.139 ms .. 1.146 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.142 ms   (1.140 ms .. 1.144 ms)
std dev              7.142 μs   (4.994 μs .. 10.98 μs)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 1.094 ms   (1.092 ms .. 1.096 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.095 ms   (1.095 ms .. 1.097 ms)
std dev              4.277 μs   (2.591 μs .. 7.228 μs)
 2016-01-12 13:07:03 -04:00
jhannwong@c9c7a67b5632a4bbc0c959cfeb3d340e02f28565
e6e4407336 removed 2016-01-12 04:19:36 +00:00