Commit graph

1631 commits

Author SHA1 Message Date
Joey Hess
9e6839fd3f
fix bug introduced in recent commit 2016-02-29 17:12:55 -04:00
Joey Hess
9731514c75
update sync branch to the orig branch when in adjusted branch 2016-02-29 16:57:42 -04:00
Joey Hess
048d513233
make assistant aware of adjusted branches when merging 2016-02-29 15:57:47 -04:00
Joey Hess
7c20bf6e7a
make sync aware of adjusted branches
So, it will pull and push the original branch, not the adjusted one.

And, for merging, it will use updateAdjustedBranch (not implemented yet).

Note that remaining uses of Git.Branch.current need to be checked too;
for things that should act on the original branch, and not the adjusted
branch.
2016-02-29 15:23:08 -04:00
Joey Hess
3b4557c754
Merge branch 'master' into adjustedbranch 2016-02-29 14:05:10 -04:00
Joey Hess
e520366c4d
metadata: Added -r to remove all current values of a field. 2016-02-29 13:00:46 -04:00
Joey Hess
98fcc23e4f
use DelMeta to unset tag
No behavior change, just simplification.
2016-02-29 13:00:23 -04:00
Joey Hess
0a1b02ce04
adjusted branches, proof of concept
"git annex adjust" may be a temporary interface, but works for a proof of
concept.

It is pretty fast at creating the adjusted branch. The main overhead is
injecting pointer files. It might be worth optimising that by reusing the
symlink target as the pointer file content. When I tried to do that,
the problem was that the clean filter doesn't use that same format, and so
git thought files had changed. Could be dealt with, perhaps make the clean
filter use symlink format for pointer files when on an adjusted branch?

But the real overhead is in checking out the branch, when git runs the
smudge filter once per file. That is perhaps too slow to be usable,
although it may only affect initial checkout of the branch, and not
updates. TBD.
2016-02-25 16:23:24 -04:00
Joey Hess
8bb80748ef
info: Mention when run in a dead repository. 2016-02-19 15:16:52 -04:00
Joey Hess
ec198fec83
fsck: When the only copy of a file is in a dead repository, mention the repository. 2016-02-19 15:12:11 -04:00
Joey Hess
15148ee9eb
annex.addunlocked
* add, addurl, import, importfeed: When in a v6 repository on a crippled
  filesystem, add files unlocked.
* annex.addunlocked: New configuration setting, makes files always be
  added unlocked. (v6 only)
2016-02-16 14:43:43 -04:00
Joey Hess
885e54df0a
fsck: Populate unlocked files in v6 repositories whose content is present in annex/objects but didn't reach the work tree.
This also handles fixing up after cf260d9a15
2016-02-14 17:27:50 -04:00
Joey Hess
675321264f
fsck: Detect and fix missing associated file mappings in v6 repositories.
This also handles fixing up after the bad data written by
cf260d9a15.
2016-02-14 17:09:54 -04:00
Joey Hess
74bbdfa888
files with only 1 linkCount may still be unlocked
When on crippled filesystem, or without annex.thin set.
2016-02-14 17:04:09 -04:00
Joey Hess
5b51db7645
clean up 2016-02-14 16:52:43 -04:00
Joey Hess
d3130930db
avoid --batch crashing if a remote fails to be accessed 2016-02-12 16:48:03 -04:00
Joey Hess
cc4d3e3d45
checkpresentkey: Allow to be run without an explicit remote and add --batch
* checkpresentkey: Allow to be run without an explicit remote.
* checkpresentkey: Added --batch.
2016-02-12 16:43:51 -04:00
Joey Hess
23cc315c38
matchexpression: Added --largefiles option to parse an annex.largefiles expression. 2016-02-03 16:58:36 -04:00
Joey Hess
403b56fb91
Limit annex.largefiles parsing to the subset of preferred content expressions that make sense in its context.
So, not "standard" or "lackingcopies", etc.
2016-02-03 15:04:42 -04:00
Joey Hess
d37fe6a547
annex.largefiles can be configured in .gitattributes too
This is particulary useful for v6 repositories, since the .gitattributes
configuration will apply in all clones of the repository.
2016-02-02 15:18:17 -04:00
Joey Hess
9a4af2324e
Fix reversion in lookupkey, contentlocation, and examinekey which caused them to sometimes output side messages. 2016-01-29 13:20:24 -04:00
Joey Hess
7c1df36d63
annex.addsmallfiles: New option controlling what is done when adding files not matching annex.largefiles. 2016-01-28 14:04:32 -04:00
Gabor Greif
7f13e8fba6
Ord constraint redundant 2016-01-28 12:34:07 -04:00
Joey Hess
039e83ed5d
Fix nasty reversion in the last release that broke sync --content's handling of many preferred content expressions.
The type checker should have noticed this, but the changes to mapM
that make it accept any Traversable hid the fact that it was not being
passed a list at all. Thus, what should have returned an empty list most
of the time instead returned [""] which was treated as the name of the
associated file, with disasterout consequences.

When I have time, I should add a test case checking what sync --content
drops. I should also consider replacing mapM with one re-specialized to
lists.
2016-01-26 14:28:43 -04:00
Joey Hess
f051b51645
remove 3 build flags
* Removed the webapp-secure build flag, rolling it into the webapp build
  flag.
* Removed the quvi and tahoe build flags, which only adds aeson to
  the core dependencies.
* Removed the feed build flag, which only adds feed to the core
  dependencies.

Build flags have cost in both code complexity and also make Setup configure
have to work harder to find a usable set of build flags when some
dependencies are missing.
2016-01-26 08:14:57 -04:00
Joey Hess
d3ba9fe5c8
matchexpression: New plumbing command to check if a preferred content expression matches some data. 2016-01-25 16:16:18 -04:00
Joey Hess
737e45156e
remove 163 lines of code without changing anything except imports 2016-01-20 16:36:33 -04:00
Joey Hess
70b8cad9c8
make noMessages disable closing of json object in --json mode
This allows things like Command.Find to use noMessages and generate their
own complete json objects. Previouly, Command.Find managed that only via a
hack, which wasn't compatable with batch mode.

Only Command.Find, Command.Smudge, and Commange.Status use noMessages
currently, and none except for Command.Find are impacted by this change.

Fixes find --json --batch output
2016-01-20 14:10:13 -04:00
Joey Hess
7aac76d40e
remove unused imports 2016-01-20 13:43:58 -04:00
Joey Hess
f1daa83118
remove no longer needed noMessages
All three of these are using batch mode to drive their processing, so
there is no automatic output, and noMessages is no longer needed.
2016-01-20 13:25:22 -04:00
Joey Hess
6d79f9e755
find --batch 2016-01-20 13:04:07 -04:00
Joey Hess
1bd4809bd2
remove excess space 2016-01-20 12:51:22 -04:00
Joey Hess
9b9b5a30e1
whereis --batch 2016-01-20 12:46:00 -04:00
Joey Hess
7d0ece86f6
add --batch 2016-01-19 17:48:42 -04:00
Joey Hess
1f3358512a
refactor 2016-01-19 15:55:32 -04:00
Joey Hess
68edd308af
registerurl: Check if a remote claims the url, same as addurl does. 2016-01-19 15:46:32 -04:00
Joey Hess
80d5feefc7
addurl --json: Include field for added key
(unless the file was added directly to git due to annex.largefiles configuration.)

(Also done by add --json and import --json)
2016-01-19 12:01:00 -04:00
Joey Hess
7b8e79c0f0
add, import: Support --json output.
Include added key in output.
2016-01-19 11:56:38 -04:00
Joey Hess
aa35f5cdf7
info: Support --batch mode. 2016-01-15 15:56:47 -04:00
Joey Hess
b9f921248e
convert existing non-annexed file to non-exception 2016-01-15 14:34:33 -04:00
Joey Hess
b26ce646e4
whereis --json: Urls are now listed inside the remote that claims them, rather than all together at the end. 2016-01-15 14:16:48 -04:00
Joey Hess
1d1cb16fe0
addurl: Refuse to overwrite any existing, non-annexed file. 2016-01-13 15:09:47 -04:00
Joey Hess
1d5b70db9c
addurl: Support --json, particularly useful in --batch mode. 2016-01-13 14:25:30 -04:00
Joey Hess
423fffcd41
change keys database to use IKey type with more efficient serialization
This breaks any existing keys database!

IKey serializes more efficiently than SKey, although this limits the
use of its Read/Show instances.

This makes the keys database use less disk space, and so should be a win.

Updated benchmark:

benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 64.04 μs   (63.95 μs .. 64.13 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.02 μs   (63.96 μs .. 64.08 μs)
std dev              218.2 ns   (172.5 ns .. 299.3 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 52.53 μs   (52.18 μs .. 53.21 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 52.31 μs   (52.18 μs .. 52.91 μs)
std dev              734.6 ns   (206.2 ns .. 1.623 μs)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 64.60 μs   (64.46 μs .. 64.77 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.74 μs   (64.57 μs .. 65.20 μs)
std dev              900.2 ns   (389.7 ns .. 1.733 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 52.46 μs   (52.29 μs .. 52.68 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 52.63 μs   (52.35 μs .. 53.37 μs)
std dev              1.362 μs   (562.7 ns .. 2.608 μs)
variance introduced by outliers: 24% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (old)
time                 487.3 μs   (484.7 μs .. 490.1 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 490.9 μs   (487.8 μs .. 496.5 μs)
std dev              13.95 μs   (6.841 μs .. 22.03 μs)
variance introduced by outliers: 20% (moderately inflated)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 6.633 ms   (5.741 ms .. 7.751 ms)
                     0.905 R²   (0.850 R² .. 0.965 R²)
mean                 8.252 ms   (7.803 ms .. 8.602 ms)
std dev              1.126 ms   (900.3 μs .. 1.430 ms)
variance introduced by outliers: 72% (severely inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 65.36 μs   (64.71 μs .. 66.37 μs)
                     0.998 R²   (0.995 R² .. 1.000 R²)
mean                 65.28 μs   (64.72 μs .. 66.45 μs)
std dev              2.576 μs   (920.8 ns .. 4.122 μs)
variance introduced by outliers: 42% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 52.34 μs   (52.25 μs .. 52.45 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.49 μs   (52.42 μs .. 52.59 μs)
std dev              255.4 ns   (205.8 ns .. 312.9 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 64.76 μs   (64.67 μs .. 64.84 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.67 μs   (64.62 μs .. 64.72 μs)
std dev              177.3 ns   (148.1 ns .. 217.1 ns)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 52.75 μs   (52.66 μs .. 52.82 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 52.69 μs   (52.63 μs .. 52.75 μs)
std dev              210.6 ns   (173.7 ns .. 265.9 ns)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 489.7 μs   (488.7 μs .. 490.7 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 490.4 μs   (489.6 μs .. 492.2 μs)
std dev              3.990 μs   (2.435 μs .. 7.604 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 9.994 ms   (9.186 ms .. 10.74 ms)
                     0.959 R²   (0.928 R² .. 0.979 R²)
mean                 9.906 ms   (9.343 ms .. 10.40 ms)
std dev              1.384 ms   (1.051 ms .. 2.100 ms)
variance introduced by outliers: 69% (severely inflated)
2016-01-12 14:01:50 -04:00
Joey Hess
015a5e485f
add benchmarks of adding an associated file
benchmarking keys database/addAssociatedFile to 1000 (old)
time                 516.1 μs   (514.7 μs .. 517.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 514.0 μs   (512.1 μs .. 515.2 μs)
std dev              4.740 μs   (2.972 μs .. 7.068 μs)

benchmarking keys database/addAssociatedFile to 1000 (new)
time                 5.750 ms   (4.857 ms .. 6.885 ms)
                     0.815 R²   (0.698 R² .. 0.904 R²)
mean                 7.858 ms   (7.311 ms .. 8.421 ms)
std dev              1.684 ms   (1.383 ms .. 2.027 ms)
variance introduced by outliers: 88% (severely inflated)

benchmarking keys database/addAssociatedFile to 10000 (old)
time                 515.7 μs   (514.8 μs .. 516.5 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 515.4 μs   (513.7 μs .. 516.6 μs)
std dev              4.824 μs   (2.957 μs .. 7.167 μs)

benchmarking keys database/addAssociatedFile to 10000 (new)
time                 8.934 ms   (7.779 ms .. 10.05 ms)
                     0.868 R²   (0.751 R² .. 0.934 R²)
mean                 11.51 ms   (10.66 ms .. 12.26 ms)
std dev              2.174 ms   (1.816 ms .. 2.747 ms)
variance introduced by outliers: 82% (severely inflated)
2016-01-12 13:22:31 -04:00
Joey Hess
647b2f33af
refactor 2016-01-12 13:15:15 -04:00
Joey Hess
f9c5aa84e0
add database benchmark
The benchmark shows that the database access is quite fast indeed!
And, it scales linearly to the number of keys, with one exception,
getAssociatedKey.

Based on this benchmark, I don't think I need worry about optimising
for cases where all files are locked and the database is mostly empty.
In those cases, database access will be misses, and according to this
benchmark, should add only 50 milliseconds to runtime.

(NB: There may be some overhead to getting the database opened and locking
the handle that this benchmark doesn't see.)

joey@darkstar:~/src/git-annex>./git-annex benchmark
setting up database with 1000
setting up database with 10000
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 62.77 μs   (62.70 μs .. 62.85 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 62.81 μs   (62.76 μs .. 62.88 μs)
std dev              201.6 ns   (157.5 ns .. 259.5 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 50.02 μs   (49.97 μs .. 50.07 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.09 μs   (50.04 μs .. 50.17 μs)
std dev              206.7 ns   (133.8 ns .. 295.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 211.2 μs   (210.5 μs .. 212.3 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 211.0 μs   (210.7 μs .. 212.0 μs)
std dev              1.685 μs   (334.4 ns .. 3.517 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 173.5 μs   (172.7 μs .. 174.2 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 173.7 μs   (173.0 μs .. 175.5 μs)
std dev              3.833 μs   (1.858 μs .. 6.617 μs)
variance introduced by outliers: 16% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 64.01 μs   (63.84 μs .. 64.18 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.85 μs   (64.34 μs .. 66.02 μs)
std dev              2.433 μs   (547.6 ns .. 4.652 μs)
variance introduced by outliers: 40% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 50.33 μs   (50.28 μs .. 50.39 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.32 μs   (50.26 μs .. 50.38 μs)
std dev              202.7 ns   (167.6 ns .. 252.0 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 1.142 ms   (1.139 ms .. 1.146 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.142 ms   (1.140 ms .. 1.144 ms)
std dev              7.142 μs   (4.994 μs .. 10.98 μs)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 1.094 ms   (1.092 ms .. 1.096 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.095 ms   (1.095 ms .. 1.097 ms)
std dev              4.277 μs   (2.591 μs .. 7.228 μs)
2016-01-12 13:07:03 -04:00
Joey Hess
d6fe7fdd7d
rekey: No longer copies over urls from the old to the new key.
It makes sense for migrate to do that, but not for this low-level (and
little used) plumbing command to.
2016-01-07 18:06:20 -04:00
Joey Hess
4b819bee2b
avoid confusing git with a modified ctime in clean filter
Linking the file to the tmp dir was not necessary in the clean
filter, and it caused the ctime to change, which caused git to think
the file was changed. This caused git status to get slow as it kept
re-cleaning unchanged files.
2016-01-07 17:48:04 -04:00
Joey Hess
3b960d1422
migrate and rekey v6 unlocked file support 2016-01-07 15:14:15 -04:00