Commit graph

1439 commits

Author SHA1 Message Date
Joey Hess
d1879df0ba
response 2016-01-22 15:39:27 -04:00
db
26b76a6b06 Added a comment 2016-01-22 19:00:58 +00:00
Joey Hess
fc9a02d600
comment 2016-01-20 15:00:13 -04:00
Joey Hess
6d79f9e755
find --batch 2016-01-20 13:04:07 -04:00
Joey Hess
9b9b5a30e1
whereis --batch 2016-01-20 12:46:00 -04:00
Joey Hess
7d0ece86f6
add --batch 2016-01-19 17:48:42 -04:00
Joey Hess
c06999df42
already done this earlier 2016-01-19 15:57:37 -04:00
fiatjaf
4dffddb343 small formatting fix. 2016-01-19 19:40:39 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
6a2a4aaba1 2016-01-19 19:26:32 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
77fc00a68f Added a comment 2016-01-19 19:24:11 +00:00
Joey Hess
432151f966
comment 2016-01-19 13:15:35 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
e35fa26c55 Added a comment 2016-01-19 16:39:49 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
9054f58e17 initial request 2016-01-19 16:31:03 +00:00
Joey Hess
aae77b1856
comment 2016-01-19 11:35:14 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
aa2e815954 marked as mine 2016-01-18 22:08:35 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
ee94708bd7 2016-01-18 22:01:41 +00:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
ee58e4601e 2016-01-18 21:59:04 +00:00
tribut
37ef508def Added a comment 2016-01-18 16:22:54 +00:00
CandyAngel
165e5da2df Added a comment 2016-01-18 13:17:00 +00:00
tribut
ca19a2dbb6 Reformat irc log to add linebreaks (is there a better way to do this in markdown?) 2016-01-18 13:11:08 +00:00
tribut
d677bfea98 2016-01-18 13:07:23 +00:00
db
94af2cd44d 2016-01-17 17:14:30 +00:00
Joey Hess
aa35f5cdf7
info: Support --batch mode. 2016-01-15 15:56:47 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
78721c9afd Added a comment: "done" but only for non --fast/--relaxed 2016-01-13 23:49:30 +00:00
Joey Hess
af457d73a4
followup 2016-01-13 15:12:48 -04:00
Joey Hess
ecd0684bfc
avoid hard linking object from other repository when annex.thin is set
This is simpler and less expensive than checking if the src file has a
link count >= 2, and also is unlocked.
2016-01-13 14:19:31 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
50fad302a1 Added a comment: another side-effect of largefiles not being supported for my usecase 2016-01-12 17:20:08 +00:00
Joey Hess
f9c5aa84e0
add database benchmark
The benchmark shows that the database access is quite fast indeed!
And, it scales linearly to the number of keys, with one exception,
getAssociatedKey.

Based on this benchmark, I don't think I need worry about optimising
for cases where all files are locked and the database is mostly empty.
In those cases, database access will be misses, and according to this
benchmark, should add only 50 milliseconds to runtime.

(NB: There may be some overhead to getting the database opened and locking
the handle that this benchmark doesn't see.)

joey@darkstar:~/src/git-annex>./git-annex benchmark
setting up database with 1000
setting up database with 10000
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time                 62.77 μs   (62.70 μs .. 62.85 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 62.81 μs   (62.76 μs .. 62.88 μs)
std dev              201.6 ns   (157.5 ns .. 259.5 ns)

benchmarking keys database/getAssociatedFiles from 1000 (miss)
time                 50.02 μs   (49.97 μs .. 50.07 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.09 μs   (50.04 μs .. 50.17 μs)
std dev              206.7 ns   (133.8 ns .. 295.3 ns)

benchmarking keys database/getAssociatedKey from 1000 (hit)
time                 211.2 μs   (210.5 μs .. 212.3 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 211.0 μs   (210.7 μs .. 212.0 μs)
std dev              1.685 μs   (334.4 ns .. 3.517 μs)

benchmarking keys database/getAssociatedKey from 1000 (miss)
time                 173.5 μs   (172.7 μs .. 174.2 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 173.7 μs   (173.0 μs .. 175.5 μs)
std dev              3.833 μs   (1.858 μs .. 6.617 μs)
variance introduced by outliers: 16% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (hit)
time                 64.01 μs   (63.84 μs .. 64.18 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 64.85 μs   (64.34 μs .. 66.02 μs)
std dev              2.433 μs   (547.6 ns .. 4.652 μs)
variance introduced by outliers: 40% (moderately inflated)

benchmarking keys database/getAssociatedFiles from 10000 (miss)
time                 50.33 μs   (50.28 μs .. 50.39 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 50.32 μs   (50.26 μs .. 50.38 μs)
std dev              202.7 ns   (167.6 ns .. 252.0 ns)

benchmarking keys database/getAssociatedKey from 10000 (hit)
time                 1.142 ms   (1.139 ms .. 1.146 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.142 ms   (1.140 ms .. 1.144 ms)
std dev              7.142 μs   (4.994 μs .. 10.98 μs)

benchmarking keys database/getAssociatedKey from 10000 (miss)
time                 1.094 ms   (1.092 ms .. 1.096 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.095 ms   (1.095 ms .. 1.097 ms)
std dev              4.277 μs   (2.591 μs .. 7.228 μs)
2016-01-12 13:07:03 -04:00
Joey Hess
55ad30d1d9
update 2016-01-08 16:30:31 -04:00
Joey Hess
c96fb11a96
devblog 2016-01-07 18:03:06 -04:00
Joey Hess
722f56a99d
update 2016-01-07 15:47:19 -04:00
Joey Hess
d667a68b7e
test: Added --keep-failures option. 2016-01-06 13:44:12 -04:00
Joey Hess
50d25c186d
update 2016-01-05 17:41:46 -04:00
https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4
1d6d7a6e16 2016-01-05 02:20:14 +00:00
Joey Hess
b1373f0d15
update 2016-01-01 16:02:11 -04:00
Joey Hess
f36f24197a
scan for unlocked files on init/upgrade of v6 repo 2016-01-01 15:09:42 -04:00
Joey Hess
b03a24dc10
update 2016-01-01 14:26:09 -04:00
Joey Hess
4be2b57606
add test: conflict resolution (mixed locked and unlocked file) 2015-12-30 16:36:39 -04:00
Joey Hess
0b8bba8031
test suite 100% pass in v6, finally!
Set annex.largefiles when adding the conflicting non-annexed file,
otherwise it would be added as an annexed file.
2015-12-30 15:12:45 -04:00
Joey Hess
7fd9fa3d72
update 2015-12-29 17:47:49 -04:00
Joey Hess
6aa19b7184
Merge branch 'master' of ssh://git-annex.branchable.com 2015-12-29 17:40:49 -04:00
umeboshi
813262448c Added a comment: Link to TH 2015-12-29 20:36:42 +00:00
Joey Hess
b6b34f4916
automatic conflict resolution for v6 unlocked files
Several tricky parts:

* When the conflict is just between the same key being locked and unlocked,
  the unlocked version wins, and the file is not renamed in this case.

* Need to update associated file map when conflict resolution renames
  an unlocked file.

* git merge runs the smudge filter on the conflicting file, and actually
  overwrites the file with the same content it had before, and so
  invalidates its inode cache. This makes it difficult to know when it's
  safe to remove such files as conflict cruft, without going so far as to
  compare their entire contents.

  Dealt with this by preventing the smudge filter from populating the file
  when a merge is run. However, that also prevents the smudge filter being
  run for non-conflicting files, so eg moving a file won't put its new
  content into place.

* Ideally, if a merge or a merge conflict resolution renames an unlocked
  file, the file in the work tree can just be moved, rather than copying
  the content to a new worktree file.

  This is attempted to be done in merge conflict resolution, but
  due to git merge's behavior of running smudge filters, what actually
  seems to happen is the old worktree file with the content is deleted and
  rewritten as a pointer file, so doesn't get reused.

So, this is probably not as efficient as it optimally could be.
If that becomes a problem, could look into running the merge in a separate
worktree and updating the real worktree more efficiently, similarly to the
direct mode merge. However, the direct mode merge had a lot of bugs, and
I'd rather not use that more error-prone method unless really needed.
2015-12-29 15:41:09 -04:00
Joey Hess
121f5d5b0c
annex.thin
Decided it's too scary to make v6 unlocked files have 1 copy by default,
but that should be available to those who need it. This is consistent with
git-annex not dropping unused content without --force, etc.

* Added annex.thin setting, which makes unlocked files in v6 repositories
  be hard linked to their content, instead of a copy. This saves disk
  space but means any modification of an unlocked file will lose the local
  (and possibly only) copy of the old version.
* Enable annex.thin by default on upgrade from direct mode to v6, since
  direct mode made the same tradeoff.
* fix: Adjusts unlocked files as configured by annex.thin.
2015-12-27 15:59:59 -04:00
Joey Hess
025f284ac1
reorg 2015-12-26 15:15:02 -04:00
Joey Hess
fcb013044b
update 2015-12-26 15:13:05 -04:00
Joey Hess
4224fae71f
optimise read and write for Keys database (untested)
Writes are optimised by queueing up multiple writes when possible.
The queue is flushed after the Annex monad action finishes. That makes it
happen on program termination, and also whenever a nested Annex monad action
finishes.

Reads are optimised by checking once (per AnnexState) if the database
exists. If the database doesn't exist yet, all reads return mempty.

Reads also cause queued writes to be flushed, so reads will always be
consistent with writes (as long as they're made inside the same Annex monad).
A future optimisation path would be to determine when that's not necessary,
which is probably most of the time, and avoid flushing unncessarily.

Design notes for this commit:

- separate reads from writes
- reuse a handle which is left open until program
  exit or until the MVar goes out of scope (and autoclosed then)
- writes are queued
  - queue is flushed periodically
  - immediate queue flush before any read
  - auto-flush queue when database handle is garbage collected
  - flush queue on exit from Annex monad
    (Note that this may happen repeatedly for a single database connection;
    or a connection may be reused for multiple Annex monad actions,
    possibly even concurrent ones.)
- if database does not exist (or is empty) the handle
  is not opened by reads; reads instead return empty results
- writes open the handle if it was not open previously
2015-12-23 19:18:52 -04:00
Joey Hess
b3690c4499
update 2015-12-22 18:19:32 -04:00
Joey Hess
ca2c977704
wip v6 support for assistant
Files are not yet added to v6 repos in unlocked mode.
2015-12-21 18:41:15 -04:00
Joey Hess
fbf6c25de5
interaction with shared clones 2015-12-17 18:46:52 -04:00