Commit graph

35701 commits

Author SHA1 Message Date
Joey Hess
2f94b5419a
use new name for new format export dbs
Delete the old export dbs on upgrade.

Testing this an exporting to a directory with both exporttree=yes and
importtree=yes, it refused to let an interrupted export proceed after
upgrade, with "unsafe to overwrite file". An import resolved the
problem.
2019-11-06 17:34:15 -04:00
Joey Hess
09241f17ed
document v8 2019-11-06 17:13:01 -04:00
Joey Hess
3b820f08f7
use new name for new format content identifier db
It will be populated automatically by the next command that needs data
from it, the same way it gets populated in a fresh clone. That may be a
little expensive, but it's a one time cost, and no slower than in a
fresh clone.
2019-11-06 16:43:52 -04:00
Joey Hess
1b5f4b67b5
use new name for new format fsck db
The old db is cleaned up when a new incremental fsck is started.

The incremental fsck won't pick up where the old one left off, but I
consider this a minor enough thing that it can just be documented and
won't be a problem.
2019-11-06 16:27:25 -04:00
Joey Hess
dc9295017f
v8 upgrade of keys db
Renamed the database to .git/annex/keysdb;
the old .git/annex/keys gets deleted during the upgrade.

It is possible that an old git-annex process is running during the
upgrade. If so, it will be able to continue using the old keys db until the
upgrade is complete, and then will presumably fail in some ugly way. Or
perhaps the upgrade will be unable to delete the open files on some
systems, and so fail with an ugly error message.

It's also possible for multiple processes to be running the upgrade
concurrently. That should be fine; they will both write the same
information into the keys db.

Other databases still need to be upgraded.
2019-11-06 16:16:00 -04:00
Joey Hess
09ee6b0ccb
Windows: Fix handling of changes to time zone.
Used to work but was broken in version 7.20181031, specifically commit
5ab0f48ffb.

That this was not noticed over at least 1 daylight savings time zone
changes makes me wonder if the TSDelta stuff is still needed.
Perhaps the mtime on Windows no longer changes when the time zone is changed?
2019-11-06 14:36:49 -04:00
Joey Hess
89bdcffdfa
found a way to extract InodeCache from git index
This will allow a race-free database transition. It is somewhat hairy in
that it depends on an unspecified git output format.
2019-11-06 14:23:00 -04:00
Joey Hess
6147130e86
Merge branch 'master' into sqlite 2019-11-05 12:59:28 -04:00
Joey Hess
68a579313d
update 2019-11-05 12:57:46 -04:00
Joey Hess
41825195ba
Merge branch 'master' of ssh://git-annex.branchable.com 2019-11-05 12:51:50 -04:00
Joey Hess
fccfcba89f
update to document current state of sqlite branch 2019-11-05 12:50:53 -04:00
Joey Hess
e2d4c133f5
init: fix data loss bug
Fix bug that lost modifications to unlocked files when init is re-ran in an
already initialized repo.

In retrospect needing scanUnlockedFiles False in the direct mode upgrade
path was a good hint that it was unsafe when used with True.

However, this bug did not affect upgrade from v5. In such an upgrade, an
unlocked file that is modified is left as-is. The only place
scanUnlockedFiles True did overwrite modified unlocked files is during an
git-annex init of a repo that was already initialized by git-annex.

(I also tried a scenario where the repo had not been initialized by
git-annex yet, but was cloned from a v7 repo with an unlocked file, and the
pointer file replaced with some other content, and the data loss did not
occur in that situation.)

Since the fixed scanUnlockedFiles avoids overwriting non-pointer files,
it should be safe to run in any situation, so there's no need any longer
for the parameter.
2019-11-05 12:41:15 -04:00
Joey Hess
3cb2427dbc
fix typo 2019-11-04 14:30:27 -04:00
yarikoptic
0173b153de added <details> to fold done/fixed things 2019-11-04 17:56:46 +00:00
Ilya_Shlyakhter
363be9e338 Added a comment 2019-11-04 16:40:53 +00:00
Joey Hess
b1cf0869db
link to todo 2019-11-04 11:48:45 -04:00
Joey Hess
96c81fa4f1
comment 2019-11-04 11:47:21 -04:00
irieger
e56521543b Added a comment: comment 5 2019-11-04 08:30:08 +00:00
hobbes@b2cacef69071743c3a831e60511062f7e014e52f
6c9f7f677b Added a comment: Ssh config 2019-11-03 23:49:05 +00:00
Ilya_Shlyakhter
59fa021c3d added todo re: git-annex-sync handling of linked worktrees 2019-11-03 23:06:39 +00:00
yarikoptic
ced983d4bf Added a comment: the issue persists 2019-11-01 18:12:27 +00:00
plasmastrike@66c1cae4abae92d03faf344111c446a1908981ff
e7962d3701 Added a comment: Yggdrasill 2019-11-01 13:34:36 +00:00
jonjitsu
047e6ce4a3 2019-11-01 11:55:22 +00:00
jonjitsu
32ff08f322 2019-11-01 11:54:28 +00:00
Ilya_Shlyakhter
e64b4c03ee Added a comment: annexing previously non-annexed files, and compatibility with v5 2019-10-31 16:19:40 +00:00
Ilya_Shlyakhter
d578f9ce13 Added a comment: not annexing files already stored in git 2019-10-31 16:03:42 +00:00
Ilya_Shlyakhter
72252e1be4 more notes on communicating breaking changes 2019-10-31 15:52:10 +00:00
bremner
d69d9088a7 Added a comment: plz don't annex already committed files 2019-10-31 13:50:10 +00:00
yarikoptic
b6c3b60a5b added meta tags 2019-10-31 12:55:54 +00:00
yarikoptic
e756f8d2a8 initial report on an error message from addurl 2019-10-31 12:55:03 +00:00
ghen1
449568e5dc Added a comment 2019-10-30 22:51:27 +00:00
Joey Hess
d3e4de0175
fix test suite
The test suite found a bug; select_ can fail now because a uniqueness
constrain has been added.

Now the test suite passes.

Also, I'm satisfied the changed PersistField instances work.
Looking over what changed, and what I've already tested, Key, FilePath,
and InodeCache are known working; ContentIdentifier is trivial
ByteString to blob; and SSha is trivial String to varchar. Both are
tested by the test suite. I've also tested the new FileSize and
EpochTime instances already, and they work.
2019-10-30 15:51:37 -04:00
Joey Hess
d7ecac66d2
notes 2019-10-30 15:31:16 -04:00
Joey Hess
4940a135af
eliminate raw sql LIKE query 2019-10-30 15:19:52 -04:00
Joey Hess
09c7cbbaa8
update for things already fixed in this branch 2019-10-30 13:57:22 -04:00
Joey Hess
9085a2cfec
make sure all sqlite selects have indexes
Bearing in mind that these indexes are really uniqueness constraints
that just happen to also make sqlite generate indexes.

In Database.ContentIndentifier, the ContentIndentifiersKeyRemoteCidIndex
is fine as a uniqueness constraint because it contains all rows from the
table. The ContentIndentifiersCidRemoteIndex is also ok because there
can only be one key for a given (cid, uuid) combination.

In Database.Export, the new ExportTreeFileKeyIndex is the same pair as
the old ExportTreeKeyFileIndex (previously ExportTreeIndex). And
in Database.Keys.SQL, the new InodeCacheKeyIndex is the same pair as the
old KeyInodeCacheIndex.
2019-10-30 13:46:52 -04:00
Joey Hess
3732f27722
document indexes
This was really confusing, though the code was ok. I think I now
understand it fully again.
2019-10-30 13:28:00 -04:00
Joey Hess
f6cfb84dfe
rename row 2019-10-30 13:22:41 -04:00
Joey Hess
aa27969e55
improve layout 2019-10-29 17:08:36 -04:00
Joey Hess
e8437ae7a3
removed now unused import 2019-10-29 17:07:15 -04:00
Joey Hess
61c9b0945d
bump version, though there is no upgrade path yet
I just don't want this branch to accidentially run in my production repos yet.
2019-10-29 17:06:35 -04:00
Joey Hess
c35a9047d3
improve data types for sqlite
This is a non-backwards compatable change, so not suitable for merging
w/o a annex.version bump and transition code. Not yet tested.

This improves performance of git-annex benchmark --databases
across the board by 10-25%, since eg Key roundtrips as a ByteString.

(serializeKey' produces a lazy ByteString, so there is still a
copy involved in converting it to a strict ByteString. It may be faster
to switch to using bytestring-strict-builder.)

FilePath and Key are both stored as blobs. This avoids mojibake in some
situations. It would be possible to use varchar instead, if persistent
could avoid converting that to Text, but it seems there is no good
way to do so. See doc/todo/sqlite_database_improvements.mdwn

Eliminated some ugly artifacts of using Read/Show serialization;
constructors and quoted strings are no longer stored in sqlite.

Renamed SRef to SSha to reflect that it is only ever a git sha,
not a ref name. Since it is limited to the characters in a sha,
it is not affected by mojibake, so still uses String.
2019-10-29 17:05:36 -04:00
Joey Hess
e1b21a0491
benchmark: Add --databases to benchmark sqlite databases
Rescued from commit 11d6e2e260 which removed
db benchmarks in favor of benchmarking arbitrary git-annex commands. Which
is nice and general, but microbenchmarks are useful too.
2019-10-29 17:05:10 -04:00
Dwk
ee841249c0 Added a comment 2019-10-29 21:04:42 +00:00
Joey Hess
25f912de5b
benchmark: Add --databases to benchmark sqlite databases
Rescued from commit 11d6e2e260 which removed
db benchmarks in favor of benchmarking arbitrary git-annex commands. Which
is nice and general, but microbenchmarks are useful too.
2019-10-29 16:59:27 -04:00
strmd
9f00137022 Added a comment 2019-10-29 20:01:01 +00:00
yarikoptic
cc62ba1da8 Added a comment: ACLs keep fighting back 2019-10-29 18:03:57 +00:00
Simon
cf03187f53 removed 2019-10-29 15:54:51 +00:00
Simon
a072c8a551 Added a comment: What is a recent enough version? 2019-10-29 15:51:55 +00:00
Ilya_Shlyakhter
5be902b324 Added a comment: re: setkey 2019-10-28 22:12:51 +00:00