Commit graph

34184 commits

Author SHA1 Message Date
Joey Hess
bc509143e5
avoid opening export db until needed
Before, it was opened when constructing the export Remote, even if it
never got used.
2019-03-04 18:11:32 -04:00
Joey Hess
cd3a2b023a
initial try at using storeExportWithContentIdentifier
Untested, and I'm not sure about the locking of the ContentIdentifier db.
2019-03-04 17:50:41 -04:00
Joey Hess
b67fa2180e
add getExportedKey
Not optimised because that would need transition code to be written for
existing export datbases.
2019-03-04 17:27:10 -04:00
Joey Hess
138d07eb97
add getContentIdentifiers
Changed the database schema for this, with a new index.
2019-03-04 16:48:07 -04:00
Joey Hess
00722ba1f8
lock before writing to the ContentIdentifier db 2019-03-04 16:47:30 -04:00
Joey Hess
aaacf431d8
handle importtree=yes config
For now, it's only allowed when exporttree=yes is also set.
That simplified the implementation, but could later be changed if
there's a remote that makes sense to be an import but not an export.
However, it may work just as well to make a remote be readonly to
prevent export to it while still allowing import.
2019-03-04 16:07:35 -04:00
Joey Hess
5f17a9cc50
docs for importtree config 2019-03-04 15:39:19 -04:00
Joey Hess
88ccfaa78c
storeExportWithContentIdentifierM for directory special remote
Not sure if my reasoning about the races really holds.

It would certianly be possible to better guard against races by using
Linux-specific renameat2 with RENAME_EXCHANGE or RENAME_NOREPLACE.

Or by using link and relying on it not overwriting existing files -- but
that would need a filesystem that supports hard links and directory can
be used in filesystems that don't.
2019-03-04 14:46:25 -04:00
Joey Hess
1ec9e1494c
use relatedTempate in viaTmp 2019-03-04 14:12:00 -04:00
Joey Hess
3cd19fb4d0
use InodeCache to avoid races in import from directory special remote
This does not avoid all possible races, but it does avoid all likely
ones, and is demonstratably better than git's own handling of races
where files get modified at the same time as it's updating the working
tree.

The main thing this won't detect are not unlikely races where part
of a file gets changed while it's being copied and then the file is
restored to its original condition before the modification check.
No, it's more likely that the limitations of checking inode, size,
and mtime won't detect certian modifications, involving eg mmapped
files.
2019-03-04 13:57:23 -04:00
Joey Hess
51fc969b66
notes 2019-03-01 16:44:34 -04:00
Joey Hess
18d7a1dbbb
make export and sync update special remote tracking branch
The branch is only updated once the export is 100% complete. This way,
if an export is started but interrupted and so the remote does not yet
contain some of the files, an import will make a commit on the old
branch, and so won't delete the missing files.
2019-03-01 16:35:48 -04:00
Joey Hess
519cadd1de
refactor RemoteTrackingBranch
Not specific to Import; export will use it too.
2019-03-01 14:47:56 -04:00
Joey Hess
d28b0a8bd0
use disconnected history for import tracking branch
This avoids the first merge from it deleting all files in the current
branch, which was very surpring and unwanted behavior.
2019-03-01 14:33:29 -04:00
Joey Hess
740f957cef
new problem 2019-03-01 13:49:26 -04:00
Joey Hess
1c8793691a
import: update location log for removed files 2019-03-01 13:26:59 -04:00
Joey Hess
5402c39882
plan for item 2019-03-01 12:56:20 -04:00
Joey Hess
7acee61adf
rename 2019-03-01 12:50:33 -04:00
Joey Hess
a3f6e07fec
todo 2019-02-27 15:56:32 -04:00
Joey Hess
d0066d9a87
fully update export db during import
This makes exporting immediately after import and merge be a no-op.
2019-02-27 15:29:41 -04:00
Joey Hess
b1f10fbb4d
update location log during import 2019-02-27 13:58:03 -04:00
Joey Hess
775e6ed86f
fix table name 2019-02-27 13:52:56 -04:00
Joey Hess
e2e57f8556
initial export support for directory special remote
This does not guard against race condition yet, it's only for testing
purposes.
2019-02-27 13:42:34 -04:00
Joey Hess
45aacd888b
import downloader complete (untested)
Made some api changes.

listImportableContents needs to provide the size
of the data, so the downloader can check disk free space.

retrieveExportWithContentIdentifier is passed the filepath to write to

Use temporary "CID" key during download of a ContentIdentifier from a
remote, so withTmp can be used and then move the content to the real key
once it's known.
2019-02-27 13:15:02 -04:00
Joey Hess
f4b773e9a1
incomplete action to download files from import 2019-02-26 15:25:28 -04:00
Joey Hess
b6e2a5e9c2
reorg 2019-02-26 14:22:08 -04:00
Joey Hess
e4e464da65
import command is updating tracking branch 2019-02-26 13:15:48 -04:00
Joey Hess
5afe4135c2
import --from option parsing 2019-02-26 12:06:19 -04:00
Joey Hess
760f26ebc6
Merge branch 'master' into importtree 2019-02-26 11:36:36 -04:00
sameerds
22e9a49bd9 Added a comment: +1 2019-02-24 17:06:30 +00:00
Joey Hess
413c101850
Merge branch 'master' of ssh://git-annex.branchable.com 2019-02-24 12:45:32 -04:00
Joey Hess
19f833b0b1
aws-0.21.1
* S3: Support enabling bucket versioning when built with aws-0.21.1.
* stack.yaml: Build with aws-0.21.1
2019-02-24 12:45:09 -04:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
ea3e6322e7 Added a comment 2019-02-24 07:01:35 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
1202a2a48c removed 2019-02-24 06:59:21 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
24ebc3d12e Added a comment 2019-02-24 06:56:43 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
8c777cb36d removed 2019-02-24 06:43:34 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
eb6c0608e6 Added a comment 2019-02-24 06:41:01 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
3f450e5271 Added a comment 2019-02-24 01:20:15 +00:00
Joey Hess
89ee156059
Merge branch 'master' of ssh://git-annex.branchable.com 2019-02-23 16:04:15 -04:00
Joey Hess
15a4f91d07
devblog 2019-02-23 16:03:29 -04:00
Joey Hess
7072f7a071
note 2019-02-23 15:57:18 -04:00
Joey Hess
4747fa923d
export: Deprecated the --tracking option.
Instead, users can configure remote.<name>.annex-tracking-branch themselves.
2019-02-23 15:54:33 -04:00
Joey Hess
d805401708
fairly happy withbuildImportCommit now
still not yet tested
2019-02-23 15:47:55 -04:00
Joey Hess
5bac8babdb
doc updated for import tree
Deprecated git annex export --tracking because it makes sense to have a
single configuration of tracking for both imports and exports.
2019-02-23 15:46:03 -04:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
663c62d7ce 2019-02-23 18:38:42 +00:00
Joey Hess
464485bffe
Merge branch 'master' into importtree 2019-02-23 13:58:22 -04:00
Joey Hess
d685b119df
more design 2019-02-23 13:55:26 -04:00
Joey Hess
c5ceefdca6
Merge branch 'master' into importtree 2019-02-22 22:03:00 -04:00
Joey Hess
3c405838f8
more design 2019-02-22 22:02:50 -04:00
Joey Hess
d65a78ff5b
Fix cleanup of git-annex:export.log after git-annex forget --drop-dead
This log, unlike all other current top-level logs, is a new format log.

I have not checked what throwing it at the old log parser did, but it seems
likely it ignored unparsable lines, and so perhaps deleted all lines from
the log.
2019-02-22 21:34:31 -04:00