Commit graph

28 commits

Author SHA1 Message Date
Joey Hess
f08cd6a4ac
set S3 version id in retrieveExportWithContentIdentifierS3
This is necessary because of checks for a S3 version id being set
done when deleting the export or overwriting or renaming it.
2019-04-24 15:13:07 -04:00
Joey Hess
2d0dd34916
initial work toward correctly merging deeper import histories
Pure code is tested working, including with even histories that merge
several lines of development. Needs to be hooked up to git histories
next.
2019-04-23 16:34:19 -04:00
Joey Hess
29705d83f4
convert History to use Set
This way the Ord instance doesn't care what order parent
Histories come in.
2019-04-23 15:08:37 -04:00
Joey Hess
833980c0bc
indicate when an old version of a file is being imported 2019-04-19 15:05:08 -04:00
Joey Hess
f95f340c73
sync: When listing contents on an import remote fails, proceed with other syncing instead of aborting
Switch listContents to being a proper CommandStart, so if it throws an
exception, it will be treated like any other command action that fails.

downloadImport apparently does not ever throw an exception,
and itself uses commandAction, so it can't be a CommandStart.
2019-04-10 17:02:56 -04:00
Joey Hess
c714a260a9
include remote name for consistency with export output 2019-03-11 14:46:37 -04:00
Joey Hess
e1fdc8b374
record only subtree in export db and log after importing a subtree 2019-03-11 13:45:01 -04:00
Joey Hess
6149a3b9bd
added progress bars
clean up old todo item I checked earlier, see commit
dec30d2b14
2019-03-08 12:43:03 -04:00
Joey Hess
e412129523
concurrency and status messages when downloading from import 2019-03-08 12:33:44 -04:00
Joey Hess
e3a704224f
fix export db locking deadlock 2019-03-07 16:06:02 -04:00
Joey Hess
4efd431136
remove obsolete TODO
updateExportDb runs addExportedLocation
2019-03-07 15:11:24 -04:00
Joey Hess
71fec9060c
move 2019-03-07 12:56:40 -04:00
Joey Hess
68d1661251
cross-repo import now working correctly 2019-03-07 12:31:35 -04:00
Joey Hess
ee251b2e2e
implement updating the ContentIdentifier db with info from the git-annex branch
untested

This won't be super slow, but it does need to diff two likely large
trees, and since the git-annex branch rarely sits still, it will most
likely be run at the beginning of every import.

A possible speed improvement would be to only run this when the database
did not contain a ContentIdentifier. But that would only speed up
imports when there is no new version of a file on the special remote,
at most renames of existing files being imported.

A better speed improvement would be to record something in the git-annex
branch that indicates when an import has been run, and only do the diff
if the git-annex branch has record of a newer import than we've seen
before. Then, it would only run when there is in fact new
ContentIdentifier information available from a remote. Certianly doable,
but didn't want to complicate things yet.
2019-03-06 18:04:30 -04:00
Joey Hess
00722ba1f8
lock before writing to the ContentIdentifier db 2019-03-04 16:47:30 -04:00
Joey Hess
aaacf431d8
handle importtree=yes config
For now, it's only allowed when exporttree=yes is also set.
That simplified the implementation, but could later be changed if
there's a remote that makes sense to be an import but not an export.
However, it may work just as well to make a remote be readonly to
prevent export to it while still allowing import.
2019-03-04 16:07:35 -04:00
Joey Hess
3cd19fb4d0
use InodeCache to avoid races in import from directory special remote
This does not avoid all possible races, but it does avoid all likely
ones, and is demonstratably better than git's own handling of races
where files get modified at the same time as it's updating the working
tree.

The main thing this won't detect are not unlikely races where part
of a file gets changed while it's being copied and then the file is
restored to its original condition before the modification check.
No, it's more likely that the limitations of checking inode, size,
and mtime won't detect certian modifications, involving eg mmapped
files.
2019-03-04 13:57:23 -04:00
Joey Hess
519cadd1de
refactor RemoteTrackingBranch
Not specific to Import; export will use it too.
2019-03-01 14:47:56 -04:00
Joey Hess
1c8793691a
import: update location log for removed files 2019-03-01 13:26:59 -04:00
Joey Hess
d0066d9a87
fully update export db during import
This makes exporting immediately after import and merge be a no-op.
2019-02-27 15:29:41 -04:00
Joey Hess
b1f10fbb4d
update location log during import 2019-02-27 13:58:03 -04:00
Joey Hess
45aacd888b
import downloader complete (untested)
Made some api changes.

listImportableContents needs to provide the size
of the data, so the downloader can check disk free space.

retrieveExportWithContentIdentifier is passed the filepath to write to

Use temporary "CID" key during download of a ContentIdentifier from a
remote, so withTmp can be used and then move the content to the real key
once it's known.
2019-02-27 13:15:02 -04:00
Joey Hess
f4b773e9a1
incomplete action to download files from import 2019-02-26 15:25:28 -04:00
Joey Hess
e4e464da65
import command is updating tracking branch 2019-02-26 13:15:48 -04:00
Joey Hess
d805401708
fairly happy withbuildImportCommit now
still not yet tested
2019-02-23 15:47:55 -04:00
Joey Hess
33bb62ff13
fix parent 2019-02-22 12:44:22 -04:00
Joey Hess
bab6c570b0
buildImportTrees is fully working
buildImportCommit not yet tested
2019-02-22 12:41:17 -04:00
Joey Hess
8fdea8f444
WIP
Added graftTree but it's buggy.

Should use graftTree in Annex.Branch.graftTreeish; it will be faster
than the current implementation there.

Started Annex.Import, but untested and it doesn't yet handle tree
grafting.
2019-02-21 17:32:59 -04:00