Commit graph

25473 commits

Author SHA1 Message Date
Joey Hess
5767b1b00d
avoid updating tracking branch when transfer to export throws exception 2019-03-05 16:51:13 -04:00
Joey Hess
dc278c059c
fix STM crash
git-annex: thread blocked indefinitely in an STM transaction
failed

git-annex: sqlite query crashed
CallStack (from HasCallStack):
  error, called at ./Database/Handle.hs:98:42 in main:Database.Handle
failed

This needs further investigation.
2019-03-05 16:37:40 -04:00
Joey Hess
46d33e804a
added checkPresentExportWithContentIdentifier
Ugh, don't like needing to add this, but I can't see a way around it.
2019-03-05 16:03:03 -04:00
Joey Hess
fd2a1aaa17
avoid using renameExport on import remotes 2019-03-05 14:57:48 -04:00
Joey Hess
9df9a3f82b
more todo 2019-03-05 14:55:22 -04:00
Joey Hess
f7be2e9d37
document how conflicts are handled with imports 2019-03-05 14:55:06 -04:00
Joey Hess
8c54604e67
import+export from directory special remote fully working
Had to add two more API calls to override export APIs that are not safe
for use in combination with import.

It's unfortunate that removeExportDirectory is documented to be allowed
to remove non-empty directories. I'm not entirely sure why it's that
way, my best guess is it was intended to make it easy to implement with
just rm -rf.
2019-03-05 14:20:14 -04:00
Joey Hess
554b7b7f3e
fix todo 2019-03-04 18:20:12 -04:00
Joey Hess
cd3a2b023a
initial try at using storeExportWithContentIdentifier
Untested, and I'm not sure about the locking of the ContentIdentifier db.
2019-03-04 17:50:41 -04:00
Joey Hess
aaacf431d8
handle importtree=yes config
For now, it's only allowed when exporttree=yes is also set.
That simplified the implementation, but could later be changed if
there's a remote that makes sense to be an import but not an export.
However, it may work just as well to make a remote be readonly to
prevent export to it while still allowing import.
2019-03-04 16:07:35 -04:00
Joey Hess
5f17a9cc50
docs for importtree config 2019-03-04 15:39:19 -04:00
Joey Hess
3cd19fb4d0
use InodeCache to avoid races in import from directory special remote
This does not avoid all possible races, but it does avoid all likely
ones, and is demonstratably better than git's own handling of races
where files get modified at the same time as it's updating the working
tree.

The main thing this won't detect are not unlikely races where part
of a file gets changed while it's being copied and then the file is
restored to its original condition before the modification check.
No, it's more likely that the limitations of checking inode, size,
and mtime won't detect certian modifications, involving eg mmapped
files.
2019-03-04 13:57:23 -04:00
Joey Hess
51fc969b66
notes 2019-03-01 16:44:34 -04:00
Joey Hess
18d7a1dbbb
make export and sync update special remote tracking branch
The branch is only updated once the export is 100% complete. This way,
if an export is started but interrupted and so the remote does not yet
contain some of the files, an import will make a commit on the old
branch, and so won't delete the missing files.
2019-03-01 16:35:48 -04:00
Joey Hess
d28b0a8bd0
use disconnected history for import tracking branch
This avoids the first merge from it deleting all files in the current
branch, which was very surpring and unwanted behavior.
2019-03-01 14:33:29 -04:00
Joey Hess
740f957cef
new problem 2019-03-01 13:49:26 -04:00
Joey Hess
1c8793691a
import: update location log for removed files 2019-03-01 13:26:59 -04:00
Joey Hess
5402c39882
plan for item 2019-03-01 12:56:20 -04:00
Joey Hess
a3f6e07fec
todo 2019-02-27 15:56:32 -04:00
Joey Hess
b1f10fbb4d
update location log during import 2019-02-27 13:58:03 -04:00
Joey Hess
e2e57f8556
initial export support for directory special remote
This does not guard against race condition yet, it's only for testing
purposes.
2019-02-27 13:42:34 -04:00
Joey Hess
45aacd888b
import downloader complete (untested)
Made some api changes.

listImportableContents needs to provide the size
of the data, so the downloader can check disk free space.

retrieveExportWithContentIdentifier is passed the filepath to write to

Use temporary "CID" key during download of a ContentIdentifier from a
remote, so withTmp can be used and then move the content to the real key
once it's known.
2019-02-27 13:15:02 -04:00
Joey Hess
f4b773e9a1
incomplete action to download files from import 2019-02-26 15:25:28 -04:00
Joey Hess
e4e464da65
import command is updating tracking branch 2019-02-26 13:15:48 -04:00
Joey Hess
760f26ebc6
Merge branch 'master' into importtree 2019-02-26 11:36:36 -04:00
sameerds
22e9a49bd9 Added a comment: +1 2019-02-24 17:06:30 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
ea3e6322e7 Added a comment 2019-02-24 07:01:35 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
1202a2a48c removed 2019-02-24 06:59:21 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
24ebc3d12e Added a comment 2019-02-24 06:56:43 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
8c777cb36d removed 2019-02-24 06:43:34 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
eb6c0608e6 Added a comment 2019-02-24 06:41:01 +00:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
3f450e5271 Added a comment 2019-02-24 01:20:15 +00:00
Joey Hess
89ee156059
Merge branch 'master' of ssh://git-annex.branchable.com 2019-02-23 16:04:15 -04:00
Joey Hess
15a4f91d07
devblog 2019-02-23 16:03:29 -04:00
Joey Hess
7072f7a071
note 2019-02-23 15:57:18 -04:00
Joey Hess
5bac8babdb
doc updated for import tree
Deprecated git annex export --tracking because it makes sense to have a
single configuration of tracking for both imports and exports.
2019-02-23 15:46:03 -04:00
david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8
663c62d7ce 2019-02-23 18:38:42 +00:00
Joey Hess
464485bffe
Merge branch 'master' into importtree 2019-02-23 13:58:22 -04:00
Joey Hess
d685b119df
more design 2019-02-23 13:55:26 -04:00
Joey Hess
c5ceefdca6
Merge branch 'master' into importtree 2019-02-22 22:03:00 -04:00
Joey Hess
3c405838f8
more design 2019-02-22 22:02:50 -04:00
Joey Hess
4e0d08b66b
Merge branch 'master' into importtree 2019-02-22 21:18:13 -04:00
Joey Hess
200dc632f5
more design 2019-02-22 21:18:01 -04:00
Joey Hess
8c836623b7
design work 2019-02-22 16:18:09 -04:00
Joey Hess
d7e5a884f7
response 2019-02-22 16:03:19 -04:00
gan
5b7daecbd3 removed 2019-02-22 18:03:32 +00:00
gan
776916d5fe Added a comment: Clarification 2019-02-22 18:03:16 +00:00
gan
9e38dfd700 Added a comment: Clarification 2019-02-22 18:03:03 +00:00
gan
baa1699570 Added a comment: Provide flags to youtube-dl? 2019-02-22 18:01:26 +00:00
Joey Hess
d67494716c
Merge branch 'master' of ssh://git-annex.branchable.com 2019-02-21 17:46:09 -04:00