Commit graph

2944 commits

Author SHA1 Message Date
Joey Hess
81a38df5a7
add missing CHECKURL-FAILURE ErrorMsg to docs 2020-09-18 11:58:18 -04:00
Joey Hess
05b2b46a82
async extension done 2020-08-14 15:24:34 -04:00
Joey Hess
198b709561
switch to TMVars for thread safety when using the async extension
TVars were not updated atomically, which was ok when each thread got its
own External that was the only thing using these TVars. But, with the
async extension, several External instances can share the same var, so
it needs to be a TMVar to avoid read/write conflicts.

In particular, this makes PREPARE only be sent once.
2020-08-14 14:50:09 -04:00
Joey Hess
7da2d4dd2d
one jobid per thread
And, relay ERROR on to all listening threads.
2020-08-14 14:24:46 -04:00
Joey Hess
72561563d9
rethought the async protocol some more
Moving jobid generation to the git-annex side lets it be simplified a
lot.

Note that it will also be possible to generate one jobid per connection,
rather than a new job per request. That will make overflow not an issue,
and will avoid some work, and will simplify some of the code.
2020-08-13 20:18:06 -04:00
Joey Hess
7546e686a2
async proto basically working
Simplified the protocol by removing END-ASYNC.

There's a STM crash when a non-async protocol message is sent, which
needs to be fixed.
2020-08-13 15:52:12 -04:00
Joey Hess
15706e6991
relayer receive loop is done
Receive loop looks right. Still need the send loop.

And, a complication is that some messages git-annex
sends need to be wrapped in REPLY_ASYNC, while others
do not. So will probably need to split externalSend
into two.
2020-08-12 15:56:58 -04:00
Joey Hess
7a21492f49
rethought this protocol again
Now that I've started implementation, I see it's really necessary that
every message the special remote sends use the protocol, otherwise
nasty edge cases abound.
2020-08-12 15:12:09 -04:00
Joey Hess
ddf69bf5b8
draft async extension 2020-08-11 16:42:09 -04:00
Joey Hess
13db029ac0
comment 2020-07-31 16:11:10 -04:00
Ilya_Shlyakhter
23d15dc985 Added a comment: key length 2020-07-30 15:13:53 +00:00
Joey Hess
ea63d1dfe3
E variant of external backend keys 2020-07-29 17:12:22 -04:00
Joey Hess
b6fa4cb42f
external backends genKey and verifyKeyContent implemented
Only key translation for HasExt remains..
2020-07-29 16:36:32 -04:00
Joey Hess
f75be32166
external backends wip
It's able to start them up, the only thing not implemented is generating
and verifying keys. And, the key translation for HasExt.
2020-07-29 15:23:18 -04:00
Joey Hess
b5d6a36db0
note external backend names cannot end with E
also markdown improvements
2020-07-29 14:01:49 -04:00
Joey Hess
3953c7a0ce
add DEBUG 2020-07-27 11:31:00 -04:00
Joey Hess
1ccb6699a1
guidance on size and mtime fields 2020-07-20 19:56:47 -04:00
Joey Hess
2630bb766a
fix link 2020-07-20 14:25:24 -04:00
Joey Hess
8b0fbc1391
formatting 2020-07-20 14:23:01 -04:00
Joey Hess
d1300eca2e
draft external backend protocol 2020-07-20 14:05:49 -04:00
Joey Hess
c016527cb5
link 2020-07-03 14:40:13 -04:00
Joey Hess
28fee95d4e
extend proposed interface with IMPORTKEY 2020-07-03 14:23:04 -04:00
Joey Hess
424b1912d6
followup and add link 2020-07-01 12:28:44 -04:00
Joey Hess
73c9331b15
remove obsolete links 2020-06-30 13:41:18 -04:00
Joey Hess
a49d300545
async exception safety for external special remote processes
Since an external process can be in the middle of some operation when an
async exception is received, it has to be shut down then. Using
cleanupProcess will close its IO handles and send it a SIGTERM.

If a special remote choses to catch SIGTERM, it's fine for it to do some
cleanup then, but until it finishes, git-annex will be blocked waiting
for it. If a special remote blocked SIGTERM, it would cause a hang.
Mentioned in docs.

Also, in passing, fixed a FD leak, it was not closing the error handle
when shutting down the external. In practice that didn't matter before because
it was only run when git-annex was itself shutting down, but now that it
can run on exception, it would have been a problem.
2020-06-09 12:22:14 -04:00
Joey Hess
2de40ba4da
improve docs around uri claiming 2020-05-21 11:58:57 -04:00
Soxofaan
ffafed4d82 Fix typo remotes.log -> remote.log 2020-03-28 13:36:45 +00:00
Joey Hess
c827ddcdb6
typo 2020-03-25 13:20:24 -04:00
Joey Hess
357f783edf
add priority tags 2020-03-25 13:05:09 -04:00
Joey Hess
d9bd645bf8
clarify LISTCONFIGS
Apparently some externals use GETCONFIG name, and the documentation as
written can be read to allow it, although IIRC it was not really my
intent to. If someone does do this, they should not include it in
LISTCONFIGS, since git-annex already tracks its own config settings.
2020-03-09 12:34:13 -04:00
Joey Hess
14dfd297a4
buglist template 2020-03-05 12:34:34 -04:00
Joey Hess
74694ece9b
pagespecs are absolute 2020-03-05 12:33:48 -04:00
Joey Hess
6ed6782c5d
exclude ones tagged needsthought 2020-03-05 12:31:28 -04:00
Joey Hess
f58a87f194
convert roadmap to list todo's that have been tagged confirmed 2020-03-05 12:29:19 -04:00
Joey Hess
a5c5a94bdf
remove done item 2020-03-05 12:26:05 -04:00
Joey Hess
99cb3e75f1
add LISTCONFIGS to external special remote protocol
Special remote programs that use GETCONFIG/SETCONFIG are recommended
to implement it.

The description is not yet used, but will be useful later when adding a way
to make initremote list all accepted configs.

configParser now takes a RemoteConfig parameter. Normally, that's not
needed, because configParser returns a parter, it does not parse it
itself. But, it's needed to look at externaltype and work out what
external remote program to run for LISTCONFIGS.

Note that, while externalUUID is changed to a Maybe UUID, checkExportSupported
used to use NoUUID. The code that now checks for Nothing used to behave
in some undefined way if the external program made requests that
triggered it.

Also, note that in externalSetup, once it generates external,
it parses the RemoteConfig strictly. That generates a
ParsedRemoteConfig, which is thrown away. The reason it's ok to throw
that away, is that, if the strict parse succeeded, the result must be
the same as the earlier, lenient parse.

initremote of an external special remote now runs the program three
times. First for LISTCONFIGS, then EXPORTSUPPORTED, and again
LISTCONFIGS+INITREMOTE. It would not be hard to eliminate at least
one of those, and it should be possible to only run the program once.
2020-01-17 16:07:17 -04:00
Joey Hess
925b2acb34
some improvements to docs 2019-11-11 12:52:45 -04:00
Joey Hess
f07d61c376
typo 2019-11-11 12:39:51 -04:00
Joey Hess
e42eb04182
fix language 2019-11-11 12:36:57 -04:00
Joey Hess
14f2c1eb92
typo 2019-11-11 12:36:05 -04:00
giuly.ippoliti@c1e2f0d5e40b128900f94f3d107d3719f87c3ff7
3d0330a555 Added a comment: Triggering which stage and when? 2019-10-24 20:30:11 +00:00
jason.dixon.email@aa0e536a2ec2877d6f666108dbbc6e39bbe87ac0
ccdb34b181 Added a comment: Friendly bump to keep on the radar 2019-10-24 09:26:23 +00:00
branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e
3f609a7742 Added a comment: Commits could be rate-limited too 2019-09-29 23:03:47 +00:00
Joey Hess
56669728d7
response 2019-09-28 12:51:58 -04:00
branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e
27b003eff9 Added a comment: Isn't this done? 2019-09-28 12:39:52 +00:00
yarikoptic
6dc5a0574e Added a comment: should there be FSCK and/or CLEAN? 2019-09-28 00:31:51 +00:00
Joey Hess
8465b6abd6
update 2019-09-18 12:16:33 -04:00
Joey Hess
0c718763fd
planning
several entangled things
2019-08-26 12:29:43 -04:00
Joey Hess
4923a3f790
update 2019-07-22 13:54:43 -04:00
Joey Hess
6e51b9ae88
clarify 2019-06-04 21:49:53 -04:00
Joey Hess
b6dfcf529e
indent 2019-05-28 16:21:35 -04:00
Joey Hess
08aa8e2010
break out export and import appending
The import protocol is WiP
2019-05-28 16:16:17 -04:00
Joey Hess
4818e3606c
reorg special remote docs
Made responses to git-annex requests be listed under each request.

This did lead to a little duplication since some replies are used for 2
requests, but it also makes it much clearer and easier to see how the
protocol works.

And, it makes each request self-contained, so they can be split out into
separate pages.
2019-05-28 14:01:19 -04:00
Joey Hess
63fa6b4a44
fix wrong statement 2019-05-28 12:05:49 -04:00
Joey Hess
c1ed0293b0
improve docs about removeExportDirectory 2019-05-28 11:16:01 -04:00
Joey Hess
2353d80906
remove done item 2019-05-23 14:24:25 -04:00
Joey Hess
428435c37c
comment 2019-05-14 15:30:38 -04:00
michael.hanke@c60e12358aa3fc6060531bdead1f530ac4d582ec
53bee6dc5f Added a comment: Re: Empty lines sent by git-annex to an external special remote 2019-05-14 08:48:27 +00:00
michael.hanke@c60e12358aa3fc6060531bdead1f530ac4d582ec
c4abfcc368 Added a comment: Empty lines sent by git-annex to an external special remote 2019-05-14 06:36:12 +00:00
Joey Hess
ae562ad4d7
update old todo item with what still needs doing
removed old comments that are no longer relevant
2019-05-10 13:52:40 -04:00
Joey Hess
459bbd9005
update roadmap 2019-05-10 12:35:14 -04:00
Joey Hess
5a570da1ab
thoughts 2019-04-09 15:50:52 -04:00
Joey Hess
2dc20e3fa4
update design doc with final design choices 2019-04-09 13:05:22 -04:00
Joey Hess
b51eceb326
reorg section and expand
conflict detection at import time is not detected, but I think it's ok
given this reasoning
2019-04-09 12:59:14 -04:00
Joey Hess
1b5f4a8e20
comment 2019-04-03 13:11:34 -04:00
Ilya_Shlyakhter
d40b2e879f Added a comment: re: key content size 2019-04-01 17:44:29 +00:00
driusan@4d47e7deeb2f5d3846792d049ed06f96a0c3ca98
eb00b3f6da removed 2019-03-31 16:31:47 +00:00
driusan@4d47e7deeb2f5d3846792d049ed06f96a0c3ca98
7879bbeafd Added a comment: key content size 2019-03-31 16:31:28 +00:00
driusan@4d47e7deeb2f5d3846792d049ed06f96a0c3ca98
f250bb092d Added a comment: key content size 2019-03-31 16:31:09 +00:00
89.200.13.45
b7512c1146 poll vote (OpenStack SWIFT) 2019-03-23 09:08:09 +00:00
Joey Hess
2912429640
better indicate when special remotes do not support renameExport
Avoid a warning message when renameExport is not supported, and just
fallback to deleting with a subsequent re-upload. Especially needed for
importtree remotes, where renameExport needs to be disabled.

This changes the external special remote protocol, but in a
backwards-compatible way. A reply of UNSUPPORTED-REQUEST to an older
version of git-annex will cause it to make renameExport return False.
2019-03-11 12:53:24 -04:00
Joey Hess
dec30d2b14
updates
Note that I tried an evil remote that lists ImportLocations with
../../../ in them and indeed this resulted in git blowing up and the
import failing, and not writing outside the repo.
2019-03-06 17:07:36 -04:00
Joey Hess
46d33e804a
added checkPresentExportWithContentIdentifier
Ugh, don't like needing to add this, but I can't see a way around it.
2019-03-05 16:03:03 -04:00
Joey Hess
8c54604e67
import+export from directory special remote fully working
Had to add two more API calls to override export APIs that are not safe
for use in combination with import.

It's unfortunate that removeExportDirectory is documented to be allowed
to remove non-empty directories. I'm not entirely sure why it's that
way, my best guess is it was intended to make it easy to implement with
just rm -rf.
2019-03-05 14:20:14 -04:00
Joey Hess
464485bffe
Merge branch 'master' into importtree 2019-02-23 13:58:22 -04:00
Joey Hess
d685b119df
more design 2019-02-23 13:55:26 -04:00
Joey Hess
c5ceefdca6
Merge branch 'master' into importtree 2019-02-22 22:03:00 -04:00
Joey Hess
3c405838f8
more design 2019-02-22 22:02:50 -04:00
Joey Hess
4e0d08b66b
Merge branch 'master' into importtree 2019-02-22 21:18:13 -04:00
Joey Hess
200dc632f5
more design 2019-02-22 21:18:01 -04:00
Joey Hess
8c836623b7
design work 2019-02-22 16:18:09 -04:00
62.226.58.176
8428a74a0e poll vote (My phone (or MP3 player)) 2019-02-21 18:36:04 +00:00
Joey Hess
a66ab6a9eb
Merge branch 'master' of ssh://git-annex.branchable.com 2019-02-20 17:36:14 -04:00
Joey Hess
0442842622
add import tree interface to Remote 2019-02-20 15:35:22 -04:00
Joey Hess
1a29d64bec
leave room for future expansion 2019-02-20 13:03:09 -04:00
Joey Hess
5af0876592
fix 2019-02-20 12:51:50 -04:00
Joey Hess
d128c8c3ec
add design document for import tree 2019-02-20 12:12:32 -04:00
crest
4bd139c5b6 poll vote (My phone (or MP3 player)) 2019-02-20 10:49:21 +00:00
Joey Hess
8230b62e06
add todo 2019-01-17 11:49:56 -04:00
Joey Hess
3003e74a9d
inconclusive thoughts 2019-01-16 15:05:37 -04:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
ab1a78115a removed 2019-01-15 15:47:58 +00:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
91c7936ca7 Added a comment: PREPARE-LOCAL 2019-01-15 15:47:39 +00:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
59269fdd59 Added a comment: PREPARE-LOCAL 2019-01-15 15:46:07 +00:00
Joey Hess
4148548084
roadmap update 2019-01-11 17:25:37 -04:00
Joey Hess
3a9a6bc56b
response 2018-12-03 12:59:42 -04:00
anarcat
14213c0e42 Added a comment: how about for regular key/value storage? 2018-12-01 17:42:18 +00:00
Joey Hess
44c0a08429
add back deltas
also switch all urls to https
2018-12-01 12:13:46 -04:00
Joey Hess
82aad90d33
update roadmap 2018-11-08 15:42:53 -04:00
Joey Hess
6ecd55a9fa
Fixed some other potential hangs in the P2P protocol
Finishes the start made in 983c9d5a53, by
handling the case where `transfer` fails for some other reason, and so the
ReadContent callback does not get run. I don't know of a case where
`transfer` does fail other than the locking dealt with in that commit, but
it's good to have a guarantee.

StoreContent and StoreContentTo had a similar problem.
Things like `getViaTmp` may decide not to run the transfer action.
And `transfer` could certianly fail, if another transfer of the same
object was in progress. (Or a different object when annex.pidlock is set.)

If the transfer action was not run, the content of the object would
not all get consumed, and so would get interpreted as protocol commands,
which would not go well.

My approach to fixing all of these things is to set a TVar only
once all the data in the transfer is known to have been read/written.
This way the internals of `transfer`, `getViaTmp` etc don't matter.

So in ReadContent, it checks if the transfer completed.
If not, as long as it didn't throw an exception, send empty and Invalid
data to the callback. On an exception the state of the protocol is unknown
so it has to raise ProtoFailureException and close the connection,
same as before.

In StoreContent, if the transfer did not complete
some portion of the DATA has been read, so the protocol is in an unknown
state and it has to close the conection as well.

(The ProtoFailureMessage used here matches the one in Annex.Transfer, which
is the most likely reason. Not ideal to duplicate it..)

StoreContent did not ever close the protocol connection before. So this is
a protocol change, but only in an exceptional circumstance, and it's not
going to break anything, because clients already need to deal with the
connection breaking at any point.

The way this new behavior looks (here origin has annex.pidlock = true so will
only accept one upload to it at a time):

git annex copy --to origin -J2
copy x (to origin...) ok
copy y (to origin...)
  Lost connection (fd:25: hGetChar: end of file)

This work is supported by the NIH-funded NICEMAN (ReproNim TR&D3) project.
2018-11-06 14:52:32 -04:00
Joey Hess
3beab402e6
response 2018-10-29 15:40:52 -04:00