Commit graph

38994 commits

Author SHA1 Message Date
Joey Hess
1a361dca37
devblog 2020-12-09 16:20:57 -04:00
Joey Hess
04c12aa6df
custom protocol for transferrer
Rather than using Read/Show, which would force me to preserve data types
into the future.

I considered just deriving json and sending that, but I don't much like
deriving json with data types that have named constructors (like Key
does) because again it locks in data type details.

So instead, used SimpleProtocol, with a fairly complex and unreadable
protocol. But it is as efficient as the p2p protocol at least, and as
future proof.

(Writing my own custom json instances would have worked but I thought
of it too late and don't want to do all the work twice. The only real
benefit might be that aeson could be faster.)

Note that, when a new protocol request type is added later, git-annex
trying to use it will cause the git-annex transferrer to display a
protocol error message. That seems ok; it would only happen if a new
git-annex found an old version of itself in PATH or the program
file. So it's unlikely, and all it can do anyway is display an error.
(The error message could perhaps be improved..)

This commit was sponsored by Jack Hill on Patreon.
2020-12-09 16:13:59 -04:00
Joey Hess
447d798987
export encode_c' 2020-12-09 15:28:45 -04:00
Joey Hess
19777d1c6f
minor improvements
Adding new instance for Integer, and some parsers for more parameters.

The conversion of readish to readMaybe is done because a serialized
exit code cannot contain additional text after the number.
2020-12-09 15:28:11 -04:00
Joey Hess
7e25c643cf
avoid P2P.Protocol defining instance Proto.Serializable AssociatedFile
It's got its own specific hacks, so it needs it own specific data type.
2020-12-09 13:46:42 -04:00
Joey Hess
004a4f5fb1
factor out Types.Transferrer 2020-12-09 13:28:49 -04:00
Joey Hess
677003a6df
rename helper
More consistent name with TransferrerPool
2020-12-09 13:24:24 -04:00
Joey Hess
3fa2bc2eed
rename back, there were links to this 2020-12-09 13:14:54 -04:00
Joey Hess
a3fb1754f2
clean up transferrer pool
Doing this at shutdown is not very important at all, but I do like to
make sure that when git-annex allocates a resource, it later cleans it
up.

More importantly, stopCoProcesses is used in eg, Remote.Git in a
situation where it needs to stop long-running processes like these.
2020-12-09 13:10:35 -04:00
Joey Hess
a8cdcf528e
fix build failure by avoiding refutable pattern match 2020-12-09 12:43:38 -04:00
Joey Hess
4bce767ca6
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-09 12:35:17 -04:00
Joey Hess
05c0543e8e
move new interface to git-annex transfer
This is to avoid breakage when upgrading or downgrading git-annex with a
process running that uses the interface. It's better to keep the
compatability code for a few years than worry about such breakage.

This commit was sponsored by Brett Eisenberg on Patreon.
2020-12-09 12:33:56 -04:00
jwodder
30defd434e Correct author 2020-12-09 14:11:07 +00:00
jwodder
6c0bd08b9d 2020-12-09 14:09:47 +00:00
Joey Hess
b9c1aa9258
reword 2020-12-08 15:55:00 -04:00
Joey Hess
592e0ada16
update 2020-12-08 15:46:09 -04:00
Joey Hess
d81bf4e018
Merge branch 'message-serialization' 2020-12-08 15:23:43 -04:00
Joey Hess
41f2c308ff
stall detection is working
New config annex.stalldetection, remote.name.annex-stalldetection, which
can be used to deal with remotes that stall during transfers, or are
sometimes too slow to want to use.

This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 15:22:18 -04:00
Joey Hess
09ed9f7d1f
reminder for later 2020-12-08 15:20:05 -04:00
Joey Hess
c4d489f7d4
add todo
Not going to do this yet, so remember for later.
2020-12-08 15:17:35 -04:00
Joey Hess
794fc72afb
avoid parseDuration succeeding on empty string 2020-12-08 12:51:56 -04:00
Joey Hess
b9cfd15e90
add killTransferrer
There is redundant code in the assistant that does the same thing,
but that code uses a PID, not a ProcessHandle, and gets the PID from,
apparently, the TransferInfo transferPid (although I can't seem to find
where that gets set on non-windows).
2020-12-08 11:43:06 -04:00
Joey Hess
822a8eadf8
rename 2020-12-08 10:53:07 -04:00
Joey Hess
9c4ec1140e
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-07 16:56:41 -04:00
Joey Hess
2cb3f3a99d
annex.stalldetection docs
Not implemented yet.

This commit was sponsored by Svenne Krap on Patreon.
2020-12-07 16:55:24 -04:00
Joey Hess
d616881057
devblog 2020-12-07 16:54:57 -04:00
Joey Hess
fcc9e01556
finally using transferkeys
Seems to work! Even progress bars. Have not tested prompting or various
error message displays yet.

transferkeys had to be made to operate in different modes for the
Assistant and Annex monads. A bit ugly, but it did relegate that
really ugly Database.Keys.closeDb in transferkeys to only the assistant
code path.

This commit was sponsored by Noam Kremen.
2020-12-07 16:18:26 -04:00
Joey Hess
4c47568876
refactoring
This is groundwork for using git-annex transferkeys to run transfers,
in order to allow stalled transfers to be interrupted and retried.

The new upload and download are closer to what git-annex transferkeys
does, so the plan is to make them use it.

Then things that were left using upload' and download' won't recover
from stalls. Notably, that includes import and export. But
at least get/move/copy will be able to. (Also the assistant hopefully,
but not yet.)

This commit was sponsored by Jake Vosloo on Patreon.
2020-12-07 14:49:17 -04:00
yarikoptic
d5bfa9bdf9 added datalad project tag 2020-12-07 18:23:28 +00:00
jwodder
7db3832e37 2020-12-07 17:36:34 +00:00
Joey Hess
a0e1650a15
Merge branch 'master' into message-serialization 2020-12-07 13:33:14 -04:00
Joey Hess
47016fc656
move TransferrerPool from Assistant state to Annex state
This commit was sponsored by Graham Spencer on Patreon.
2020-12-07 13:21:35 -04:00
Joey Hess
72e5764a87
move TransferrerPool from assistant
This old code will now be useful for git-annex beyond the assistant.

git-annex won't use the CheckTransferrer part, and won't run transferkeys
as a batch process, and will want withTransferrer to not shut down
transferkeys processes. Still, the rest of this is a good fit for what I
need now.

Also removed some dead code, and simplified a little bit.

This commit was sponsored by Mark Reidenbach on Patreon.
2020-12-07 12:50:48 -04:00
Lukey
a1b2271713 Added a comment 2020-12-05 13:07:31 +00:00
Joey Hess
74a5a604bf
reword 2020-12-04 23:52:33 -04:00
Joey Hess
c61a024f59
correction 2020-12-04 23:49:23 -04:00
Joey Hess
0c15d90076
typo 2020-12-04 23:44:49 -04:00
Joey Hess
9e85c95ebe
clarify 2020-12-04 23:43:14 -04:00
Joey Hess
dcca24dc95
simplify 2020-12-04 23:41:44 -04:00
Joey Hess
55c5df5162
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-04 23:38:42 -04:00
Joey Hess
c9fbf00b96
update 2020-12-04 23:36:52 -04:00
eric.w@eee65cd362d995ced72640c7cfae388ae93a4234
73e8b79cce 2020-12-05 00:43:43 +00:00
eric.w@eee65cd362d995ced72640c7cfae388ae93a4234
7e58bc8e70 2020-12-05 00:42:42 +00:00
kyle
d1ffd246bf bug: alwayscommit=false on windows 2020-12-04 21:05:25 +00:00
Joey Hess
efbc77f505
reword 2020-12-04 15:46:46 -04:00
Joey Hess
fa082f1f4f
fix link 2020-12-04 15:45:37 -04:00
Joey Hess
5fc60b8bd9
devblog 2020-12-04 15:43:03 -04:00
Joey Hess
2878ab4566
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-04 15:01:12 -04:00
Joey Hess
27f221cd7e
update 2020-12-04 15:00:49 -04:00
Joey Hess
438d5be1f7
support prompt in message serialization
That seems to be the last thing needed for message serialization.
Although it's only used in the assistant currently, so hard to tell if I
forgot something.

At this point, it should be possible to start using transferkeys
when performing transfers, which will allow killing a transferkeys
process if a transfer times out or stalls. But that's for another day.

This commit was sponsored by Ethan Aubin.
2020-12-04 14:54:09 -04:00