Commit graph

38808 commits

Author SHA1 Message Date
kyle
e0780877d7 Added a comment 2020-12-11 15:29:26 +00:00
Joey Hess
a6ed23b82f
todo 2020-12-11 11:09:19 -04:00
muks@6bded6d879d80c43cdf4893fabb785876eea0a7b
16f2fcd778 Added a comment 2020-12-11 02:18:30 +00:00
muks@6bded6d879d80c43cdf4893fabb785876eea0a7b
cf4a669d86 2020-12-11 01:56:38 +00:00
yarikoptic
e0965b831c Added a comment 2020-12-10 21:55:20 +00:00
Joey Hess
108c7d96dd
comment 2020-12-10 17:52:26 -04:00
Joey Hess
8c0ff6b8cd
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-10 17:47:22 -04:00
Joey Hess
d3c11eac5d
2 bugs involving new stalldetection feature 2020-12-10 17:46:58 -04:00
yarikoptic
c2071567ca Initial report on export -J6 to S3 failing due to "transfer already in progress" 2020-12-10 21:16:56 +00:00
Joey Hess
cedad7b37d
refactor 2020-12-10 16:33:52 -04:00
Joey Hess
6a11b6fab8
Support special remotes that are configured with importtree=yes but without exporttree=yes
There was no particular reason not to support this, other than maybe a lack
of a use case. One use case would of course be a remote that you want to
avoid overwriting content on. A new use case is the idea of importing from
backups, eg borg, where exporting is not necessarily supported at all.

This commit was sponsored by Brock Spratlen on Patreon.
2020-12-10 13:17:40 -04:00
Joey Hess
38e0e2e471
bug 2020-12-10 13:09:57 -04:00
anarcat
6b3880b539 2020-12-10 15:53:37 +00:00
Joey Hess
714e665c92
reorg 2020-12-10 11:25:02 -04:00
Joey Hess
1a361dca37
devblog 2020-12-09 16:20:57 -04:00
Joey Hess
04c12aa6df
custom protocol for transferrer
Rather than using Read/Show, which would force me to preserve data types
into the future.

I considered just deriving json and sending that, but I don't much like
deriving json with data types that have named constructors (like Key
does) because again it locks in data type details.

So instead, used SimpleProtocol, with a fairly complex and unreadable
protocol. But it is as efficient as the p2p protocol at least, and as
future proof.

(Writing my own custom json instances would have worked but I thought
of it too late and don't want to do all the work twice. The only real
benefit might be that aeson could be faster.)

Note that, when a new protocol request type is added later, git-annex
trying to use it will cause the git-annex transferrer to display a
protocol error message. That seems ok; it would only happen if a new
git-annex found an old version of itself in PATH or the program
file. So it's unlikely, and all it can do anyway is display an error.
(The error message could perhaps be improved..)

This commit was sponsored by Jack Hill on Patreon.
2020-12-09 16:13:59 -04:00
Joey Hess
447d798987
export encode_c' 2020-12-09 15:28:45 -04:00
Joey Hess
19777d1c6f
minor improvements
Adding new instance for Integer, and some parsers for more parameters.

The conversion of readish to readMaybe is done because a serialized
exit code cannot contain additional text after the number.
2020-12-09 15:28:11 -04:00
Joey Hess
7e25c643cf
avoid P2P.Protocol defining instance Proto.Serializable AssociatedFile
It's got its own specific hacks, so it needs it own specific data type.
2020-12-09 13:46:42 -04:00
Joey Hess
004a4f5fb1
factor out Types.Transferrer 2020-12-09 13:28:49 -04:00
Joey Hess
677003a6df
rename helper
More consistent name with TransferrerPool
2020-12-09 13:24:24 -04:00
Joey Hess
3fa2bc2eed
rename back, there were links to this 2020-12-09 13:14:54 -04:00
Joey Hess
a3fb1754f2
clean up transferrer pool
Doing this at shutdown is not very important at all, but I do like to
make sure that when git-annex allocates a resource, it later cleans it
up.

More importantly, stopCoProcesses is used in eg, Remote.Git in a
situation where it needs to stop long-running processes like these.
2020-12-09 13:10:35 -04:00
Joey Hess
a8cdcf528e
fix build failure by avoiding refutable pattern match 2020-12-09 12:43:38 -04:00
Joey Hess
4bce767ca6
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-09 12:35:17 -04:00
Joey Hess
05c0543e8e
move new interface to git-annex transfer
This is to avoid breakage when upgrading or downgrading git-annex with a
process running that uses the interface. It's better to keep the
compatability code for a few years than worry about such breakage.

This commit was sponsored by Brett Eisenberg on Patreon.
2020-12-09 12:33:56 -04:00
jwodder
30defd434e Correct author 2020-12-09 14:11:07 +00:00
jwodder
6c0bd08b9d 2020-12-09 14:09:47 +00:00
Joey Hess
b9c1aa9258
reword 2020-12-08 15:55:00 -04:00
Joey Hess
592e0ada16
update 2020-12-08 15:46:09 -04:00
Joey Hess
d81bf4e018
Merge branch 'message-serialization' 2020-12-08 15:23:43 -04:00
Joey Hess
41f2c308ff
stall detection is working
New config annex.stalldetection, remote.name.annex-stalldetection, which
can be used to deal with remotes that stall during transfers, or are
sometimes too slow to want to use.

This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 15:22:18 -04:00
Joey Hess
09ed9f7d1f
reminder for later 2020-12-08 15:20:05 -04:00
Joey Hess
c4d489f7d4
add todo
Not going to do this yet, so remember for later.
2020-12-08 15:17:35 -04:00
Joey Hess
794fc72afb
avoid parseDuration succeeding on empty string 2020-12-08 12:51:56 -04:00
Joey Hess
b9cfd15e90
add killTransferrer
There is redundant code in the assistant that does the same thing,
but that code uses a PID, not a ProcessHandle, and gets the PID from,
apparently, the TransferInfo transferPid (although I can't seem to find
where that gets set on non-windows).
2020-12-08 11:43:06 -04:00
Joey Hess
822a8eadf8
rename 2020-12-08 10:53:07 -04:00
Joey Hess
9c4ec1140e
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-07 16:56:41 -04:00
Joey Hess
2cb3f3a99d
annex.stalldetection docs
Not implemented yet.

This commit was sponsored by Svenne Krap on Patreon.
2020-12-07 16:55:24 -04:00
Joey Hess
d616881057
devblog 2020-12-07 16:54:57 -04:00
Joey Hess
fcc9e01556
finally using transferkeys
Seems to work! Even progress bars. Have not tested prompting or various
error message displays yet.

transferkeys had to be made to operate in different modes for the
Assistant and Annex monads. A bit ugly, but it did relegate that
really ugly Database.Keys.closeDb in transferkeys to only the assistant
code path.

This commit was sponsored by Noam Kremen.
2020-12-07 16:18:26 -04:00
Joey Hess
4c47568876
refactoring
This is groundwork for using git-annex transferkeys to run transfers,
in order to allow stalled transfers to be interrupted and retried.

The new upload and download are closer to what git-annex transferkeys
does, so the plan is to make them use it.

Then things that were left using upload' and download' won't recover
from stalls. Notably, that includes import and export. But
at least get/move/copy will be able to. (Also the assistant hopefully,
but not yet.)

This commit was sponsored by Jake Vosloo on Patreon.
2020-12-07 14:49:17 -04:00
yarikoptic
d5bfa9bdf9 added datalad project tag 2020-12-07 18:23:28 +00:00
jwodder
7db3832e37 2020-12-07 17:36:34 +00:00
Joey Hess
a0e1650a15
Merge branch 'master' into message-serialization 2020-12-07 13:33:14 -04:00
Joey Hess
47016fc656
move TransferrerPool from Assistant state to Annex state
This commit was sponsored by Graham Spencer on Patreon.
2020-12-07 13:21:35 -04:00
Joey Hess
72e5764a87
move TransferrerPool from assistant
This old code will now be useful for git-annex beyond the assistant.

git-annex won't use the CheckTransferrer part, and won't run transferkeys
as a batch process, and will want withTransferrer to not shut down
transferkeys processes. Still, the rest of this is a good fit for what I
need now.

Also removed some dead code, and simplified a little bit.

This commit was sponsored by Mark Reidenbach on Patreon.
2020-12-07 12:50:48 -04:00
Lukey
a1b2271713 Added a comment 2020-12-05 13:07:31 +00:00
Joey Hess
74a5a604bf
reword 2020-12-04 23:52:33 -04:00
Joey Hess
c61a024f59
correction 2020-12-04 23:49:23 -04:00