Commit graph

38621 commits

Author SHA1 Message Date
Joey Hess
3c76a31b15
response and related todo 2020-12-11 16:21:16 -04:00
Joey Hess
4a8723246d
avoid transferrer committing the git-annex branch on shutdown
The parent is will do it when it shuts down, and having both of them
trying to do it at the same time seems like something good to avoid.
2020-12-11 16:16:07 -04:00
Joey Hess
e6692b66f1
remove
I need to think about this some more, not clear if it's a todo
item specific to stalldetection at all. Remotes with this behavior
also show no progress when run with -J. And some other remotes don't
update any progress meters at all, eg adb is that way and so are hook
remotes and of course external remotes don't have to send progress info.
2020-12-11 15:49:39 -04:00
Joey Hess
fadf47557f
note 2020-12-11 15:48:03 -04:00
Joey Hess
90eadbce49
close 2020-12-11 15:45:55 -04:00
Joey Hess
d3f78da0ed
propagate signals to the transferrer process group
Done on unix, could not implement it on windows quite.

The signal library gets part of the way needed for windows.
But I had to open https://github.com/pmlodawski/signal/issues/1 because
it lacks raiseSignal.

Also, I don't know what the equivilant of getProcessGroupIDOf is on
windows. And System.Process does not provide a way to send any signal to
a process group except for SIGINT.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2020-12-11 15:32:00 -04:00
Joey Hess
79c765b727
meant to close this earlier 2020-12-11 14:25:54 -04:00
Joey Hess
e1cf8bd35c
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-11 12:53:14 -04:00
Joey Hess
0c46ee5ce0
simplify transferr protocol 2020-12-11 12:52:22 -04:00
Joey Hess
095cdc7e83
extend transferrer protocol to send progress bar total size updates
New protocol is not back-compat with old one, but it's never been
released so that's ok.
2020-12-11 12:42:28 -04:00
Joey Hess
94b323a8e8
use TotalSize more extensively 2020-12-11 12:10:43 -04:00
Joey Hess
263fd1d459
ugh 2020-12-11 11:51:46 -04:00
Joey Hess
a422a056f2
make getViaTmpFrom no longer update location log
All callers adjusted to update it themselves.

In Command.ReKey, and Command.SetKey, the cleanup action already did,
so it was updating the log twice before.

This fixes a bug when annex.stalldetection is set, as now
Command.Transferrer can skip updating the location log, and let it be
updated by the calling process.
2020-12-11 11:50:13 -04:00
kyle
e0780877d7 Added a comment 2020-12-11 15:29:26 +00:00
Joey Hess
a6ed23b82f
todo 2020-12-11 11:09:19 -04:00
muks@6bded6d879d80c43cdf4893fabb785876eea0a7b
16f2fcd778 Added a comment 2020-12-11 02:18:30 +00:00
muks@6bded6d879d80c43cdf4893fabb785876eea0a7b
cf4a669d86 2020-12-11 01:56:38 +00:00
yarikoptic
e0965b831c Added a comment 2020-12-10 21:55:20 +00:00
Joey Hess
108c7d96dd
comment 2020-12-10 17:52:26 -04:00
Joey Hess
8c0ff6b8cd
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-10 17:47:22 -04:00
Joey Hess
d3c11eac5d
2 bugs involving new stalldetection feature 2020-12-10 17:46:58 -04:00
yarikoptic
c2071567ca Initial report on export -J6 to S3 failing due to "transfer already in progress" 2020-12-10 21:16:56 +00:00
Joey Hess
cedad7b37d
refactor 2020-12-10 16:33:52 -04:00
Joey Hess
6a11b6fab8
Support special remotes that are configured with importtree=yes but without exporttree=yes
There was no particular reason not to support this, other than maybe a lack
of a use case. One use case would of course be a remote that you want to
avoid overwriting content on. A new use case is the idea of importing from
backups, eg borg, where exporting is not necessarily supported at all.

This commit was sponsored by Brock Spratlen on Patreon.
2020-12-10 13:17:40 -04:00
Joey Hess
38e0e2e471
bug 2020-12-10 13:09:57 -04:00
anarcat
6b3880b539 2020-12-10 15:53:37 +00:00
Joey Hess
714e665c92
reorg 2020-12-10 11:25:02 -04:00
Joey Hess
1a361dca37
devblog 2020-12-09 16:20:57 -04:00
Joey Hess
04c12aa6df
custom protocol for transferrer
Rather than using Read/Show, which would force me to preserve data types
into the future.

I considered just deriving json and sending that, but I don't much like
deriving json with data types that have named constructors (like Key
does) because again it locks in data type details.

So instead, used SimpleProtocol, with a fairly complex and unreadable
protocol. But it is as efficient as the p2p protocol at least, and as
future proof.

(Writing my own custom json instances would have worked but I thought
of it too late and don't want to do all the work twice. The only real
benefit might be that aeson could be faster.)

Note that, when a new protocol request type is added later, git-annex
trying to use it will cause the git-annex transferrer to display a
protocol error message. That seems ok; it would only happen if a new
git-annex found an old version of itself in PATH or the program
file. So it's unlikely, and all it can do anyway is display an error.
(The error message could perhaps be improved..)

This commit was sponsored by Jack Hill on Patreon.
2020-12-09 16:13:59 -04:00
Joey Hess
447d798987
export encode_c' 2020-12-09 15:28:45 -04:00
Joey Hess
19777d1c6f
minor improvements
Adding new instance for Integer, and some parsers for more parameters.

The conversion of readish to readMaybe is done because a serialized
exit code cannot contain additional text after the number.
2020-12-09 15:28:11 -04:00
Joey Hess
7e25c643cf
avoid P2P.Protocol defining instance Proto.Serializable AssociatedFile
It's got its own specific hacks, so it needs it own specific data type.
2020-12-09 13:46:42 -04:00
Joey Hess
004a4f5fb1
factor out Types.Transferrer 2020-12-09 13:28:49 -04:00
Joey Hess
677003a6df
rename helper
More consistent name with TransferrerPool
2020-12-09 13:24:24 -04:00
Joey Hess
3fa2bc2eed
rename back, there were links to this 2020-12-09 13:14:54 -04:00
Joey Hess
a3fb1754f2
clean up transferrer pool
Doing this at shutdown is not very important at all, but I do like to
make sure that when git-annex allocates a resource, it later cleans it
up.

More importantly, stopCoProcesses is used in eg, Remote.Git in a
situation where it needs to stop long-running processes like these.
2020-12-09 13:10:35 -04:00
Joey Hess
a8cdcf528e
fix build failure by avoiding refutable pattern match 2020-12-09 12:43:38 -04:00
Joey Hess
4bce767ca6
Merge branch 'master' of ssh://git-annex.branchable.com 2020-12-09 12:35:17 -04:00
Joey Hess
05c0543e8e
move new interface to git-annex transfer
This is to avoid breakage when upgrading or downgrading git-annex with a
process running that uses the interface. It's better to keep the
compatability code for a few years than worry about such breakage.

This commit was sponsored by Brett Eisenberg on Patreon.
2020-12-09 12:33:56 -04:00
jwodder
30defd434e Correct author 2020-12-09 14:11:07 +00:00
jwodder
6c0bd08b9d 2020-12-09 14:09:47 +00:00
Joey Hess
b9c1aa9258
reword 2020-12-08 15:55:00 -04:00
Joey Hess
592e0ada16
update 2020-12-08 15:46:09 -04:00
Joey Hess
d81bf4e018
Merge branch 'message-serialization' 2020-12-08 15:23:43 -04:00
Joey Hess
41f2c308ff
stall detection is working
New config annex.stalldetection, remote.name.annex-stalldetection, which
can be used to deal with remotes that stall during transfers, or are
sometimes too slow to want to use.

This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 15:22:18 -04:00
Joey Hess
09ed9f7d1f
reminder for later 2020-12-08 15:20:05 -04:00
Joey Hess
c4d489f7d4
add todo
Not going to do this yet, so remember for later.
2020-12-08 15:17:35 -04:00
Joey Hess
794fc72afb
avoid parseDuration succeeding on empty string 2020-12-08 12:51:56 -04:00
Joey Hess
b9cfd15e90
add killTransferrer
There is redundant code in the assistant that does the same thing,
but that code uses a PID, not a ProcessHandle, and gets the PID from,
apparently, the TransferInfo transferPid (although I can't seem to find
where that gets set on non-windows).
2020-12-08 11:43:06 -04:00
Joey Hess
822a8eadf8
rename 2020-12-08 10:53:07 -04:00