Commit graph

34873 commits

Author SHA1 Message Date
Joey Hess
44de3fff0b
avoid rsync/gcrypt ssh startup delay with -J
Avoid a delay at startup when concurrency is enabled and there are
rsync or gcrypt special remotes, which was caused by git-annex
opening a ssh connection to the remote too early.

sshOptions makes a connection to the ssh server if one is not already open,
when concurrency is enabled. Avoid doing that at startup, when the remote
list is being built, but the remote may not be used at all.

Instead, rsync/gcrypt now runs sshOptions once per ssh connection to the
server. This should not be significant overhead since Remote.Git already
has the same overhead (as do Bup and Ddar).
2019-06-13 11:16:38 -04:00
Joey Hess
43805a0be9
devblog 2019-06-12 15:10:17 -04:00
Joey Hess
157f41427f
bug 2019-06-12 15:00:28 -04:00
Joey Hess
e589a9b3fc
moving this to a bug 2019-06-12 15:00:14 -04:00
Joey Hess
44c971b9ec
Merge branch 'master' into starting
This reverts commit e07003ab73, adding back
the separate queue for cleanup actions.
2019-06-12 14:51:57 -04:00
Joey Hess
7b5aad2452
Merge commit '6f8322b8f72f3399d4c28426749db5d01742001d' into starting 2019-06-12 14:50:59 -04:00
Joey Hess
e07003ab73
Revert "separate queue for cleanup actions"
This reverts commit 659640e224
and 4932972487

Too early to include these in a release; they'll be de-reverted after
the release.
2019-06-12 14:47:40 -04:00
Joey Hess
6f8322b8f7
close; not a bug 2019-06-12 14:45:25 -04:00
Joey Hess
e1c48509d7
remove incorrect changelog entry
I didn't speed up -J seek yet
2019-06-12 14:13:45 -04:00
Joey Hess
ba2551da6f
add startingNoMessage
Fixes the last wart in the StartMessage transition. A few commands
include other CommandStart actions that generate output, and
do not themselves need to display a start/end message.
2019-06-12 14:11:23 -04:00
Joey Hess
70bc30acb1
get rid of implicitMessages state
Oh joyous day, this is probably git-annex's oldest implementation wart,
source of much unncessary bother.

Now that we have a StartMessage, showEndResult' can look at it to know
if it needs to display an end message or not.

This is also going to be faster, because it avoids an uncessary state
lookup for each file processed.
2019-06-12 14:01:41 -04:00
Joey Hess
8e5ea28c26
finish CommandStart transition
The hoped for optimisation of CommandStart with -J did not materialize.
In fact, not runnign CommandStart in parallel is slower than -J3.
So, CommandStart are still run in parallel.

(The actual bad performance I've been seeing with -J in my big repo
has to do with building the remoteList.)

But, this is still progress toward making -J faster, because it gets rid
of the onlyActionOn roadblock in the way of making CommandCleanup jobs
run separate from CommandPerform jobs.

Added OnlyActionOn constructor for ActionItem which fixes the
onlyActionOn breakage in the last commit.

Made CustomOutput include an ActionItem, so even things using it can
specify OnlyActionOn.

In Command.Move and Command.Sync, there were CommandStarts that used
includeCommandAction, so output messages, which is no longer allowed.
Fixed by using startingCustomOutput, but that's still not quite right,
since it prevents message display for the includeCommandAction run
inside it too.
2019-06-12 13:24:01 -04:00
Ilya_Shlyakhter
3820829407 re: status of v7 2019-06-12 15:53:12 +00:00
Ilya_Shlyakhter
d2658d9537 an issue involving repos cloned with --single-branch 2019-06-11 23:30:26 +00:00
anthony@ad39673d230d75cbfd19d2757d754030049c7673
fd7f316482 Added a comment 2019-06-10 16:52:23 +00:00
anthony@ad39673d230d75cbfd19d2757d754030049c7673
d5a0bf3ae9 clarify it's the new android installer 2019-06-10 16:38:44 +00:00
anthony@ad39673d230d75cbfd19d2757d754030049c7673
856affe859 initial report 2019-06-10 16:37:31 +00:00
kirelagin@6d93475882c55a329fedae6be1971868a775ec7e
c0f4788cdb Added a comment: Workaround? 2019-06-08 13:03:50 +00:00
Joey Hess
8d573a653b
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-07 19:34:46 -04:00
Joey Hess
1d92846e54
bug report from MacGyver.mdwn 2019-06-07 19:34:21 -04:00
jamie@b5676b90eec0401ca8faac7c972eaf5676891601
0424549599 removed 2019-06-07 15:19:15 +00:00
Joey Hess
132ec9e005
devblog 2019-06-06 17:16:30 -04:00
Joey Hess
436f107715
make CommandStart return a StartMessage
The goal is to be able to run CommandStart in the main thread when -J is
used, rather than unncessarily passing it off to a worker thread, which
incurs overhead that is signficant when the CommandStart is going to
quickly decide to stop.

To do that, the message it displays needs to be displayed in the worker
thread, after the CommandStart has run.

Also, the change will mean that CommandStart will no longer necessarily
run with the same Annex state as CommandPerform. While its docs already
said it should avoid modifying Annex state, I audited all the
CommandStart code as part of the conversion. (Note that CommandSeek
already sometimes runs with a different Annex state, and that has not been
a source of any problems, so I am not too worried that this change will
lead to breakage going forward.)

The only modification of Annex state I found was it calling
allowMessages in some Commands that default to noMessages. Dealt with
that by adding a startCustomOutput and a startingUsualMessages.
This lets a command start with noMessages and then select the output it
wants for each CommandStart.

One bit of breakage: onlyActionOn has been removed from commands that used it.
The plan is that, since a StartMessage contains an ActionItem,
when a Key can be extracted from that, the parallel job runner can
run onlyActionOn' automatically. Then commands won't need to worry about
this detail. Future work.

Otherwise, this was a fairly straightforward process of making each
CommandStart compile again. Hopefully other behavior changes were mostly
avoided.

In a few cases, a command had a CommandStart that called a CommandPerform
that then called showStart multiple times. I have collapsed those
down to a single start action. The main command to perhaps suffer from it
is Command.Direct, which used to show a start for each file, and no
longer does.

Another minor behavior change is that some commands used showStart
before, but had an associated file and a Key available, so were changed
to ShowStart with an ActionItemAssociatedFile. That will not change the
normal output or behavior, but --json output will now include the key.
This should not break it for anyone using a real json parser.
2019-06-06 17:13:54 -04:00
Joey Hess
258a7c5cd1
add Key to all ActionItem constructors 2019-06-06 12:53:24 -04:00
Joey Hess
3893d84764
todo 2019-06-06 12:02:27 -04:00
Joey Hess
6cefe071b1
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-05 20:19:27 -04:00
Joey Hess
4088ce49ce
devblog 2019-06-05 20:18:59 -04:00
Joey Hess
4932972487
fix STM deadlock
659640e224 was buggy, it had a STM
deadlock because two actions both wanted to takeTMVar the WorkerPool
and so blocked one-another.

Fixed by completely reworking how the pool is maintained. Maintenace
threads now wait for the Async actions and update the WorkerPool. This
means twice as many threads as before, but green threads so will only
use a few extra bytes ram per thread.
2019-06-05 20:07:35 -04:00
Joey Hess
3eac4e01a4
idea 2019-06-05 19:43:01 -04:00
Joey Hess
659640e224
separate queue for cleanup actions
When running multiple concurrent actions, the cleanup phase is run in a
separate queue than the main action queue. This can make some commands
faster, because less time is spent on bookkeeping in between each file
transfer.

But as far as I can see, nothing will be sped up much by this yet, because
all the existing cleanup actions are very light-weight. This is just groundwork
for deferring checksum verification to cleanup time.

This change does mean that if the user expects -J2 will mean that they see no
more than 2 jobs running at a time, they may be surprised to see 4 in some
cases (if the cleanup actions are slow enough to notice).

It might also make sense to enable background cleanup without the -J,
for at least one cleanup action. Indeed, that's the behavior that -J1
has now. At some point in the future, it make make sense to make the
behavior with no -J the same as -J1. The only reason it's not currently
is that git-annex can build w/o concurrent-output, and also any bugs
in concurrent-output (such as perhaps misbehaving on non-VT100 compatible
terminals) are avoided by default by only using it when -J is used.
2019-06-05 17:54:35 -04:00
Joey Hess
c04b2af3e1
improved WorkerPool abstraction
No behavior changes.
2019-06-05 14:26:48 -04:00
Joey Hess
30286bf067
improve layout 2019-06-05 12:20:20 -04:00
Joey Hess
0b5cc89687
wording 2019-06-05 12:20:11 -04:00
kyle
b390256bd1 Added a comment: uuid.log is also not created 2019-06-05 15:18:30 +00:00
kyle
0d60b2f2c5 2019-06-05 13:57:02 +00:00
Joey Hess
6e51b9ae88
clarify 2019-06-04 21:49:53 -04:00
Joey Hess
082e1f1738
Don't try to import .git directories from special remotes
Because git does not support storing git repositories inside a git
repository.
2019-06-04 15:14:20 -04:00
Joey Hess
500f72ec3d
comment typo 2019-06-04 14:40:07 -04:00
Joey Hess
7dcc815c29
more thoughts 2019-06-04 14:38:55 -04:00
Joey Hess
cd20dc4158
thoughts 2019-06-04 14:13:15 -04:00
Joey Hess
fe49747fc8
add missing case
and fix name shadowing warning
2019-06-04 11:24:32 -04:00
Joey Hess
418e842d93
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-02 15:38:37 -04:00
Joey Hess
f0cba96a92
update 2019-06-02 15:26:50 -04:00
git-annex.branchable@794aec3ca2af16d925f13a04f7e00a6dfe523186
7a5b534ea8 2019-06-02 10:16:58 +00:00
Joey Hess
fa2c8d8fd9
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-01 09:41:59 -04:00
Joey Hess
f05875f00b
bug 2019-06-01 09:41:39 -04:00
Ilya_Shlyakhter
196548a1b2 a place for pull requests 2019-05-31 17:05:23 +00:00
joer962500
22f1b6673e Added a comment: asus tablet stuck loading screen 2019-05-31 07:24:27 +00:00
Joey Hess
5d681a3ce6
Merge branch 'master' of ssh://git-annex.branchable.com 2019-05-30 16:05:41 -04:00
Joey Hess
6136e299a2
add back support for following http to ftp redirects
Did not test build with http-client < 0.5 and while I tried to support
it, the ifdefed parts may needs some fixes.
2019-05-30 16:04:59 -04:00