Commit graph

34818 commits

Author SHA1 Message Date
Ilya_Shlyakhter
8e6d544906 Added a comment: bug fix release 2019-06-22 16:14:59 +00:00
stephanlindner
8496d44fb9 Added a comment: comment 4 2019-06-21 19:10:33 +00:00
Ilya_Shlyakhter
5c82b4c39e Added a comment 2019-06-21 13:49:37 +00:00
CandyAngel
82982fedb9 Added a comment 2019-06-21 07:38:36 +00:00
kyle
66b8f7c751 Added a comment: thanks 2019-06-21 02:05:24 +00:00
Joey Hess
989b738671
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-20 20:32:18 -04:00
Joey Hess
84e729fda5
fix init default description reversion
init: Fix a reversion in the last release that prevented automatically
generating and setting a description for the repository.

Seemed best to factor out uuidDescMapRaw that does not
have the default mempty descrition behavior.

I don't much like that behavior, but I know things depend on it.
One thing in particular is `git annex info` which lists the uuids and
descriptions; if the current repo has been initialized in some way that
means it does not have a description, it would not show up w/o that.

(Not only repos created due to this bug might lack that. For example a repo
that was marked dead and had --drop-dead delete its git-annex branch info,
and then came back from the dead would similarly not be in the uuid.log.
Also there have been other versions of git-annex that didn't set a default
description; for years there was no default description.)
2019-06-20 20:30:24 -04:00
Ilya_Shlyakhter
7d1469bfd7 Added a comment 2019-06-20 23:48:36 +00:00
grmat@f46c69b114fc77408ff25d75efa4c7dc10b4c0b1
c389763453 Added a comment: Thank you very much! 2019-06-20 22:54:50 +00:00
stephanlindner
fdd07a0e7c 2019-06-20 20:38:00 +00:00
kyle
247eb94248 Add "meta author" so this entry is (hopefully) listed on users/datalad page 2019-06-20 20:08:49 +00:00
kyle
566e4f6d91 Added a comment 2019-06-20 19:42:35 +00:00
Ilya_Shlyakhter
390749afb0 bug-fix release 2019-06-20 16:51:42 +00:00
Joey Hess
7264203eb1
importfeed: When there's a problem parsing the feed, --debug will output the feed content that was downloaded.
And let the user know about it in the failure messages.
2019-06-20 12:37:07 -04:00
Joey Hess
759fd9ea68
avoid url resume from 0
When downloading an url and the destination file exists but is empty,
avoid using http range to resume, since a range "bytes=0-" is an unusual
edge case that it's best to avoid relying on working.

This is known to fix a case where importfeed downloaded a partial feed from
such a server. Since importfeed uses withTmpFile, the destination always exists
empty, so it would particularly tickle such problem servers. Resuming from 0
is otherwise possible, but unlikely.
2019-06-20 12:26:17 -04:00
Joey Hess
06ea1c4228
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-20 11:07:05 -04:00
grmat@f46c69b114fc77408ff25d75efa4c7dc10b4c0b1
19de32c4e4 2019-06-20 09:07:09 +00:00
kyle
49d416089a Added a comment: annex-ignore and syncing 2019-06-20 02:37:27 +00:00
Joey Hess
ba433bdc85
refactor 2019-06-19 20:19:38 -04:00
Joey Hess
26f0f8b20f
optimisation
Avoid an unncessary STM transaction. This will happen when the worker
pool is not completely full of the new stage, which is the common case.

In the uncommon case, this adds only a tiny bit of overhead for the
extra traversal of the worker pool. And the thread is going to block
for some time anyway.
2019-06-19 20:13:19 -04:00
Joey Hess
0e1140ac47
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-19 18:45:17 -04:00
Joey Hess
ad6b8c5f77
fix STM deadlock in finishCommandActions
Happened every time, because it was taking the pool TMVar while threads
were still running, and then the thread would try to switch state.
2019-06-19 18:34:26 -04:00
Joey Hess
19321e6892
devblog 2019-06-19 18:18:37 -04:00
Joey Hess
37d505dd6b
avoid STM deadlock
When all worker threads are running and enteringStage is called,
it waits for an idle slot. If all off the other threads then call it in
turn, a deadlock occurrs.

This is the same problem I didn't actually fix in
5a9842d7ed.

Fixed by doing two separate STM transactions, the first replaces its
active thread with an idle thread, and the second waits for another idle
thread. That guarantees there will eventually be an idle thread to find.

The changes to WorkerPool were necessary because it can't add an idle
thread containing the Annex state and go on to run an action using that
same state, so I had to remove the Annex state from IdleWorker.
2019-06-19 18:15:25 -04:00
Joey Hess
a0d3a699e2
fix concurrency
Broken by recent commits, because before dupState is called, the Annex
state needs to have concurrent output enabled, and the thread pool
populated.
2019-06-19 16:12:39 -04:00
Joey Hess
9671248fff
speed up enteringStage in non-concurrent mode
Avoid a STM transaction.

Also got rid of UnallocatedWorkerPool.
2019-06-19 15:47:54 -04:00
Joey Hess
05a908c3c9
fix oops 2019-06-19 14:52:44 -04:00
Joey Hess
9d36c826c0
use fine-grained WorkerStages when transferring and verifying
This means that Command.Move and Command.Get don't need to
manually set the stage, and is a lot cleaner conceptually.

Also, this makes Command.Sync.syncFile use the worker pool better.
In the scenario where it first downloads content and then uploads it to
some other remotes, it will start in TransferStage, then enter VerifyStage
and then go back to TransferStage for each transfer to the remotes.
Before, it entered CleanupStage after the download, and stayed in it for
the upload, so too many transfer jobs could run at the same time.

Note that, in Remote.Git, it uses runTransfer and also verifyKeyContent
inside onLocal. That has a Annex state for the remote, with no worker pool.
So the resulting calls to enteringStage won't block in there.

While Remote.Git.copyToRemote does do checksum verification, I
realized that should not use a verification slot in the WorkerPool
to do it. Because, it's reading back from eg, a removable disk to checksum.
That will contend with other writes to that disk. It's best to treat
that checksum verification as just part of the transer. So, removed the todo
item about that, as there's nothing needing to be done.
2019-06-19 13:24:20 -04:00
Joey Hess
53882ab4a7
make WorkerStage an open type
Rather than limiting it to PerformStage and CleanupStage, this opens it
up so any number of stages can be added as needed by commands.

Each concurrent command has a set of stages that it uses, and only
transitions between those can block waiting for a free slot in the
worker pool. Calling enteringStage for some other stage does not block,
and has very little overhead.

Note that while before the Annex state was duplicated on the first call
to commandAction, this now happens earlier, in startConcurrency.
That means that seek stage actions should that use startConcurrency
and then modify Annex state won't modify the state of worker threads
they then start. I audited all of them, and only Command.Seek
did so; prepMerge changes the working directory and so has to come
before startConcurrency.

Also, the remote list is built before duplicating the state, which means
that it gets built earlier now than it used to. This would only have an
effect of making commands that end up not needing to perform any actions
unncessary build the remote list (only when they're run with concurrency
enable), but that's a minor overhead compared to commands seeking
through the work tree and determining they don't need to do anything.
2019-06-19 13:05:03 -04:00
Ilya_Shlyakhter
3a19ef7994 Added a comment 2019-06-19 16:51:31 +00:00
kyle
247bc21331 Added a comment: Set remote.<name>.annex-ignore 2019-06-19 14:18:20 +00:00
Ilya_Shlyakhter
03c7e06c2b added forum question about git-annex-sync errors with bitbucket.org 2019-06-18 19:56:21 +00:00
kyle
ce7b9ecae6 Added a comment: Issue with description cache? 2019-06-17 21:11:14 +00:00
anarcat
41ef54276b Added a comment: thanks! 2019-06-17 20:17:27 +00:00
Joey Hess
e19408ed9d
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-17 15:26:57 -04:00
Joey Hess
c31f4c0e66
devblog 2019-06-17 15:26:46 -04:00
Joey Hess
04cc470201
run download checksum verification in separate job pool
get, move, copy, sync: When -J or annex.jobs has enabled concurrency,
checksum verification uses a separate job pool than is used for
downloads, to keep bandwidth saturated.

Not yet done for upload checksum verification, but that only affects
remotes on local disks.
2019-06-17 14:58:02 -04:00
Joey Hess
5a9842d7ed
avoid STM deadlock onredundant call to changeStageTo
I couldn't find a way to avoid the deadlock w/o rewriting it to clearly
not have one. I'm not quite sure what was the actual cause of the
deadlock.

This makes me unsure how I now know it clearly doesn't have a
deadlock. But, it was easy to reproduce before (just call it twice in a
row) and doesn't happen now.
2019-06-17 14:51:30 -04:00
Joey Hess
ecbd456312
fix restoring worker pool bug
The bug might have led to a STM deadlock, if this case could ever
actually fire.
2019-06-17 12:52:57 -04:00
Joey Hess
1a8d06d251
thought 2019-06-17 11:50:18 -04:00
jsag@f84637fe752e0235291a118b1cd007bafad0997e
ae9f2d5e6a 2019-06-17 12:43:17 +00:00
Joey Hess
502ce3f243
Merge branch 'starting' 2019-06-15 12:42:10 -04:00
Joey Hess
76c0a38025
add news item for git-annex 7.20190615 2019-06-15 12:39:48 -04:00
Joey Hess
0bd9e8c0e2
releasing package git-annex version 7.20190615 2019-06-15 12:39:16 -04:00
artem
9c4744c3c2 2019-06-14 04:24:47 +00:00
Joey Hess
44de3fff0b
avoid rsync/gcrypt ssh startup delay with -J
Avoid a delay at startup when concurrency is enabled and there are
rsync or gcrypt special remotes, which was caused by git-annex
opening a ssh connection to the remote too early.

sshOptions makes a connection to the ssh server if one is not already open,
when concurrency is enabled. Avoid doing that at startup, when the remote
list is being built, but the remote may not be used at all.

Instead, rsync/gcrypt now runs sshOptions once per ssh connection to the
server. This should not be significant overhead since Remote.Git already
has the same overhead (as do Bup and Ddar).
2019-06-13 11:16:38 -04:00
Joey Hess
43805a0be9
devblog 2019-06-12 15:10:17 -04:00
Joey Hess
157f41427f
bug 2019-06-12 15:00:28 -04:00
Joey Hess
e589a9b3fc
moving this to a bug 2019-06-12 15:00:14 -04:00
Joey Hess
44c971b9ec
Merge branch 'master' into starting
This reverts commit e07003ab73, adding back
the separate queue for cleanup actions.
2019-06-12 14:51:57 -04:00