Commit graph

3336 commits

Author SHA1 Message Date
Joey Hess
9e676f062f
split out todo 2020-09-28 10:40:13 -04:00
Joey Hess
6a41a615b9
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-25 13:51:20 -04:00
Joey Hess
13f9c88123
add todo 2020-09-25 13:51:04 -04:00
Lukey
12eb7a3ceb Added a comment 2020-09-25 16:33:42 +00:00
Joey Hess
ace02f41b0
seek: defer matcher check until more info is known
Sped up seeking for files to operate on, when using options like --copies
or --in, by around 20%.

Benchmark showed an increase for --copies from 155 seconds to 121
seconds, and --in remote will be similar to that.

For --in here, the speedup was less, 5-10% or so.

(both warm cache)

This commit was sponsored by Jack Hill on Patreon.
2020-09-24 17:59:12 -04:00
Joey Hess
d89984b121
sync --all avoid unncessary first pass
Sped up seeking to around twice as fast, by avoiding a pass over the
worktree files when preferred content expressions of the local repo and
remotes don't use include=/exclude=.

Thanks to Lukey for identifying the optimisation.

This commit was sponsored by Brock Spratlen on Patreon.
2020-09-24 15:12:09 -04:00
Joey Hess
c1b4d76e6b
make MatchFiles introspectable
matchNeedsFileContent is not used yet, but shows how to add information
about terminals. That one would be needed for
https://git-annex.branchable.com/todo/sync_fast_import/

Note the tricky bit in Annex.FileMatcher.call where it folds over the
included matcher to propagate the information.

This commit was sponsored by Svenne Krap on Patreon.
2020-09-24 14:01:53 -04:00
Joey Hess
6d95361f35
add meta todo 2020-09-24 12:54:54 -04:00
Joey Hess
4d4f963c46
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-24 12:42:32 -04:00
Joey Hess
68f9766544
Improve --debug output to show pid of processes that are started and stopped
getPid returns Nothing if the process has already been stopped, and in that
case, the pid will not be displayed. I think that would only happen if
waitForProcess or similar gets called more than once on the same process
handle though.

getPid on unix has an overhead of only a MVar read. On Windows it needs to
make a syscall, so will be probably more expensive. While the added expense
happens even when debug logging is disabled, it should be small enough
compared with the overhead of starting a process that it's not a problem.

(It does occur to me that a debugM that took an IO String could only run it
when debugging is really enabled, which would improve performance. It does
not seem possible to use the current hslogger interface to do that though;
it does not expose the information that would be needed.)
2020-09-24 12:39:57 -04:00
Lukey
221b47162d 2020-09-24 16:36:12 +00:00
yarikoptic
9e033f3001 initial TODO for making failure messages for processes be more informative 2020-09-23 13:03:38 +00:00
Joey Hess
6a5e0cbfc7
Improve the "Try making some of these repositories available" message
With some hints for the user for what to do.

Took care to avoid changing the json output. It would have been ok to add
the new separated lists to it, in addition to the old list, but I didn't
do that because I didn't see much point.
2020-09-22 14:10:30 -04:00
Joey Hess
5cfcf1f05f
cache remote.log
Unlikely to speed up any of the existing uses much, but I want to use it
in a message that might be displayed many times.
2020-09-22 13:52:26 -04:00
Joey Hess
ebdce707da
fix typo 2020-09-22 13:26:49 -04:00
Joey Hess
361ef19999
wording 2020-09-22 12:39:33 -04:00
Joey Hess
41044de833
comment 2020-09-22 12:24:22 -04:00
yarikoptic
c44cd27520 Added a comment 2020-09-18 20:11:27 +00:00
Joey Hess
46a7fcef0d
close 2020-09-18 13:21:32 -04:00
Joey Hess
186c3827d0
comment 2020-09-18 13:21:00 -04:00
Joey Hess
500454935f
comment 2020-09-18 12:08:11 -04:00
Joey Hess
956ff1350a
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-18 12:00:12 -04:00
Joey Hess
81a38df5a7
add missing CHECKURL-FAILURE ErrorMsg to docs 2020-09-18 11:58:18 -04:00
yarikoptic
b48acde47d some whining about check-ignore 2020-09-18 15:30:23 +00:00
yarikoptic
f913822c03 initial todo to add ErrorMsg to all -FAILURE responses 2020-09-17 23:39:42 +00:00
Joey Hess
83df401d93
Merge branch 'batchasync' into master 2020-09-16 13:02:58 -04:00
Joey Hess
10f9107c1b
close 2020-09-16 13:02:35 -04:00
Joey Hess
929de3bb37
groundwork complete 2020-09-15 16:29:38 -04:00
Joey Hess
c6e159550d
update 2020-09-14 16:57:47 -04:00
Joey Hess
63d6cb27a9
thoughts 2020-09-10 13:13:39 -04:00
Joey Hess
6813373490
todo 2020-09-10 09:08:40 -04:00
Joey Hess
2bb933eb60
import: Retry downloads that fail
Also, using the transfer machinery for this makes eg, git-annex info show
in-progress imports, and makes --notify-start/finish work.
2020-09-04 13:54:05 -04:00
Joey Hess
46eb48d7c0
Retry transfers to exporttree=yes remotes same as for other remotes
The comment about noRetry is not well-justified, because transfers to many
remotes cannot be resumed, but retries are still allowed for those.
2020-09-04 13:24:08 -04:00
Joey Hess
1d244bafbd
Limit retrying of failed transfers when forward progress is being made to 5
To avoid some unusual edge cases where too much retrying could result in
far more data transfer than makes sense.
2020-09-04 12:46:37 -04:00
Joey Hess
5a9f518a42
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-02 12:25:52 -04:00
Joey Hess
854cd2ad47
httpalso: support exporttree=yes
Also tested what happens if the other special remote has importtree=yes
and exporttree=yes, and in that case, download via httpalso works too,
without needing to implement any importtree methods here.

It might be possible to make it automatically set exporttree=yes if the
--sameas does. Didn't try, will probably be layering issues.

Or perhaps it should be inherited by sameas like some
other configs? But then, wouldn't it also make sense to inherit
importree=yes? But as shown here, it's not needed by this kind of
remote.
2020-09-02 11:26:00 -04:00
Joey Hess
8656afd3e1
rename http special remote to httpalso
"http" was too generic and easy to confuse with web. The new name makes
clear it's used in addition to some other remote. And other protocols
can use the same naming scheme.
2020-09-02 10:41:53 -04:00
yarikoptic
871257ee23 Added a comment 2020-09-01 20:50:02 +00:00
Joey Hess
20b06266d1
thought 2020-09-01 16:04:40 -04:00
Joey Hess
48dde6d5b0
link 2020-09-01 16:02:03 -04:00
Joey Hess
4bcfd56902
todo 2020-09-01 16:00:49 -04:00
Joey Hess
955f309cd5
comment 2020-09-01 15:51:41 -04:00
Joey Hess
d80876920f
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-01 15:36:56 -04:00
Joey Hess
5b177317b4
comment 2020-09-01 15:36:28 -04:00
yarikoptic
a70260160f noise about 500s 2020-09-01 19:29:29 +00:00
Joey Hess
571ec900ac
Added http special remote, which is useful for accessing other remotes that publish content stored in them via http/https.
With automatic layout learning!
2020-09-01 15:16:35 -04:00
Joey Hess
fccc9ab442
note common need of these two todos 2020-09-01 12:13:41 -04:00
Joey Hess
d91b2b9fe2
close as dup 2020-09-01 11:58:08 -04:00
Joey Hess
95d9a3cf8a
Merge branch 'asyncexternal' 2020-08-14 16:00:49 -04:00
Joey Hess
05b2b46a82
async extension done 2020-08-14 15:24:34 -04:00