Commit graph

3205 commits

Author SHA1 Message Date
Joey Hess
2bb933eb60
import: Retry downloads that fail
Also, using the transfer machinery for this makes eg, git-annex info show
in-progress imports, and makes --notify-start/finish work.
2020-09-04 13:54:05 -04:00
Joey Hess
46eb48d7c0
Retry transfers to exporttree=yes remotes same as for other remotes
The comment about noRetry is not well-justified, because transfers to many
remotes cannot be resumed, but retries are still allowed for those.
2020-09-04 13:24:08 -04:00
Joey Hess
1d244bafbd
Limit retrying of failed transfers when forward progress is being made to 5
To avoid some unusual edge cases where too much retrying could result in
far more data transfer than makes sense.
2020-09-04 12:46:37 -04:00
Joey Hess
5a9f518a42
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-02 12:25:52 -04:00
Joey Hess
854cd2ad47
httpalso: support exporttree=yes
Also tested what happens if the other special remote has importtree=yes
and exporttree=yes, and in that case, download via httpalso works too,
without needing to implement any importtree methods here.

It might be possible to make it automatically set exporttree=yes if the
--sameas does. Didn't try, will probably be layering issues.

Or perhaps it should be inherited by sameas like some
other configs? But then, wouldn't it also make sense to inherit
importree=yes? But as shown here, it's not needed by this kind of
remote.
2020-09-02 11:26:00 -04:00
Joey Hess
8656afd3e1
rename http special remote to httpalso
"http" was too generic and easy to confuse with web. The new name makes
clear it's used in addition to some other remote. And other protocols
can use the same naming scheme.
2020-09-02 10:41:53 -04:00
yarikoptic
871257ee23 Added a comment 2020-09-01 20:50:02 +00:00
Joey Hess
20b06266d1
thought 2020-09-01 16:04:40 -04:00
Joey Hess
48dde6d5b0
link 2020-09-01 16:02:03 -04:00
Joey Hess
4bcfd56902
todo 2020-09-01 16:00:49 -04:00
Joey Hess
955f309cd5
comment 2020-09-01 15:51:41 -04:00
Joey Hess
d80876920f
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-01 15:36:56 -04:00
Joey Hess
5b177317b4
comment 2020-09-01 15:36:28 -04:00
yarikoptic
a70260160f noise about 500s 2020-09-01 19:29:29 +00:00
Joey Hess
571ec900ac
Added http special remote, which is useful for accessing other remotes that publish content stored in them via http/https.
With automatic layout learning!
2020-09-01 15:16:35 -04:00
Joey Hess
fccc9ab442
note common need of these two todos 2020-09-01 12:13:41 -04:00
Joey Hess
d91b2b9fe2
close as dup 2020-09-01 11:58:08 -04:00
Joey Hess
95d9a3cf8a
Merge branch 'asyncexternal' 2020-08-14 16:00:49 -04:00
Joey Hess
05b2b46a82
async extension done 2020-08-14 15:24:34 -04:00
Joey Hess
1ecbac4025
branch 2020-08-12 16:27:02 -04:00
Joey Hess
ddf69bf5b8
draft async extension 2020-08-11 16:42:09 -04:00
Joey Hess
db1c6da84b
close 2020-08-11 14:01:22 -04:00
Lukey
7077a7c2e7 Added a comment 2020-08-04 14:10:33 +00:00
Joey Hess
88e5ebcda7
runshell LD_HWCAP_MASK=0 optimisation 2020-08-03 14:34:15 -04:00
yarikoptic
7bd1e392dc Added a comment 2020-07-31 21:48:21 +00:00
yarikoptic
592db0629e Added a comment 2020-07-31 21:37:48 +00:00
kyle
960180dece Added a comment 2020-07-31 21:30:34 +00:00
yarikoptic
2b9d8b4e08 Added a comment 2020-07-31 21:23:57 +00:00
Joey Hess
f5724d78d4
comment 2020-07-31 16:47:23 -04:00
Joey Hess
8b4e5e6f68
Merge branch 'master' of ssh://git-annex.branchable.com 2020-07-31 14:58:02 -04:00
Joey Hess
c4ec52b9ae
Slightly sped up the linux standalone bundle
Reduce the number of directories listed in libdirs, which makes the linker
check a lot less dead ends looking for directories.

Eliminated some directories that didn't really contain shared libraries,
or only contained the linker.

That left only 2, one in lib and one in usr/lib, so consolidate those two.

Doing it this way, rather than just consolidating all libs that might exist
into a single directory means that, if there are optimised versions of some
libs, eg in lib/subarch/foo.so, and lib/subarch2/foo.so, they don't get
moved around in a way that would make the linker pick the wrong one.
2020-07-31 14:42:03 -04:00
Joey Hess
676257dfa8
comment 2020-07-31 13:29:48 -04:00
Ilya_Shlyakhter
85e57d8260 Added a comment: streaming data and external backends 2020-07-30 15:58:24 +00:00
Joey Hess
049807dbba
external backends implemented 2020-07-29 17:24:34 -04:00
yarikoptic
6b26802047 Added a comment 2020-07-28 15:26:43 +00:00
yarikoptic
61e96d7be4 Added a comment 2020-07-28 15:18:44 +00:00
yarikoptic
a846655d69 Added a comment 2020-07-27 21:40:55 +00:00
Joey Hess
3fafcc47bb
comment 2020-07-27 16:53:21 -04:00
Joey Hess
9d9f1f85d6
comment 2020-07-27 11:37:12 -04:00
Joey Hess
36d1621c35
Merge branch 'master' of ssh://git-annex.branchable.com 2020-07-27 11:34:03 -04:00
Joey Hess
3953c7a0ce
add DEBUG 2020-07-27 11:31:00 -04:00
ghen1
ad391da6b8 2020-07-27 15:24:21 +00:00
Joey Hess
32e1d7bc31
add 2020-07-24 14:11:08 -04:00
Joey Hess
c5ea2e9d12
better benchmark for move/copy speedup 2020-07-24 13:34:12 -04:00
Joey Hess
18f1fb5841
drop performance improvements
Sped up seeking files to drop by 2x, and also some performance
improvements to checking numcopies.

Interestingly, the seek speedup is not due to precaching, but I think is
due to calling getParsed earlier.

Annex.Drop had to be changed to check inAnnex there, since it was removed
from Command.Drop. All other users of Command.Drop already checked inAnnex
themselves.

This commit was sponsored by Ryan Newton on Patreon.
2020-07-24 13:27:46 -04:00
Joey Hess
d732ef1a89
move, copy: Sped up seeking for annexed files to operate on by a factor of nearly 2x. 2020-07-24 12:56:02 -04:00
Joey Hess
4685612f43
small git-annex get speedup
Remove an redundant inAnnex check. The checkContentPresent handles that,
and after the last commit also does in batch mode.
2020-07-22 14:29:30 -04:00
Joey Hess
1be92381ec
unify batch mode with non-batch by using AnnexedFileSeeker 2020-07-22 14:23:28 -04:00
Ilya_Shlyakhter
59917f8a6d Added a comment: external backend protocol 2020-07-21 17:43:27 +00:00
Joey Hess
abd56fb019
Fix a bug in find --batch in the previous version. 2020-07-20 19:50:53 -04:00