Joey Hess
ebdce707da
fix typo
2020-09-22 13:26:49 -04:00
Joey Hess
361ef19999
wording
2020-09-22 12:39:33 -04:00
Joey Hess
41044de833
comment
2020-09-22 12:24:22 -04:00
yarikoptic
c44cd27520
Added a comment
2020-09-18 20:11:27 +00:00
Joey Hess
46a7fcef0d
close
2020-09-18 13:21:32 -04:00
Joey Hess
186c3827d0
comment
2020-09-18 13:21:00 -04:00
Joey Hess
500454935f
comment
2020-09-18 12:08:11 -04:00
Joey Hess
956ff1350a
Merge branch 'master' of ssh://git-annex.branchable.com into master
2020-09-18 12:00:12 -04:00
Joey Hess
81a38df5a7
add missing CHECKURL-FAILURE ErrorMsg to docs
2020-09-18 11:58:18 -04:00
yarikoptic
b48acde47d
some whining about check-ignore
2020-09-18 15:30:23 +00:00
yarikoptic
f913822c03
initial todo to add ErrorMsg to all -FAILURE responses
2020-09-17 23:39:42 +00:00
Joey Hess
83df401d93
Merge branch 'batchasync' into master
2020-09-16 13:02:58 -04:00
Joey Hess
10f9107c1b
close
2020-09-16 13:02:35 -04:00
Joey Hess
929de3bb37
groundwork complete
2020-09-15 16:29:38 -04:00
Joey Hess
c6e159550d
update
2020-09-14 16:57:47 -04:00
Joey Hess
63d6cb27a9
thoughts
2020-09-10 13:13:39 -04:00
Joey Hess
6813373490
todo
2020-09-10 09:08:40 -04:00
Joey Hess
2bb933eb60
import: Retry downloads that fail
...
Also, using the transfer machinery for this makes eg, git-annex info show
in-progress imports, and makes --notify-start/finish work.
2020-09-04 13:54:05 -04:00
Joey Hess
46eb48d7c0
Retry transfers to exporttree=yes remotes same as for other remotes
...
The comment about noRetry is not well-justified, because transfers to many
remotes cannot be resumed, but retries are still allowed for those.
2020-09-04 13:24:08 -04:00
Joey Hess
1d244bafbd
Limit retrying of failed transfers when forward progress is being made to 5
...
To avoid some unusual edge cases where too much retrying could result in
far more data transfer than makes sense.
2020-09-04 12:46:37 -04:00
Joey Hess
5a9f518a42
Merge branch 'master' of ssh://git-annex.branchable.com into master
2020-09-02 12:25:52 -04:00
Joey Hess
854cd2ad47
httpalso: support exporttree=yes
...
Also tested what happens if the other special remote has importtree=yes
and exporttree=yes, and in that case, download via httpalso works too,
without needing to implement any importtree methods here.
It might be possible to make it automatically set exporttree=yes if the
--sameas does. Didn't try, will probably be layering issues.
Or perhaps it should be inherited by sameas like some
other configs? But then, wouldn't it also make sense to inherit
importree=yes? But as shown here, it's not needed by this kind of
remote.
2020-09-02 11:26:00 -04:00
Joey Hess
8656afd3e1
rename http special remote to httpalso
...
"http" was too generic and easy to confuse with web. The new name makes
clear it's used in addition to some other remote. And other protocols
can use the same naming scheme.
2020-09-02 10:41:53 -04:00
yarikoptic
871257ee23
Added a comment
2020-09-01 20:50:02 +00:00
Joey Hess
20b06266d1
thought
2020-09-01 16:04:40 -04:00
Joey Hess
48dde6d5b0
link
2020-09-01 16:02:03 -04:00
Joey Hess
4bcfd56902
todo
2020-09-01 16:00:49 -04:00
Joey Hess
955f309cd5
comment
2020-09-01 15:51:41 -04:00
Joey Hess
d80876920f
Merge branch 'master' of ssh://git-annex.branchable.com into master
2020-09-01 15:36:56 -04:00
Joey Hess
5b177317b4
comment
2020-09-01 15:36:28 -04:00
yarikoptic
a70260160f
noise about 500s
2020-09-01 19:29:29 +00:00
Joey Hess
571ec900ac
Added http special remote, which is useful for accessing other remotes that publish content stored in them via http/https.
...
With automatic layout learning!
2020-09-01 15:16:35 -04:00
Joey Hess
fccc9ab442
note common need of these two todos
2020-09-01 12:13:41 -04:00
Joey Hess
d91b2b9fe2
close as dup
2020-09-01 11:58:08 -04:00
Joey Hess
95d9a3cf8a
Merge branch 'asyncexternal'
2020-08-14 16:00:49 -04:00
Joey Hess
05b2b46a82
async extension done
2020-08-14 15:24:34 -04:00
Joey Hess
1ecbac4025
branch
2020-08-12 16:27:02 -04:00
Joey Hess
ddf69bf5b8
draft async extension
2020-08-11 16:42:09 -04:00
Joey Hess
db1c6da84b
close
2020-08-11 14:01:22 -04:00
Lukey
7077a7c2e7
Added a comment
2020-08-04 14:10:33 +00:00
Joey Hess
88e5ebcda7
runshell LD_HWCAP_MASK=0 optimisation
2020-08-03 14:34:15 -04:00
yarikoptic
7bd1e392dc
Added a comment
2020-07-31 21:48:21 +00:00
yarikoptic
592db0629e
Added a comment
2020-07-31 21:37:48 +00:00
kyle
960180dece
Added a comment
2020-07-31 21:30:34 +00:00
yarikoptic
2b9d8b4e08
Added a comment
2020-07-31 21:23:57 +00:00
Joey Hess
f5724d78d4
comment
2020-07-31 16:47:23 -04:00
Joey Hess
8b4e5e6f68
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-31 14:58:02 -04:00
Joey Hess
c4ec52b9ae
Slightly sped up the linux standalone bundle
...
Reduce the number of directories listed in libdirs, which makes the linker
check a lot less dead ends looking for directories.
Eliminated some directories that didn't really contain shared libraries,
or only contained the linker.
That left only 2, one in lib and one in usr/lib, so consolidate those two.
Doing it this way, rather than just consolidating all libs that might exist
into a single directory means that, if there are optimised versions of some
libs, eg in lib/subarch/foo.so, and lib/subarch2/foo.so, they don't get
moved around in a way that would make the linker pick the wrong one.
2020-07-31 14:42:03 -04:00
Joey Hess
676257dfa8
comment
2020-07-31 13:29:48 -04:00
Ilya_Shlyakhter
85e57d8260
Added a comment: streaming data and external backends
2020-07-30 15:58:24 +00:00
Joey Hess
049807dbba
external backends implemented
2020-07-29 17:24:34 -04:00
yarikoptic
6b26802047
Added a comment
2020-07-28 15:26:43 +00:00
yarikoptic
61e96d7be4
Added a comment
2020-07-28 15:18:44 +00:00
yarikoptic
a846655d69
Added a comment
2020-07-27 21:40:55 +00:00
Joey Hess
3fafcc47bb
comment
2020-07-27 16:53:21 -04:00
Joey Hess
9d9f1f85d6
comment
2020-07-27 11:37:12 -04:00
Joey Hess
36d1621c35
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-27 11:34:03 -04:00
Joey Hess
3953c7a0ce
add DEBUG
2020-07-27 11:31:00 -04:00
ghen1
ad391da6b8
2020-07-27 15:24:21 +00:00
Joey Hess
32e1d7bc31
add
2020-07-24 14:11:08 -04:00
Joey Hess
c5ea2e9d12
better benchmark for move/copy speedup
2020-07-24 13:34:12 -04:00
Joey Hess
18f1fb5841
drop performance improvements
...
Sped up seeking files to drop by 2x, and also some performance
improvements to checking numcopies.
Interestingly, the seek speedup is not due to precaching, but I think is
due to calling getParsed earlier.
Annex.Drop had to be changed to check inAnnex there, since it was removed
from Command.Drop. All other users of Command.Drop already checked inAnnex
themselves.
This commit was sponsored by Ryan Newton on Patreon.
2020-07-24 13:27:46 -04:00
Joey Hess
d732ef1a89
move, copy: Sped up seeking for annexed files to operate on by a factor of nearly 2x.
2020-07-24 12:56:02 -04:00
Joey Hess
4685612f43
small git-annex get speedup
...
Remove an redundant inAnnex check. The checkContentPresent handles that,
and after the last commit also does in batch mode.
2020-07-22 14:29:30 -04:00
Joey Hess
1be92381ec
unify batch mode with non-batch by using AnnexedFileSeeker
2020-07-22 14:23:28 -04:00
Ilya_Shlyakhter
59917f8a6d
Added a comment: external backend protocol
2020-07-21 17:43:27 +00:00
Joey Hess
abd56fb019
Fix a bug in find --batch in the previous version.
2020-07-20 19:50:53 -04:00
Joey Hess
f71310fed0
comment
2020-07-20 14:19:13 -04:00
Joey Hess
d1300eca2e
draft external backend protocol
2020-07-20 14:05:49 -04:00
Joey Hess
1489fbbdde
bug
2020-07-19 18:26:57 -04:00
yarikoptic
6a05388877
Added a comment
2020-07-18 05:09:54 +00:00
yarikoptic
7ee0bcbee7
Added a comment
2020-07-18 05:09:32 +00:00
yarikoptic
4ab711e153
Added a comment
2020-07-18 04:50:12 +00:00
yarikoptic
a6b0147b7f
Added a comment
2020-07-18 04:49:49 +00:00
yarikoptic
d7b4df85e4
Added a comment
2020-07-18 04:34:26 +00:00
yarikoptic
5215fe92b9
Added a comment
2020-07-18 04:34:05 +00:00
yarikoptic
360de9446e
Added a comment
2020-07-18 03:57:20 +00:00
yarikoptic
c46b9ac4ae
initial 2nd wave of whining about startup time and to consider prelink or alike
2020-07-18 03:54:34 +00:00
yarikoptic
4f152089eb
Added a comment: Windows build of file (which includes libmagic)
2020-07-16 22:01:17 +00:00
Joey Hess
a3a8779501
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-16 15:08:51 -04:00
Ilya_Shlyakhter
77299ae6e5
Added a comment: external backends
2020-07-16 17:30:55 +00:00
Joey Hess
5ab3849da3
thought
2020-07-15 20:42:53 -04:00
Joey Hess
034f958b09
comment
2020-07-15 14:02:31 -04:00
Joey Hess
360dc386e7
comment
2020-07-15 10:08:37 -04:00
Joey Hess
1bc015bff4
tag datalad at yoh's req
2020-07-15 09:51:57 -04:00
Joey Hess
e66ba410fc
todo
2020-07-14 21:44:31 -04:00
Joey Hess
f9b4a9f650
update
2020-07-14 14:47:22 -04:00
Joey Hess
7b2d236556
importfeed: stream metadata for 5% speedup
...
On top of the 10% speedup from streaming url logs.
2020-07-14 14:35:26 -04:00
Joey Hess
535cdc8d48
importfeed: Made checking known urls step around 10% faster.
...
This was a bit disappointing, I was hoping for a 2x speedup. But, I think
the metadata lookup is wasting a lot of time and also needs to be made to
stream.
The changes to catObjectStreamLsTree were benchmarked to not also speed
up --all around 3% more. Seems I managed to make it polymorphic after all.
2020-07-14 12:47:51 -04:00
Joey Hess
75aab72d23
mostly done with location log precaching
...
Some nice wins.
2020-07-13 17:04:02 -04:00
Joey Hess
df58609804
convert sync to use seekFilteredKeys
...
This only speeds up sync --content from 34.75 to 33.17 seconds;
location log precaching will probably be a bigger win.
2020-07-13 15:02:52 -04:00
Joey Hess
c70ae68d7e
update
2020-07-13 11:49:24 -04:00
Joey Hess
415d394222
thought
2020-07-13 11:04:57 -04:00
Joey Hess
a32b6f9812
update
2020-07-10 15:49:03 -04:00
Joey Hess
412b09e17e
update
2020-07-10 15:23:12 -04:00
Joey Hess
2468eefc6d
2x speedup for annex file seeking on the horizon
2020-07-10 14:02:48 -04:00
Joey Hess
1df9e72a78
update
2020-07-10 13:31:47 -04:00
Joey Hess
6b9d1c1317
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-10 13:16:11 -04:00
Joey Hess
6e9fcf468d
streamkeys branch
2020-07-09 14:48:03 -04:00
branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e
bbc3800369
Added a comment: Update on my auto-commit / auto-sync scripts
2020-07-09 14:23:15 +00:00
Ilya_Shlyakhter
96aad5458b
Added a comment: re: git-annex-cat
2020-07-09 01:06:37 +00:00
Ilya_Shlyakhter
75b96059af
Added a comment: git-annex-cat
2020-07-09 00:21:02 +00:00
Joey Hess
9f6bd6cc05
add inRepoDetails
...
planned to use for an optimisation
most things using stagedDetails were not expecting to get dup files in a
conflicted merge and deal with them, so converted them to use
inRepoDetails.
2020-07-08 15:36:35 -04:00
Joey Hess
c1eaf5b930
note
2020-07-08 14:21:37 -04:00
Joey Hess
d08c178f97
avoid catObjectStream skipping over unavailable shas
...
Not needed as it's used for --all, but will be needed later.
2020-07-08 13:57:17 -04:00
Joey Hess
de3d7d044d
make catObjectStream support newline and carriage return in filenames
...
Turns out the %(rest) trick was not needed. Instead, just maintain a
list of files we've asked for, and each cat-file response is for the
next file in the list.
This actually benchmarks 25% faster than before! Very surprising, but it
must be due to needing to shove less data through the pipe, and parse
less.
2020-07-08 13:49:03 -04:00
Joey Hess
2cf6717aec
thoughts
2020-07-08 10:51:24 -04:00
Joey Hess
5849bd6340
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-07 16:50:26 -04:00
Joey Hess
afd9b2f667
idea
2020-07-07 16:49:44 -04:00
yarikoptic
c9d0bf0e6a
reassign to datalad - generic enhancement
2020-07-07 19:05:59 +00:00
Joey Hess
ba0adefe4c
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-07 14:19:46 -04:00
Joey Hess
d010ab04be
sped up the --all option by 2x to 16x by using git cat-file --buffer
...
This assumes that no location log files will have a newline or carriage
return in their name. catObjectStream skips any such files due to
cat-file not supporting them.
Keys have been prevented from containing newlines since 2011,
commit 480495beb4
2020-07-07 13:54:04 -04:00
timothy.sanders@a7ce3a8bae11a60e0c4cda9cb4aef24ec459bbab
3b6754e2a5
2020-07-07 10:26:00 +00:00
timothy.sanders@a7ce3a8bae11a60e0c4cda9cb4aef24ec459bbab
8a9323f5b5
2020-07-07 10:24:29 +00:00
Lukey
56f5d99ceb
Added a comment
2020-07-06 21:20:58 +00:00
Joey Hess
9468675ba9
note
2020-07-06 15:12:26 -04:00
Joey Hess
d66fc1a464
Revert "async exception safety for coprocesses"
...
This reverts commit 7013798df5
2020-07-06 15:11:28 -04:00
Joey Hess
dfa1c21b8a
comment
...
and update changelog with benchmark results
2020-07-06 13:39:42 -04:00
Joey Hess
9a2fbc2ea8
comment
2020-07-06 11:58:14 -04:00
Ilya_Shlyakhter
f6af30a7af
Added a comment
2020-07-03 19:55:36 +00:00
Joey Hess
d89b52086e
close
2020-07-03 14:31:12 -04:00
Joey Hess
85506a7015
import: Added --no-content option, which avoids downloading files from a special remote
...
Only supported by some special remotes: directory
I need to check the rest and they're currently missing methods until I do.
git-annex sync --no-content does not yet use this to do imports
2020-07-03 13:41:57 -04:00
Joey Hess
a8099b9896
thought
2020-07-03 12:02:07 -04:00
Joey Hess
89108d6f5a
thought
2020-07-02 21:56:00 -04:00
Joey Hess
e463ef1b91
comment
2020-07-02 20:13:19 -04:00
Joey Hess
8fc9788363
fix commment
2020-07-02 20:05:36 -04:00
yarikoptic
edef3c25b3
Added a comment: map2url?
2020-07-02 20:41:15 +00:00
Ilya_Shlyakhter
df65c4796d
Added a comment
2020-07-02 20:22:26 +00:00
yarikoptic
b7a78cbb26
Added a comment
2020-07-02 20:14:20 +00:00
Joey Hess
3353ff236a
comment
2020-07-02 15:30:16 -04:00
Joey Hess
f8ed8a916c
design
2020-07-02 14:35:59 -04:00
Joey Hess
a88b671bd9
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-02 14:17:31 -04:00
Joey Hess
caaeba0be9
thoughts
2020-07-02 14:15:47 -04:00
yarikoptic
1d51db3b02
Added a comment: more ideas for async implementation
2020-07-02 17:44:44 +00:00
Joey Hess
fe1f4632a4
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-02 10:00:11 -04:00
Joey Hess
00c9eb4c78
comment
2020-07-01 20:12:10 -04:00
Ilya_Shlyakhter
d03902f7ff
Added a comment: annex.thin for importing from directory special remote
2020-07-01 22:23:58 +00:00
Lukey
5a64acf790
Added a comment
2020-07-01 20:37:13 +00:00
Joey Hess
640dbaaaf8
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-01 15:15:47 -04:00
Joey Hess
11c2886578
overlapping todos
2020-07-01 15:06:36 -04:00
Ilya_Shlyakhter
d1232e385b
Added a comment
2020-07-01 17:33:49 +00:00
Ilya_Shlyakhter
6eb318cd53
Added a comment: git pack files
2020-07-01 17:32:45 +00:00
Joey Hess
424b1912d6
followup and add link
2020-07-01 12:28:44 -04:00
Joey Hess
a496ab602d
todo
2020-07-01 12:07:11 -04:00
Joey Hess
07dff32bd4
Merge branch 'master' of ssh://git-annex.branchable.com
2020-07-01 11:23:39 -04:00
Lukey
ffb03cc959
Added a comment
2020-07-01 14:32:12 +00:00
Joey Hess
98a8a6da81
todo
2020-06-30 18:41:47 -04:00
Joey Hess
8f508d4406
comments
2020-06-30 16:41:31 -04:00
Joey Hess
1d335520df
Merge branch 'master' of ssh://git-annex.branchable.com
2020-06-30 12:27:19 -04:00
Joey Hess
137450c9fe
thoughts
2020-06-30 12:24:08 -04:00
