Commit graph

38070 commits

Author SHA1 Message Date
Joey Hess
e0ca1236ee
runshell: Update files atomically when preparing to run git-annex
This does not make it entirely idempotent, but it's a start.
2020-10-05 13:38:34 -04:00
Joey Hess
b91d97c3d2
small runshell optimisation
avoid some test -d for locale variables that are not set
2020-10-05 13:37:00 -04:00
Joey Hess
9e12e4c419
comment 2020-10-05 12:57:16 -04:00
Joey Hess
cd9a60bc7d
runshell: Fix a edge case where rm errors were sent to stdout, which could confuse things parsing git-annex output. 2020-10-05 12:44:40 -04:00
Joey Hess
3363cc7515
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-10-05 12:37:43 -04:00
Joey Hess
8c221eac83
update 2020-10-05 12:36:36 -04:00
yarikoptic
4bb980bfb9 Added a comment 2020-10-05 15:26:10 +00:00
Ilya_Shlyakhter
574d7a7cce Added a comment: standalone build 2020-10-04 15:45:16 +00:00
yarikoptic
77dab95e69 Initial report on hanging batched processes 2020-10-03 04:04:28 +00:00
yarikoptic
3be96e60b0 initial report on rm failing to remove cache/locales 2020-10-03 03:52:33 +00:00
Joey Hess
22ab6762cc
update to lts-16.16 2020-10-02 16:39:10 -04:00
Joey Hess
5555697ae6
Enable building with git-annex benchmark by default
Only turning it off when the criterion library is not installed.

Not enabled for osx or i386ancient yet since that will need some
invesitgation to update their respective stack.yaml files.
2020-10-02 13:57:10 -04:00
Joey Hess
10e62a810e
remove unused import 2020-10-02 13:45:09 -04:00
Joey Hess
37426920d8
Fix build with Benchmark build flag
Broke a while ago during optimisation work, and not noticed since the flag
is disabled by default.

This commit was sponsored by Brock Spratlen on Patreon.
2020-10-02 13:30:24 -04:00
kyle
742ce7ec87 Added a comment 2020-10-01 19:07:48 +00:00
yarikoptic
1bf55a8fc1 Added a comment 2020-10-01 17:26:10 +00:00
Joey Hess
37b1f2f2ed
response 2020-10-01 13:08:09 -04:00
Joey Hess
c6674b6271
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-10-01 12:53:24 -04:00
yarikoptic
23103e225a Added a comment 2020-10-01 14:06:34 +00:00
Joey Hess
41271e4eb4
avoid git check-ignore overhead on importing known files
isKnownImportLocation does a database lookup and there's an index
to make that lookup fast, so it's probably faster than talking to git
check-ignore. Checking the matcher is faster still.

While before the gitignore check was added it did not need to always
check isknown, now it does, because it's that or the more expensive
notignored. But at least we can skip notignored when a file is known,
which will often be the common case: Importing from a remote that's been
exported to, and/or imported from before, only new files will not be
known, so only those will need to check notignored.

At first, I had this:
	(matches <&&> (isknown <||> notignored)) <||> isknown
Notice that checks isknown every time, whether it matches or not.

So, it's no slower to instead do this:
	isknown <||> (matches <&&> notignored)
That has the benefit that, when it's known, it doesn't need to run
matches, which while faster than isknown, is still going to use some CPU.

And it perhaps more clearly expresses the condition: Any known file is
wanted, otherwise it's down to what matches and is not ignored.

This commit was sponsored by Jack Hill on Patren.
2020-09-30 11:20:44 -04:00
Joey Hess
c56efbbdb6
import: Check gitignores when importing trees from special remotes
It seemed best to do this, for consistency with every other way files can
get into a git-annex repo. Although it's just a bit strange that a local
.gitignore file affects the pseudo-commits made for the remote that's
imported from.

This commit was sponsored by Brett Eisenberg on Patreon.
2020-09-30 10:41:59 -04:00
Joey Hess
0033e08193
avoid a second traversal of the ImportableContents
Do all filtering in one pass.
2020-09-30 10:10:03 -04:00
Joey Hess
a9128d4b45
typo 2020-09-30 09:40:06 -04:00
Joey Hess
4c32499e82
Parse youtube-dl progress output
Which lets progress be displayed when doing concurrent downloads.
Amoung other things, like --json-progress etc.

The youtube-dl output is no longer displayed, except for any errors.

This commit was sponsored by Denis Dzyubenko on Patreon.
2020-09-29 17:53:48 -04:00
Joey Hess
4c7335caf3
add a pointer for git-annex repos over http
So users are less likely to try to use httpalso for them, which is not
its purpose.
2020-09-29 13:59:51 -04:00
Joey Hess
084b502c7a
httpalso: Support being used with special remotes that do not have encryption= in their config. 2020-09-29 13:56:27 -04:00
Joey Hess
59263d2c6f
add import 2020-09-29 13:51:51 -04:00
Joey Hess
b2cf284d2a
upgrade: Avoid an upgrade failure of a bare repo in unusual circumstances 2020-09-29 13:45:14 -04:00
Joey Hess
aa06c69a6f
close 2020-09-29 13:26:48 -04:00
Joey Hess
8615d4b4b3
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-29 13:25:31 -04:00
Joey Hess
1610d94776
addurl: Avoid a redundant git ignores check for speed
Ensure that checkCanAdd is used everywhere a file is added to git,
so git add is run with -f, presumably avoiding the work it would usually
do to check ignores.
2020-09-29 13:00:41 -04:00
rto@914936d87a43105f8c5df5ae4787140e7eb5d846
42bd4f1110 2020-09-29 16:51:49 +00:00
Joey Hess
d10cbaa084
comment 2020-09-29 12:25:40 -04:00
Joey Hess
c80d71fad1
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-29 12:12:25 -04:00
Joey Hess
dc274a6804
fix inverted logic in recent commit 2020-09-29 12:11:50 -04:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
a3083436dc removed 2020-09-29 08:36:37 +00:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
7b37f22880 Added a comment 2020-09-29 08:36:19 +00:00
lykos@d125a37d89b1cfac20829f12911656c40cb70018
ff0784dad9 Added a comment 2020-09-29 08:32:22 +00:00
mark@6b90344cdab3158eacb94a3944460d138afc9bef
26842956b5 Added a comment 2020-09-28 21:40:23 +00:00
mark@6b90344cdab3158eacb94a3944460d138afc9bef
a8e4539d15 Added a comment 2020-09-28 21:10:58 +00:00
mark@6b90344cdab3158eacb94a3944460d138afc9bef
32cd9a44ca 2020-09-28 21:09:27 +00:00
yarikoptic
67144b1c46 Added a comment 2020-09-28 19:44:08 +00:00
Joey Hess
e44a132085
Merge branch 'master' of ssh://git-annex.branchable.com into master 2020-09-28 15:30:23 -04:00
Joey Hess
658ea7ca3c
sync --no-content import from directory special remote
sync: When run without --content, import without copying from
importtree=yes directory special remotes. (Other special remotes may
support this later as well.)

This commit was sponsored by Svenne Krap on Patreon.
2020-09-28 15:29:08 -04:00
Joey Hess
3eaaec3113
consistently use importKey when available
This avoids import with --no-content and with --content potentially
generating two different trees, leading to a merge conflict when run in
two different clones of a repo. And it's necessary groundwork to make
git-annex sync --no-content import from special remotes that support
importKey.

Only the directory special remote currently supports importKey, and it
generates the same key as git-annex usually does, so there is no
behavior change for it.

Future special remotes will need to take care when adding importKey,
if it generates different keys. Added some warnings about that to
comments.

This commit was sponsored by Noam Kremen on Patreon.
2020-09-28 15:27:46 -04:00
Joey Hess
15c1ee16d9
import --no-content: Check annex.largefiles
Import small files into git, the same as is done when importing with content.
Which means, for small files, --no-content does download them.

If the largefiles expression needs the file content available
(due to mimetype or mimeencoding being used), the import will fail.

This commit was sponsored by Jake Vosloo on Patreon.
2020-09-28 13:28:57 -04:00
Joey Hess
8b74f01a26
split ProvidedInfo and UserProvidedInfo
The latter is for git-annex matchexpression and matching against it can
throw an exception. Splitting out the former reduces the potential for
mistakes and avoids needing to worry about matching against that
throwing an exception.

This is more groundwork for matching largefiles while importing,
without downloading content.

This commit was sponsored by Graham Spencer on Patreon.
2020-09-28 12:12:38 -04:00
Joey Hess
00dbe35fbc
allow matching on files whose content is not present
Anything that needs to examine the file content will fail to match,
or fall back to other available information. But the intent is that the
matcher be checked for matchNeedsFileContent and only be used if it does
not, so the exact behavior doesn't much matter as it should never
happen.

The real point of this is to not need to provide a dummy content file
when matching.

This commit was sponsored by Martin D on Patreon.
2020-09-28 11:17:46 -04:00
Joey Hess
9e676f062f
split out todo 2020-09-28 10:40:13 -04:00
Joey Hess
1aec0fc6b9
close as unreproducible 2020-09-28 10:13:15 -04:00