CI to automate build of cargo lockfiles on different Alpine releases for git-annex aports https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/git-annex
Find a file
Joey Hess c20358b671
incremental verify for byteRetriever special remotes
Several special remotes verify content while it is being retrieved,
avoiding a separate checksum pass. They are: S3, bup, ddar, and
gcrypt (with a local repository).

Not done when using chunking, yet.

Complicated by Retriever needing to change to be polymorphic. Which in turn
meant RankNTypes is needed, and also needed some code changes. The
change in Remote.External does not change behavior at all but avoids
the type checking failing because of a "rigid, skolem type" which
"would escape its scope". So I refactored slightly to make the type
checker's job easier there.

Unfortunately, directory uses fileRetriever (except when chunked),
so it is not amoung the improved ones. Fixing that would need a way for
FileRetriever to return a Verification. But, since the file retrieved
may be encrypted or chunked, it would be extra work to always
incrementally checksum the file while retrieving it. Hm.

Some other special remotes use fileRetriever, and so don't get incremental
verification, but could be converted to byteRetriever later. One is
GitLFS, which uses downloadConduit, which writes to the file, so could
verify as it goes. Other special remotes like web could too, but don't
use Remote.Helper.Special and so will need to be addressed separately.

Sponsored-by: Dartmouth College's DANDI project
2021-08-11 14:20:38 -04:00
Annex simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Assistant simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Backend simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Build don't use changelog version in commit message 2021-08-09 12:31:48 -04:00
CmdLine simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Command simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Config add: Significantly speed up adding lots of non-large files to git 2021-01-04 13:12:28 -04:00
Database simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
debian Enable building with git-annex benchmark by default 2020-10-02 13:57:10 -04:00
doc Added a comment: sorry 2021-08-10 16:28:33 +00:00
Git simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Limit fix longstanding indeterminite preferred content for duplicated file problem 2021-05-24 14:07:05 -04:00
Logs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Messages avoid using temp file size when deciding whether to retry failed transfer 2021-06-25 12:04:23 -04:00
P2P simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Remote incremental verify for byteRetriever special remotes 2021-08-11 14:20:38 -04:00
RemoteDaemon switch from hslogger to purpose-built Utility.Debug 2021-04-05 13:40:31 -04:00
standalone bump filepath-bytestring dep 2020-11-11 11:49:19 -04:00
static
templates
Test simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Types incremental verify for byteRetriever special remotes 2021-08-11 14:20:38 -04:00
Upgrade simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Utility simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
.ghci
.gitattributes
.gitignore
.mailmap
Annex.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Assistant.hs assistant: Fix a crash on startup by avoiding using forkProcess 2021-05-12 15:08:03 -04:00
Backend.hs move verifyKeyContent to Annex.Verify 2021-07-27 14:07:23 -04:00
bash-completion.bash
Benchmark.hs --size-limit exit 101 2021-06-04 16:43:47 -04:00
BuildFlags.hs Removed the S3 and WebDAV build flags 2020-09-08 12:42:59 -04:00
BuildInfo.hs
CHANGELOG incremental verify for byteRetriever special remotes 2021-08-11 14:20:38 -04:00
CmdLine.hs --size-limit exit 101 2021-06-04 16:43:47 -04:00
Command.hs addon commands 2021-02-02 16:32:49 -04:00
Common.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
Config.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
COPYRIGHT update year 2021-01-04 12:45:19 -04:00
Creds.hs avoid getting creds from environment during autoenable 2021-03-17 09:41:12 -04:00
Crypto.hs
git-annex.cabal releasing package git-annex version 8.20210803 2021-08-03 12:20:45 -04:00
git-annex.hs
git-union-merge.hs
Git.hs avoid crashing when there are remotes using unparseable urls 2021-01-18 14:59:08 -04:00
Key.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Limit.hs New matching options --excludesamecontent and --includesamecontent 2021-05-25 13:08:18 -04:00
Logs.hs filter-branch working aside from some edge cases 2021-05-17 13:24:58 -04:00
Makefile remove TAGS target 2021-07-02 14:38:28 -04:00
Messages.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
NEWS fix dist in latest NEWS entry 2020-09-14 12:05:12 -04:00
README
Remote.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Setup.hs
stack.yaml update to lts-16.27 2020-12-29 13:51:08 -04:00
Test.hs sync, merge: Added --allow-unrelated-histories option 2021-07-19 12:14:26 -04:00
Types.hs
Upgrade.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00

git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
to limitations in memory, checksumming time, or disk space.

For documentation, see doc/ or <https://git-annex.branchable.com/>