Merge branch 'master' into v8

This commit is contained in:
Joey Hess 2020-01-01 14:26:43 -04:00
commit 2cea674d1e
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
44 changed files with 665 additions and 140 deletions

View file

@ -0,0 +1,41 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2019-12-30T18:15:40Z"
content="""
When -J is used, recent versions use a separate pool of worker threads for
the checksumming than the downloading. So even with -J1 checksum of the
previous download will not block the next download.
I've thought about making this the default without -J.. It relies on
concurrent-output working well, which it sometimes may not, eg when
filenames are not valid unicode, or perhaps on a non-ANSI terminal, and so
far it's been worth not defaulting to -J1 to avoid breaking in such edge
cases.
Anyway, it seems to me using -J should avoid most of the overhead, except
of course for the remaining checksumming after all downloads finish.
Incremental checksumming could be done for some of the built-in remotes,
but not others like bittorrent which write out of order. Some transfers
can resume, and the checksumming would have to somehow catch
up to resume point, which adds significant complexity.
External remotes would need to send the content over a pipe for incremental
checksumming, so it would need a protocol extension.
git-annex's remote API does have the concept that a remote can sufficiently
verify the content of a file during transfer that additional checksumming
is not necessary. Currently only used for git remotes when hard linking an
object from a sibling remote. I don't think it actually matters what
checksum a remote uses to do such verification, as long as it's
cryptographically secure and runs on the local machine.
A protocol extension that let an external remote communicate to git-annex
that it had done such verification at the end of transfer is worth thinking
about.
Re Ilya's security concerns, as long as the external remote runs the
verification on the local machine, it seems there is no added security
impact.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="named pipes and external remotes"
date="2019-12-31T01:38:03Z"
content="""
\"External remotes would need to send the content over a pipe for incremental checksumming, so it would need a protocol extension\" -- in the current protocol, if you pass to the TRANSFER request, as the FILE parameter, a named pipe, would something break?
"""]]

View file

@ -0,0 +1,16 @@
Make `git-annex add --force-large` and `git-annex add --force-small`
add a specific file to annex or git, bypassing annex.largefiles
and all other configuration and state.
One reason to want this is that it avoids users doing stuff like this:
git -c annex.largefiles=anything annex add foo.c
Such a temporary setting of annex.largefiles can be problimatic, as explored in
<https://git-annex.branchable.com/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/>
Also, this could also be used to easily switch a file from one storage to
the other. I suppose the file would have to be touched first to make git-annex
add process it?
> [[done]] --[[Joey]]

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2019-12-26T20:34:02Z"
content="""
I've implemented annex.dotfiles in the `v8` branch.
I did not tie it to annex.largefiles in the end, spwhitton is right.
`git-annex add` behavior around dotfiles did change in a potentially
surprising way, since dotfiles are assumed to be non-large, they get added
to git. Users who have dotfiles that are large (or dotdirs containing large
files) will need to configure annex.largefiles and annex.dotfiles to avoid
those files being added to git. But, I don't think that will affect many
users, and it avoided a lot of complexity. At least such users can use this
and other semi-recent git-annex configs to avoid `git add` adding their
large dotfiles directly to git.
"""]]