Merge branch 'master' into v8
This commit is contained in:
commit
2cea674d1e
44 changed files with 665 additions and 140 deletions
|
@ -0,0 +1,41 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2019-12-30T18:15:40Z"
|
||||
content="""
|
||||
When -J is used, recent versions use a separate pool of worker threads for
|
||||
the checksumming than the downloading. So even with -J1 checksum of the
|
||||
previous download will not block the next download.
|
||||
|
||||
I've thought about making this the default without -J.. It relies on
|
||||
concurrent-output working well, which it sometimes may not, eg when
|
||||
filenames are not valid unicode, or perhaps on a non-ANSI terminal, and so
|
||||
far it's been worth not defaulting to -J1 to avoid breaking in such edge
|
||||
cases.
|
||||
|
||||
Anyway, it seems to me using -J should avoid most of the overhead, except
|
||||
of course for the remaining checksumming after all downloads finish.
|
||||
|
||||
Incremental checksumming could be done for some of the built-in remotes,
|
||||
but not others like bittorrent which write out of order. Some transfers
|
||||
can resume, and the checksumming would have to somehow catch
|
||||
up to resume point, which adds significant complexity.
|
||||
|
||||
External remotes would need to send the content over a pipe for incremental
|
||||
checksumming, so it would need a protocol extension.
|
||||
|
||||
git-annex's remote API does have the concept that a remote can sufficiently
|
||||
verify the content of a file during transfer that additional checksumming
|
||||
is not necessary. Currently only used for git remotes when hard linking an
|
||||
object from a sibling remote. I don't think it actually matters what
|
||||
checksum a remote uses to do such verification, as long as it's
|
||||
cryptographically secure and runs on the local machine.
|
||||
|
||||
A protocol extension that let an external remote communicate to git-annex
|
||||
that it had done such verification at the end of transfer is worth thinking
|
||||
about.
|
||||
|
||||
Re Ilya's security concerns, as long as the external remote runs the
|
||||
verification on the local machine, it seems there is no added security
|
||||
impact.
|
||||
"""]]
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="named pipes and external remotes"
|
||||
date="2019-12-31T01:38:03Z"
|
||||
content="""
|
||||
\"External remotes would need to send the content over a pipe for incremental checksumming, so it would need a protocol extension\" -- in the current protocol, if you pass to the TRANSFER request, as the FILE parameter, a named pipe, would something break?
|
||||
"""]]
|
16
doc/todo/git_annex_add_option_to_control_to_where.mdwn
Normal file
16
doc/todo/git_annex_add_option_to_control_to_where.mdwn
Normal file
|
@ -0,0 +1,16 @@
|
|||
Make `git-annex add --force-large` and `git-annex add --force-small`
|
||||
add a specific file to annex or git, bypassing annex.largefiles
|
||||
and all other configuration and state.
|
||||
|
||||
One reason to want this is that it avoids users doing stuff like this:
|
||||
|
||||
git -c annex.largefiles=anything annex add foo.c
|
||||
|
||||
Such a temporary setting of annex.largefiles can be problimatic, as explored in
|
||||
<https://git-annex.branchable.com/bugs/A_case_where_file_tracked_by_git_unexpectedly_becomes_annex_pointer_file/>
|
||||
|
||||
Also, this could also be used to easily switch a file from one storage to
|
||||
the other. I suppose the file would have to be touched first to make git-annex
|
||||
add process it?
|
||||
|
||||
> [[done]] --[[Joey]]
|
|
@ -0,0 +1,18 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 5"""
|
||||
date="2019-12-26T20:34:02Z"
|
||||
content="""
|
||||
I've implemented annex.dotfiles in the `v8` branch.
|
||||
|
||||
I did not tie it to annex.largefiles in the end, spwhitton is right.
|
||||
|
||||
`git-annex add` behavior around dotfiles did change in a potentially
|
||||
surprising way, since dotfiles are assumed to be non-large, they get added
|
||||
to git. Users who have dotfiles that are large (or dotdirs containing large
|
||||
files) will need to configure annex.largefiles and annex.dotfiles to avoid
|
||||
those files being added to git. But, I don't think that will affect many
|
||||
users, and it avoided a lot of complexity. At least such users can use this
|
||||
and other semi-recent git-annex configs to avoid `git add` adding their
|
||||
large dotfiles directly to git.
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue