mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

only delete bundles on pushEmpty

Browse source 
This avoids some apparently otherwise unsolveable problems involving
races that resulted in the manifest listing bundles that were deleted.

Removed the annex-max-git-bundles config because it can't actually
result in deleting old bundles. It would still be possible to have a
config that controls how often to do a full push, which would avoid
needing to download too many bundles on clone, as well as needing to
checkpresent too many bundles in verifyManifest. But it would need a
different name and description.
This commit is contained in:
Joey Hess 2024-05-21 10:41:48 -04:00
parent f544946b09
commit 3e7324bbcb
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 53 additions and 103 deletions
Download patch file Download diff file Expand all files Collapse all files

11
doc/git-annex.mdwn
View file

@ -1648,17 +1648,6 @@ Remotes are configured using these settings in `.git/config`.
remotes, and is set when using [[git-annex-initremote]](1) with the
`--private` option.
* `remote.<name>.annex-max-git-bundles`, `annex.max-git-bundles`
When using [[git-remote-annex]] to store a git repository in a special
remote, this configures how many separate git bundle objects to store
in the special remote before re-uploading a single git bundle that contains
the entire git repository.
The default is 100, which aims to avoid often needing to often re-upload,
while preventing a new clone needing to download too many objects. Set to
0 to disable re-uploading.
* `remote.<name>.annex-allow-encrypted-gitrepo`
Setting this to true allows using [[git-remote-annex]] to push the git

16
doc/git-remote-annex.mdwn
View file

@ -38,23 +38,29 @@ variables. Some special remotes may also need environment variables to be
set when pulling or pushing.
The git repository is stored in the special remote using special annex objects
with names starting with "GITMANIFEST--" and "GITBUNDLE--". For details about
with names starting with "GITMANIFEST" and "GITBUNDLE". For details about
how the git repository is stored, see
<https://git-annex.branchable.com/internals/git-remote-annex/>
Pushes to a special remote are usually done incrementally. However,
sometimes the whole git repository (but not the annex) needs to be
re-uploaded. That is done when force pushing a ref, or deleting a
ref from the remote. It's also done when too many git bundles
accumulate in the special remote, as configured by the
`remote.<name>.annex-max-git-bundles` git config.
ref from the remote.
The special remote accumulates one GITBUNDLE object per push, and old
objects are usually not deleted. This means that refs pushed to the special
remote can still be accessed even after deleting or overwriting them.
A push that deletes every ref from the special remote does delete all
the accumulated GITBUNDLE objects. But of course, making such a push
means that someone clones from the special remote at that point in time
will see an empty remote.
Like any git repository, a git repository stored on a special remote can
have conflicting things pushed to it from different places. This mostly
works the same as any other git repository, eg a push that overwrites other
work will be prevented unless forced. However, it is possible, when
conflicting pushes are being done at the same time, for one of the pushes
to be overwritten by the other one. In this sitiation, the push will appear
to be overwritten by the other one. In this situation, the push will appear
to have succeeded, but pulling later will show the true situation.
# SEE ALSO

13
doc/internals/git-remote-annex.mdwn
View file

@ -43,6 +43,13 @@ stored in such a special remote, this procedure will work:
(Note that later bundles can update refs from the versions in previous
bundles.)
When the special remote is encrypted, the GITMANIFEST and GITBUNDLE will
also be encrypted. To decrypt those manually, see this
[[fairly simple shell script using standard tools|tips/Decrypting_files_in_special_remotes_without_git-annex]].
Note that, if a GITBUNDLE listed in the GITMANIFEST turns out not to exist,
a clone should treat this the same as if the GITMANIFEST were empty.
bundle objects are deleted when a push is made to the remote that
deletes all refs from it, and in a race between such a push and another
push of some refs, it is possible for the GITMANIFEST to refer to deleted
bundles.
When the special remote is encrypted, both the names and content of
the GITMANIFEST and GITBUNDLE will also be encrypted. To
decrypt those manually, see this [[fairly simple shell script using standard tools|tips/Decrypting_files_in_special_remotes_without_git-annex]].

61
doc/todo/git-remote-annex.mdwn
View file

@ -12,6 +12,9 @@ This is implememented and working. Remaining todo list for it:
* Test incremental push edge cases involving checkprereq.
* Cloning a special remote with an empty manifest results in a repo where
git fetch fails, claiming the special remote is encrypted, when it's not.
* Cloning from an annex:: url with importtree=yes doesn't work
(with or without exporttree=yes). This is because the ContentIdentifier
db is not populated. It should be possible to work around this.
@ -60,61 +63,3 @@ This is implememented and working. Remaining todo list for it:
They would have to use git fetch --prune to notice the deletion.
Once the user does notice, they can re-push their ref to recover.
Can this be improved?
## bundle deletion problems
Deleting bundles results in some problems involving races,
detailed below, that result in the manifest file listing a bundle that has
been deleted. Which breaks cloning, and is data loss, and so *must*
be solved before release.
* A race between an incremental push and a full push can result in
a bundle that the incremental push is based on being deleted by the full
push, and then incremental push's manifest file being written later.
Which will prevent cloning or some pulls from working.
A fix: Make each full push (and emptying push) also write to a fallback
manifest file that is only written by full pushes (and emptying pushes),
not incremental pushes. When fetching the main manifest file, always
check that all bundles mentioned in it are still in the remote. If any
are missing, fetch and use the fallback manifest file instead.
* A race between two full pushes can also result in the manifest file listing
a bundle that has been deleted:
Start with a full push of bundle A.
Then there are 2 racing full pushes X and Y, of bundle A and B
respectively. With this series of operations:
1. Y: write bundle B
1. Y: read manifest (listing A)
1. Y: write B to manifest
1. X: write bundle A
1. Y: delete bundle A
1. X: read manifest (listing B)
1. X: write A to manifest
1. X: delete bundle B
Which results in a manifest that lists A, but that bundle was deleted.
The problems above *could* be solved by not deleting bundles, but that is
unsatisfactory.
Old bundles could be deleted after some period of time. But a process can
be suspended at any point and resumed later, so the race windows can be
arbitrarily wide.
What if only emptying pushes delete bundles? If a manifest file refers to a
bundle that has been deleted, that can be treated the same as if the
manifest file was empty, because we know that, for that bundle to have been
deleted, there must have been an emptying push. So this would work.
It is kind of a cop-out, because it requires the user to do an emptying
push from time to time. But by doing that, the user will expect that
someone who pulls at that point gets an empty repository.
Note that a race between an emptying push an a ref push will result in the
emptying push winning, so the ref push is lost. This is the same behavior
as can happen in a push race not involving deletion though, and any
improvements that are made to the UI around that will also help with this.