fix conflicting push situation
In a situation where there are two repos that are diverged and each pushes in turn to git-remote-annex, the first to push updates it. Then the second push fails because it is not a fast-forward. The problem is, before git push fails with "non-fast-forward", it actually calls git-remote-annex with push. So, to the user it appears as if the push failed, but it actually reached the remote, and overwrote the other push! The only solution to this seems to be for git-remote-annex push to notice when a non-force-push would overwrite a ref stored in the remote, and refuse to push that ref, returning an error to git. This seems strange, why would git make remote helpers implement that when it later checks the same thing itself? With this fix, it's still possible for a race to overwrite a change to the MANIFEST and lose work that was pushed from the other repo. But that needs two pushes to be running at the same time. From the user's perspective, that situation is the same as if one repo pushed new work, then the other repo did a git push --force, overwriting the first repo's push. In the first repo, another push will then fail as a non fast-forward, and the user can recover as usual. But, a MANIFEST overwrite will leave bundle files in the remote that are not listed in the MANIFEST. It seems likely that git-annex will eventually be able to detect that after the fact and clean it up. Eg, it can learn all bundles that are stored in the remote using the location log, and compare them to the MANIFEST to find bundles that got lost. The race can also appear to the user as if they pushed a ref, but then it got deleted from the remote. This happens when two two pushes are pushing different ref names. This might be harder for the user to notice; git fetch does not indicate that a remote ref got deleted. They would have to use git fetch --prune to notice the deletion. Once the user does notice, they can re-push their ref to recover. Sponsored-by: Jack Hill on Patreon
This commit is contained in:
Joey Hess
parent
99491f572f
commit
8b56d6b283
2 changed files with 107 additions and 66 deletions
|
|
@ -35,11 +35,9 @@ objects, but not refs.
|
||||||
objects
|
objects
|
||||||
2. hash to calculate GITBUNDLE object name
|
2. hash to calculate GITBUNDLE object name
|
||||||
3. upload GITBUNDLE object
|
3. upload GITBUNDLE object
|
||||||
4. download current manifest
|
4. download old manifest
|
||||||
5. remove all old GITBUNDLES from the manifest, and add new GITBUNDLE at
|
4. upload new manifest listing only the single new GITBUNDLE
|
||||||
the end. Note that it's possible for the manifest to contain GITBUNDLES
|
5. delete all other GITBUNDLEs that were listed in the old manifest
|
||||||
that were not in the last fetched manifest, if so those must be
|
|
||||||
preserved, and the new GITBUNDLE appended
|
|
||||||
|
|
||||||
# multiple GITMANIFEST files
|
# multiple GITMANIFEST files
|
||||||
|
|
||||||
|
|
|
||||||
165
git-remote-annex
165
git-remote-annex
|
|
@ -7,6 +7,8 @@ set -x
|
||||||
# remember the refs that were uploaded already
|
# remember the refs that were uploaded already
|
||||||
git for-each-ref refs/namespaces/mine/ > .git/old-refs
|
git for-each-ref refs/namespaces/mine/ > .git/old-refs
|
||||||
|
|
||||||
|
rm -f .git/push-response
|
||||||
|
|
||||||
# Unfortunately, git bundle omits prerequisites that are omitted once,
|
# Unfortunately, git bundle omits prerequisites that are omitted once,
|
||||||
# even if they are used by a later ref.
|
# even if they are used by a later ref.
|
||||||
# For example, where x is a ref that points at A, and y is a ref
|
# For example, where x is a ref that points at A, and y is a ref
|
||||||
|
|
@ -42,10 +44,13 @@ while read foo; do
|
||||||
# includes all refs in its bundle.
|
# includes all refs in its bundle.
|
||||||
f=$(tail -n 1 $TOPDIR/MANIFEST)
|
f=$(tail -n 1 $TOPDIR/MANIFEST)
|
||||||
if [ -n "$f" ]; then
|
if [ -n "$f" ]; then
|
||||||
|
# stash the listed refs for later
|
||||||
|
# checking in push
|
||||||
|
git bundle list-heads $TOPDIR/$f > .git/listed-refs
|
||||||
# refs in the bundle may end up prefixed with refs/namespaces/mine/
|
# refs in the bundle may end up prefixed with refs/namespaces/mine/
|
||||||
# when the intent is for the bundle to include a
|
# when the intent is for the bundle to include a
|
||||||
# ref with the name that comes after that.
|
# ref with the name that comes after that.
|
||||||
git bundle list-heads $TOPDIR/$f | sed 's/refs\/namespaces\/mine\///'
|
sed 's/refs\/namespaces\/mine\///' .git/listed-refs
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
echo
|
echo
|
||||||
|
|
@ -56,21 +61,46 @@ while read foo; do
|
||||||
push*)
|
push*)
|
||||||
set -- $foo
|
set -- $foo
|
||||||
x="$2"
|
x="$2"
|
||||||
# src ref if prefixed with a + in a forced push
|
# src ref is prefixed with a + in a forced push
|
||||||
|
forcedpush=""
|
||||||
|
if echo "$x" | cut -d : -f 1 | egrep -q '^\+'; then
|
||||||
|
forcedpush=1
|
||||||
|
fi
|
||||||
srcref="$(echo "$x" | cut -d : -f 1 | sed 's/^\+//')"
|
srcref="$(echo "$x" | cut -d : -f 1 | sed 's/^\+//')"
|
||||||
dstref="$(echo "$x" | cut -d : -f 2)"
|
dstref="$(echo "$x" | cut -d : -f 2)"
|
||||||
|
# Need to create a bundle containing $dstref, but
|
||||||
|
# don't want to overwrite that ref in the local
|
||||||
|
# repo. Unfortunately, git bundle does not support
|
||||||
|
# GIT_NAMESPACE, so it's not possible to do that
|
||||||
|
# without making a clone of the whole git repo.
|
||||||
|
# Instead, just create a ref under the namespace
|
||||||
|
# refs/namespaces/mine/ that will be put in the
|
||||||
|
# bundle.
|
||||||
|
mydstref=refs/namespaces/mine/"$dstref"
|
||||||
if [ -z "$srcref" ]; then
|
if [ -z "$srcref" ]; then
|
||||||
git update-ref -d refs/namespaces/mine/"$dstref"
|
git update-ref -d "$mydstref"
|
||||||
|
touch .git/push-response
|
||||||
|
echo "ok $dstref" >> .git/push-response
|
||||||
else
|
else
|
||||||
# Need to create a bundle containing $dstref, but
|
if [ ! "$forcedpush" ]; then
|
||||||
# don't want to overwrite that ref in the local
|
# check if the push would overwrite
|
||||||
# repo. Unfortunately, git bundle does not support
|
# work in the ref currently stored in the
|
||||||
# GIT_NAMESPACE, so it's not possible to do that
|
# remote, if so refuse to do it
|
||||||
# without making a clone of the whole git repo.
|
prevsha=$(grep " $mydstref$" .git/listed-refs | awk '{print $1}')
|
||||||
# Instead, just create a ref under the namespace
|
newsha=$(git rev-parse "$srcref")
|
||||||
# refs/namespaces/mine/ that will be put in the
|
if [ -n "$prevsha" ] && [ "$prevsha" != "$newsha" ] && [ -z "$(git log --oneline $prevsha..$newsha 2>/dev/null)" ]; then
|
||||||
# bundle.
|
touch .git/push-response
|
||||||
git update-ref refs/namespaces/mine/"$dstref" "$srcref"
|
echo "error $dstref non-fast-forward" >> .git/push-response
|
||||||
|
else
|
||||||
|
touch .git/push-response
|
||||||
|
echo "ok $dstref" >> .git/push-response
|
||||||
|
git update-ref "$mydstref" "$srcref"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
git update-ref "$mydstref" "$srcref"
|
||||||
|
touch .git/push-response
|
||||||
|
echo "ok $dstref" >> .git/push-response
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
dopush=1
|
dopush=1
|
||||||
;;
|
;;
|
||||||
|
|
@ -89,63 +119,76 @@ while read foo; do
|
||||||
dofetch=""
|
dofetch=""
|
||||||
fi
|
fi
|
||||||
if [ "$dopush" ]; then
|
if [ "$dopush" ]; then
|
||||||
if [ -z "$(git for-each-ref refs/namespaces/mine/)" ]; then
|
# if some refs cannot be pushed, refuse to
|
||||||
# deleted all refs
|
# push anything. It would be difficult to
|
||||||
if [ -e "$TOPDIR/MANIFEST" ]; then
|
# push only some refs, because the bundle
|
||||||
for f in $(cat $TOPDIR/MANIFEST); do
|
# needs to contain all refs, and some refs
|
||||||
rm "$TOPDIR/$f"
|
# on the remote may contain objects we have
|
||||||
done
|
# not fetched yet.
|
||||||
rm $TOPDIR/MANIFEST
|
if egrep -q "^error" .git/push-response; then
|
||||||
touch $TOPDIR/MANIFEST
|
sed 's/^ok \(.*\)/error \1 unable to push this due to other pushed ref being non-fast-forward/' .git/push-response > .git/push-response.new
|
||||||
fi
|
mv .git/push-response.new .git/push-response
|
||||||
else
|
else
|
||||||
# set REPUSH=1 to do a full push
|
if [ -z "$(git for-each-ref refs/namespaces/mine/)" ]; then
|
||||||
# rather than incremental
|
# deleted all refs
|
||||||
if [ "$REPUSH" ]; then
|
if [ -e "$TOPDIR/MANIFEST" ]; then
|
||||||
rm $TOPDIR/MANIFEST
|
for f in $(cat $TOPDIR/MANIFEST); do
|
||||||
rm $TOPDIR/*.bundle
|
rm "$TOPDIR/$f"
|
||||||
git for-each-ref refs/namespaces/mine/ | awk '{print $3}' | \
|
done
|
||||||
git bundle create --quiet $TOPDIR/new.bundle --stdin
|
rm $TOPDIR/MANIFEST
|
||||||
|
touch $TOPDIR/MANIFEST
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
# incremental bundle
|
# set REPUSH=1 to do a full push
|
||||||
IFS="
|
# rather than incremental
|
||||||
|
if [ "$REPUSH" ]; then
|
||||||
|
rm $TOPDIR/MANIFEST
|
||||||
|
rm $TOPDIR/*.bundle
|
||||||
|
git for-each-ref refs/namespaces/mine/ | awk '{print $3}' | \
|
||||||
|
git bundle create --quiet $TOPDIR/new.bundle --stdin
|
||||||
|
else
|
||||||
|
# incremental bundle
|
||||||
|
IFS="
|
||||||
"
|
"
|
||||||
(for l in $(git for-each-ref refs/namespaces/mine/); do
|
(for l in $(git for-each-ref refs/namespaces/mine/); do
|
||||||
r=$(echo "$l" | awk '{print $3}')
|
r=$(echo "$l" | awk '{print $3}')
|
||||||
newsha=$(echo "$l" | awk '{print $1}')
|
newsha=$(echo "$l" | awk '{print $1}')
|
||||||
oldsha=$(grep -P "\t$r$" .git/old-refs | awk '{print $1}')
|
oldsha=$(grep -P "\t$r$" .git/old-refs | awk '{print $1}')
|
||||||
if [ -n "$oldsha" ]; then
|
if [ -n "$oldsha" ]; then
|
||||||
# include changes from $oldsha to $r when there are some
|
# include changes from $oldsha to $r when there are some
|
||||||
if [ -n "$(git log --oneline $oldsha..$r)" ]; then
|
if [ -n "$(git log --oneline $oldsha..$r)" ]; then
|
||||||
check_prereq "$oldsha" "$r"
|
check_prereq "$oldsha" "$r"
|
||||||
else
|
else
|
||||||
if [ "$oldsha" = "$newsha" ]; then
|
if [ "$oldsha" = "$newsha" ]; then
|
||||||
# $r is unchanged from last push, so include
|
# $r is unchanged from last push, so include
|
||||||
# the minimum data to make the bundle contain $r
|
# the minimum data to make the bundle contain $r
|
||||||
rparentsha=$(git log -n 2 "$r" --format='%H' | tail -n+2)
|
rparentsha=$(git log -n 2 "$r" --format='%H' | tail -n+2)
|
||||||
if [ -n "$rparentsha" ]; then
|
if [ -n "$rparentsha" ]; then
|
||||||
check_prereq "$rparentsha" "$r"
|
check_prereq "$rparentsha" "$r"
|
||||||
|
else
|
||||||
|
# $r has no parent so include it as is
|
||||||
|
echo "$r"
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
# $r has no parent so include it as is
|
# $oldsha is not a parent of $r, so
|
||||||
|
# include $r and all its parents
|
||||||
echo "$r"
|
echo "$r"
|
||||||
fi
|
fi
|
||||||
else
|
fi
|
||||||
# $oldsha is not a parent of $r, so
|
else
|
||||||
# include $r and all its parents
|
# no old version was pushed so include $r and all its parents
|
||||||
echo "$r"
|
echo "$r"
|
||||||
fi
|
fi
|
||||||
fi
|
done) \
|
||||||
else
|
| git bundle create --quiet $TOPDIR/new.bundle --stdin
|
||||||
# no old version was pushed so include $r and all its parents
|
fi
|
||||||
echo "$r"
|
sha1=$(sha1sum $TOPDIR/new.bundle | awk '{print $1}')
|
||||||
fi
|
mv $TOPDIR/new.bundle "$TOPDIR/$sha1.bundle"
|
||||||
done) \
|
echo "$sha1.bundle" >> $TOPDIR/MANIFEST
|
||||||
| git bundle create --quiet $TOPDIR/new.bundle --stdin
|
|
||||||
fi
|
fi
|
||||||
sha1=$(sha1sum $TOPDIR/new.bundle | awk '{print $1}')
|
|
||||||
mv $TOPDIR/new.bundle "$TOPDIR/$sha1.bundle"
|
|
||||||
echo "$sha1.bundle" >> $TOPDIR/MANIFEST
|
|
||||||
fi
|
fi
|
||||||
|
cat .git/push-response
|
||||||
|
rm -f .git/push-response
|
||||||
echo
|
echo
|
||||||
dopush=""
|
dopush=""
|
||||||
fi
|
fi
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue