git-annex/git-remote-annex
Joey Hess 8b56d6b283
fix conflicting push situation
In a situation where there are two repos that are diverged and each pushes
in turn to git-remote-annex, the first to push updates it. Then the second
push fails because it is not a fast-forward. The problem is, before git
push fails with "non-fast-forward", it actually calls git-remote-annex
with push.

So, to the user it appears as if the push failed, but it actually reached
the remote, and overwrote the other push!

The only solution to this seems to be for git-remote-annex push to notice
when a non-force-push would overwrite a ref stored in the remote, and
refuse to push that ref, returning an error to git. This seems strange,
why would git make remote helpers implement that when it later checks the
same thing itself?

With this fix, it's still possible for a race to overwrite a change to
the MANIFEST and lose work that was pushed from the other repo. But that
needs two pushes to be running at the same time. From the user's
perspective, that situation is the same as if one repo pushed new work,
then the other repo did a git push --force, overwriting the first repo's
push. In the first repo, another push will then fail as a non
fast-forward, and the user can recover as usual. But, a MANIFEST
overwrite will leave bundle files in the remote that are not listed in
the MANIFEST. It seems likely that git-annex will eventually be able to
detect that after the fact and clean it up. Eg, it can learn all bundles
that are stored in the remote using the location log, and compare them
to the MANIFEST to find bundles that got lost.

The race can also appear to the user as if they pushed a ref, but then
it got deleted from the remote. This happens when two two pushes are
pushing different ref names. This might be harder for the user to
notice; git fetch does not indicate that a remote ref got deleted.
They would have to use git fetch --prune to notice the deletion.
Once the user does notice, they can re-push their ref to recover.

Sponsored-by: Jack Hill on Patreon
2024-04-26 15:03:04 -04:00

197 lines
6.3 KiB
Bash
Executable file

#!/bin/sh
TOPDIR=..
set -x
# remember the refs that were uploaded already
git for-each-ref refs/namespaces/mine/ > .git/old-refs
rm -f .git/push-response
# Unfortunately, git bundle omits prerequisites that are omitted once,
# even if they are used by a later ref.
# For example, where x is a ref that points at A, and y is a ref
# that points at B (which has A as its parent), git bundle x A..y
# will omit inclding the x ref in the bundle at all.
check_prereq () {
# So, if a sha is one of the other refs that will be included in the
# bundle, it cannot be treated as a prerequisite.
if git for-each-ref refs/namespaces/mine/ | grep -Pv "\t$2$" | awk '{print $1}' | grep -q "$1"; then
echo "$2"
else
# And, if one of the other refs that will be included in the bundle
# is an ancestor of the sha, it cannot be treated as a prerequisite.
if [ -n "$(for x in $(git for-each-ref refs/namespaces/mine/ | grep -Pv "\t$2$" | awk '{print $1}'); do git log --oneline -n1 $x..$1; done)" ]; then
echo "$2"
else
echo "$1..$2"
fi
fi
}
while read foo; do
case "$foo" in
capabilities)
echo fetch
echo push
echo
;;
list*)
if [ -e "$TOPDIR/MANIFEST" ]; then
# Only list the refs in the last bundle
# listed in the manifest. Each push
# includes all refs in its bundle.
f=$(tail -n 1 $TOPDIR/MANIFEST)
if [ -n "$f" ]; then
# stash the listed refs for later
# checking in push
git bundle list-heads $TOPDIR/$f > .git/listed-refs
# refs in the bundle may end up prefixed with refs/namespaces/mine/
# when the intent is for the bundle to include a
# ref with the name that comes after that.
sed 's/refs\/namespaces\/mine\///' .git/listed-refs
fi
fi
echo
;;
fetch*)
dofetch=1
;;
push*)
set -- $foo
x="$2"
# src ref is prefixed with a + in a forced push
forcedpush=""
if echo "$x" | cut -d : -f 1 | egrep -q '^\+'; then
forcedpush=1
fi
srcref="$(echo "$x" | cut -d : -f 1 | sed 's/^\+//')"
dstref="$(echo "$x" | cut -d : -f 2)"
# Need to create a bundle containing $dstref, but
# don't want to overwrite that ref in the local
# repo. Unfortunately, git bundle does not support
# GIT_NAMESPACE, so it's not possible to do that
# without making a clone of the whole git repo.
# Instead, just create a ref under the namespace
# refs/namespaces/mine/ that will be put in the
# bundle.
mydstref=refs/namespaces/mine/"$dstref"
if [ -z "$srcref" ]; then
git update-ref -d "$mydstref"
touch .git/push-response
echo "ok $dstref" >> .git/push-response
else
if [ ! "$forcedpush" ]; then
# check if the push would overwrite
# work in the ref currently stored in the
# remote, if so refuse to do it
prevsha=$(grep " $mydstref$" .git/listed-refs | awk '{print $1}')
newsha=$(git rev-parse "$srcref")
if [ -n "$prevsha" ] && [ "$prevsha" != "$newsha" ] && [ -z "$(git log --oneline $prevsha..$newsha 2>/dev/null)" ]; then
touch .git/push-response
echo "error $dstref non-fast-forward" >> .git/push-response
else
touch .git/push-response
echo "ok $dstref" >> .git/push-response
git update-ref "$mydstref" "$srcref"
fi
else
git update-ref "$mydstref" "$srcref"
touch .git/push-response
echo "ok $dstref" >> .git/push-response
fi
fi
dopush=1
;;
# docs say a blank line ends communication, but that's not
# accurate, actually a blank line comes after a series of
# fetch or push commands, and also according to the docs,
# another series of commands could follow
"")
if [ "$dofetch" ]; then
if [ -e "$TOPDIR/MANIFEST" ]; then
for f in $(cat $TOPDIR/MANIFEST); do
git bundle unbundle "$TOPDIR/$f" >/dev/null 2>&1
done
fi
echo
dofetch=""
fi
if [ "$dopush" ]; then
# if some refs cannot be pushed, refuse to
# push anything. It would be difficult to
# push only some refs, because the bundle
# needs to contain all refs, and some refs
# on the remote may contain objects we have
# not fetched yet.
if egrep -q "^error" .git/push-response; then
sed 's/^ok \(.*\)/error \1 unable to push this due to other pushed ref being non-fast-forward/' .git/push-response > .git/push-response.new
mv .git/push-response.new .git/push-response
else
if [ -z "$(git for-each-ref refs/namespaces/mine/)" ]; then
# deleted all refs
if [ -e "$TOPDIR/MANIFEST" ]; then
for f in $(cat $TOPDIR/MANIFEST); do
rm "$TOPDIR/$f"
done
rm $TOPDIR/MANIFEST
touch $TOPDIR/MANIFEST
fi
else
# set REPUSH=1 to do a full push
# rather than incremental
if [ "$REPUSH" ]; then
rm $TOPDIR/MANIFEST
rm $TOPDIR/*.bundle
git for-each-ref refs/namespaces/mine/ | awk '{print $3}' | \
git bundle create --quiet $TOPDIR/new.bundle --stdin
else
# incremental bundle
IFS="
"
(for l in $(git for-each-ref refs/namespaces/mine/); do
r=$(echo "$l" | awk '{print $3}')
newsha=$(echo "$l" | awk '{print $1}')
oldsha=$(grep -P "\t$r$" .git/old-refs | awk '{print $1}')
if [ -n "$oldsha" ]; then
# include changes from $oldsha to $r when there are some
if [ -n "$(git log --oneline $oldsha..$r)" ]; then
check_prereq "$oldsha" "$r"
else
if [ "$oldsha" = "$newsha" ]; then
# $r is unchanged from last push, so include
# the minimum data to make the bundle contain $r
rparentsha=$(git log -n 2 "$r" --format='%H' | tail -n+2)
if [ -n "$rparentsha" ]; then
check_prereq "$rparentsha" "$r"
else
# $r has no parent so include it as is
echo "$r"
fi
else
# $oldsha is not a parent of $r, so
# include $r and all its parents
echo "$r"
fi
fi
else
# no old version was pushed so include $r and all its parents
echo "$r"
fi
done) \
| git bundle create --quiet $TOPDIR/new.bundle --stdin
fi
sha1=$(sha1sum $TOPDIR/new.bundle | awk '{print $1}')
mv $TOPDIR/new.bundle "$TOPDIR/$sha1.bundle"
echo "$sha1.bundle" >> $TOPDIR/MANIFEST
fi
fi
cat .git/push-response
rm -f .git/push-response
echo
dopush=""
fi
;;
esac
done