proof of concent for push to git bundles with MANIFEST
This is a shell script, so not final code, and it does not use git-annex at all, but it shows how to push to git bundles, listed in a MANIFEST, the same as the git-remote-annex program will eventually do. While developing this, I realized that the design needed to be changed slightly regarding where refs are stored. Since a push can delete a ref from a remote, storing each newly pushed ref in a bundle won't work, because deleting a ref would then entail deleting all old bundles and re-uploading from scratch. So instead, only the refs in the last bundle listed in the MANIFEST are the active refs. Any refs in prior bundles are just old refs that were stored previously (a reflog as it were). That means that, in a situation where two different people are pushing to the same special remote from different repos, whoever pushes last wins. Any refs pushed by the other person earlier will be ignored. This may not be desirable, and git-annex might be able use the git-annex branch to detect such situations and rescue the refs that got lost. Even without such a recovery process though, the refs that the other person thought they pushed will be preserved in their refs/namespaces/mine, so a pull followed by a push will generally resolve the situation. Note that the use of refs/namespaces/mine in the bundle is not really desirable, and it might be worth making a local clone of the repo in order to set up the refs that will be put in the bundle. Which seems to be the only way to avoid needing that. But it does need to maintain the refs/namespaces/mine/ in the git repo in order to remember what refs have been pushed to the remote before, in order to include them in the next bundle pushed. A name that includes the remote uuid will be needed in the final implementation. Anyway, this shell script seems to fully work, including incremental pushing, force pushing, and pushes that delete refs. Sponsored-by: Brett Eisenberg on Patreon
This commit is contained in:
parent
f900c56ca3
commit
6ff4300bd1
2 changed files with 163 additions and 5 deletions
|
@ -1,6 +1,6 @@
|
|||
This adds two new object types to git-annex, GITMANIFEST and a GITBUNDLE.
|
||||
|
||||
GITMANIFEST-$UUID is the manifest for a git repository stored in the
|
||||
GITMANIFEST--$UUID is the manifest for a git repository stored in the
|
||||
git-annex repository with that UUID.
|
||||
|
||||
GITBUNDLE--sha256 is a git bundle.
|
||||
|
@ -9,16 +9,21 @@ GITBUNDLE--sha256 is a git bundle.
|
|||
|
||||
An ordered list of bundle keys, one per line.
|
||||
|
||||
The last bundle in the list provides all refs that are currently stored in
|
||||
the repository. The bundles before it in the list can incrementally provide
|
||||
objects, but not refs.
|
||||
|
||||
# fetching
|
||||
|
||||
1. download GITMANIFEST for the uuid of the special remote
|
||||
2. download each listed GITBUNDLE object that we don't have
|
||||
3. git fetch from bundles in timestamp order
|
||||
3. `git bundle unpack` each bundle in order
|
||||
4. `git fetch` from the last bundle listed in the manifest
|
||||
|
||||
# pushing (incrementally)
|
||||
|
||||
1. create git bundle containing refs to push, and objects since
|
||||
the previously pushed refs
|
||||
1. create git bundle all refs that will be stored in the repository,
|
||||
and objects since the previously pushed refs
|
||||
2. hash to calculate GITBUNDLE key
|
||||
3. upload GITBUNDLE object
|
||||
4. download current manifest
|
||||
|
@ -26,7 +31,8 @@ An ordered list of bundle keys, one per line.
|
|||
|
||||
# pushing (replacing incrementals with single bundle)
|
||||
|
||||
1. create git bundle containing refs to push and all objects
|
||||
1. create git bundle containing all refs stored in the repository, and all
|
||||
objects
|
||||
2. hash to calculate GITBUNDLE object name
|
||||
3. upload GITBUNDLE object
|
||||
4. download current manifest
|
||||
|
|
152
git-remote-annex
Executable file
152
git-remote-annex
Executable file
|
@ -0,0 +1,152 @@
|
|||
#!/bin/sh
|
||||
|
||||
set -x
|
||||
|
||||
# remember the refs that were uploaded already
|
||||
git for-each-ref refs/namespaces/mine/ > .git/old-refs
|
||||
|
||||
# Unfortunately, git bundle omits prerequisites that are omitted once,
|
||||
# even if they are used by a later ref.
|
||||
# For example, where x is a ref that points at A, and y is a ref
|
||||
# that points at B (which has A as its parent), git bundle x A..y
|
||||
# will omit inclding the x ref in the bundle at all.
|
||||
check_prereq () {
|
||||
# So, if a sha is one of the other refs that will be included in the
|
||||
# bundle, it cannot be treated as a prerequisite.
|
||||
if git for-each-ref refs/namespaces/mine/ | grep -Pv "\t$2$" | awk '{print $1}' | grep -q "$1"; then
|
||||
echo "$2"
|
||||
else
|
||||
# And, if one of the other refs that will be included in the bundle
|
||||
# is an ancestor of the sha, it cannot be treated as a prerequisite.
|
||||
if [ -n "$(for x in $(git for-each-ref refs/namespaces/mine/ | grep -Pv "\t$2$" | awk '{print $1}'); do git log --oneline -n1 $x..$1; done)" ]; then
|
||||
echo "$2"
|
||||
else
|
||||
echo "$1..$2"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
while read foo; do
|
||||
case "$foo" in
|
||||
capabilities)
|
||||
echo fetch
|
||||
echo push
|
||||
echo
|
||||
;;
|
||||
list*)
|
||||
if [ -e "MANIFEST" ]; then
|
||||
# Only list the refs in the last bundle
|
||||
# listed in the manifest. Each push
|
||||
# includes all refs in its bundle.
|
||||
f=$(tail -n 1 MANIFEST)
|
||||
if [ -n "$f" ]; then
|
||||
# refs in the bundle may end up prefixed with refs/namespaces/mine/
|
||||
# when the intent is for the bundle to include a
|
||||
# ref with the name that comes after that.
|
||||
git bundle list-heads $f | sed 's/refs\/namespaces\/mine\///'
|
||||
fi
|
||||
fi
|
||||
echo
|
||||
;;
|
||||
fetch*)
|
||||
dofetch=1
|
||||
;;
|
||||
push*)
|
||||
set -- $foo
|
||||
x="$2"
|
||||
# src ref if prefixed with a + in a forced push
|
||||
srcref="$(echo "$x" | cut -d : -f 1 | sed 's/^\+//')"
|
||||
dstref="$(echo "$x" | cut -d : -f 2)"
|
||||
if [ -z "$srcref" ]; then
|
||||
git update-ref -d refs/namespaces/mine/"$dstref"
|
||||
else
|
||||
# Need to create a bundle containing $dstref, but
|
||||
# don't want to overwrite that ref in the local
|
||||
# repo. Unfortunately, git bundle does not support
|
||||
# GIT_NAMESPACE, so it's not possible to do that
|
||||
# without making a clone of the whole git repo.
|
||||
# Instead, just create a ref under the namespace
|
||||
# refs/namespaces/mine/ that will be put in the
|
||||
# bundle.
|
||||
git update-ref refs/namespaces/mine/"$dstref" "$srcref"
|
||||
fi
|
||||
dopush=1
|
||||
;;
|
||||
# docs say a blank line ends communication, but that's not
|
||||
# accurate, actually a blank line comes after a series of
|
||||
# fetch or push commands, and also according to the docs,
|
||||
# another series of commands could follow
|
||||
"")
|
||||
if [ "$dofetch" ]; then
|
||||
if [ -e "MANIFEST" ]; then
|
||||
for f in $(cat MANIFEST); do
|
||||
git bundle unbundle "$f" >/dev/null 2>&1
|
||||
done
|
||||
fi
|
||||
echo
|
||||
dofetch=""
|
||||
fi
|
||||
if [ "$dopush" ]; then
|
||||
if [ -z "$(git for-each-ref refs/namespaces/mine/)" ]; then
|
||||
# deleted all refs
|
||||
if [ -e "MANIFEST" ]; then
|
||||
for f in $(cat MANIFEST); do
|
||||
rm "$f"
|
||||
done
|
||||
rm MANIFEST
|
||||
touch MANIFEST
|
||||
fi
|
||||
else
|
||||
# set REPUSH=1 to do a full push
|
||||
# rather than incremental
|
||||
if [ "$REPUSH" ]; then
|
||||
rm MANIFEST
|
||||
rm *.bundle
|
||||
git for-each-ref refs/namespaces/mine/ | awk '{print $3}' | \
|
||||
git bundle create --quiet new.bundle --stdin
|
||||
else
|
||||
# incremental bundle
|
||||
IFS="
|
||||
"
|
||||
(for l in $(git for-each-ref refs/namespaces/mine/); do
|
||||
r=$(echo "$l" | awk '{print $3}')
|
||||
newsha=$(echo "$l" | awk '{print $1}')
|
||||
oldsha=$(grep -P "\t$r$" .git/old-refs | awk '{print $1}')
|
||||
if [ -n "$oldsha" ]; then
|
||||
# include changes from $oldsha to $r when there are some
|
||||
if [ -n "$(git log --oneline $oldsha..$r)" ]; then
|
||||
check_prereq "$oldsha" "$r"
|
||||
else
|
||||
if [ "$oldsha" = "$newsha" ]; then
|
||||
# $r is unchanged from last push, so include
|
||||
# the minimum data to make the bundle contain $r
|
||||
rparentsha=$(git log -n 2 "$r" --format='%H' | tail -n+2)
|
||||
if [ -n "$rparentsha" ]; then
|
||||
check_prereq "$rparentsha" "$r"
|
||||
else
|
||||
# $r has no parent so include it as is
|
||||
echo "$r"
|
||||
fi
|
||||
else
|
||||
# $oldsha is not a parent of $r, so
|
||||
# include $r and all its parents
|
||||
echo "$r"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
# no old version was pushed so include $r and all its parents
|
||||
echo "$r"
|
||||
fi
|
||||
done) \
|
||||
| git bundle create --quiet new.bundle --stdin
|
||||
fi
|
||||
sha1=$(sha1sum new.bundle | awk '{print $1}')
|
||||
mv new.bundle "$sha1.bundle"
|
||||
echo "$sha1.bundle" >> MANIFEST
|
||||
fi
|
||||
echo
|
||||
dopush=""
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
Loading…
Add table
Add a link
Reference in a new issue