git-annex/doc/internals/git-remote-annex.mdwn

The [[git-remote-annex|/git-remote-annex]] command allows pushing a git
repository to a special remote, and later cloning from it.

This adds two new key types to git-annex, GITMANIFEST and a GITBUNDLE.

GITMANIFEST--$UUID is the manifest for a git repository stored in the
git-annex repository with that UUID. When that is not present,
GITMANIFEST--$UUID.bak is a backup copy that can be used instead.

GITBUNDLE--$UUID-$sha256 is a git bundle.

# format of the manifest file

An ordered list of bundle keys, one per line. 

Additionally, there may be bundle keys that are prefixed with "-".
These keys are not part of the current content of the git remote
and are in the process of being deleted.

(Lines end with unix `"\n"`, not `"\r\n"`.)

# exporttree=yes remotes

In an exporttree=yes remote, the GITMANIFEST and GITBUNDLE objects are
stored in the remote, under the `.git/annex/objects/` path.

# multiple special remotes in the same place

It's possible for multiple special remotes to point to the same
object storage.

This is why the UUID of the special remote is included in the GITMANIFEST
key, and in the annex:: uri.

# manually cloning from these files

If you are unable to use git-annex and need to clone a git repository
stored in such a special remote, this procedure will work:

* Find and download the GITMANIFEST
* Download each listed GITBUNDLE
* `git fetch` from each new bundle in order.
   (Note that later bundles can update refs from the versions in previous
   bundles.)

Note that, if a GITBUNDLE listed in the GITMANIFEST turns out not to exist,
a clone should treat this the same as if the GITMANIFEST were empty.
bundle objects are deleted when a push is made to the remote that
deletes all refs from it, and in a race between such a push and another
push of some refs, it is possible for the GITMANIFEST to refer to deleted
bundles.

When the special remote is encrypted, both the names and content of
the GITMANIFEST and GITBUNDLE will also be encrypted. To
decrypt those manually, see this [[fairly simple shell script using standard tools|tips/Decrypting_files_in_special_remotes_without_git-annex]].
preparing to merge git-remote-annex Update its todo with remaining items. Add changelog entry. Simplified internals document to no longer be notes to myself, but target users who want to understand how the data is stored and might want to extract these repos manually. Sponsored-by: Kevin Mueller on Patreon 2024-05-10 18:41:18 +00:00			`The [[git-remote-annex\|/git-remote-annex]] command allows pushing a git`
			`repository to a special remote, and later cloning from it.`

add Backend.GitRemoteAnnex Making GITBUNDLE be in the backend list allows those keys to be hashed to verify, both when git-remote-annex downloads them, and by other transfers and by git fsck. GITMANIFEST is not in the backend list, because those keys will never be stored in .git/annex/objects and can't be verified in any case. This does mean that git-annex version will include GITBUNDLE in the list of backends. Also documented these in backends.mdwn Sponsored-by: Kevin Mueller on Patreon 2024-05-07 17:42:12 +00:00			`This adds two new key types to git-annex, GITMANIFEST and a GITBUNDLE.`
added docs for git-remote-annex special remote contents Designed with the help of Timothy Sanders and Michael Hanke at Distribits 2024 2024-04-06 09:28:29 +00:00
proof of concent for push to git bundles with MANIFEST This is a shell script, so not final code, and it does not use git-annex at all, but it shows how to push to git bundles, listed in a MANIFEST, the same as the git-remote-annex program will eventually do. While developing this, I realized that the design needed to be changed slightly regarding where refs are stored. Since a push can delete a ref from a remote, storing each newly pushed ref in a bundle won't work, because deleting a ref would then entail deleting all old bundles and re-uploading from scratch. So instead, only the refs in the last bundle listed in the MANIFEST are the active refs. Any refs in prior bundles are just old refs that were stored previously (a reflog as it were). That means that, in a situation where two different people are pushing to the same special remote from different repos, whoever pushes last wins. Any refs pushed by the other person earlier will be ignored. This may not be desirable, and git-annex might be able use the git-annex branch to detect such situations and rescue the refs that got lost. Even without such a recovery process though, the refs that the other person thought they pushed will be preserved in their refs/namespaces/mine, so a pull followed by a push will generally resolve the situation. Note that the use of refs/namespaces/mine in the bundle is not really desirable, and it might be worth making a local clone of the repo in order to set up the refs that will be put in the bundle. Which seems to be the only way to avoid needing that. But it does need to maintain the refs/namespaces/mine/ in the git repo in order to remember what refs have been pushed to the remote before, in order to include them in the next bundle pushed. A name that includes the remote uuid will be needed in the final implementation. Anyway, this shell script seems to fully work, including incremental pushing, force pushing, and pushes that delete refs. Sponsored-by: Brett Eisenberg on Patreon 2024-04-25 20:38:34 +00:00			`GITMANIFEST--$UUID is the manifest for a git repository stored in the`
avoid interrupted push leaving remote without a manifest Added a backup manifest key, which is used if the main manifest key is not present. When uploading a new Manifest, it makes sure that it never drops one key except when the other key is present. It's entirely possible for the two manifest keys to get out of sync, due to races. The main one wins when it's present, it is possible for the main one being dropped to expose the backup one, which has a different push recorded. 2024-05-20 19:41:09 +00:00			`git-annex repository with that UUID. When that is not present,`
			`GITMANIFEST--$UUID.bak is a backup copy that can be used instead.`
added docs for git-remote-annex special remote contents Designed with the help of Timothy Sanders and Michael Hanke at Distribits 2024 2024-04-06 09:28:29 +00:00
avoid interrupted push leaving remote without a manifest Added a backup manifest key, which is used if the main manifest key is not present. When uploading a new Manifest, it makes sure that it never drops one key except when the other key is present. It's entirely possible for the two manifest keys to get out of sync, due to races. The main one wins when it's present, it is possible for the main one being dropped to expose the backup one, which has a different push recorded. 2024-05-20 19:41:09 +00:00			`GITBUNDLE--$UUID-$sha256 is a git bundle.`
added docs for git-remote-annex special remote contents Designed with the help of Timothy Sanders and Michael Hanke at Distribits 2024 2024-04-06 09:28:29 +00:00
			`# format of the manifest file`

working toward git-remote-annex using a special remote Not quite there yet. Also, changed the format of GITBUNDLE keys to use only one '-' after the UUID. A sha256 does not contain that character, so can just split at the last one. Amusingly, the sha256 will probably not actually be verified. A git bundle contains its own checksums that git uses to verify it. And if someone wanted to replace the content of a GITBUNDLE object, they could just edit the manifest to use a new one whose sha256 does verify. Sponsored-by: Nicholas Golder-Manning 2024-05-06 20:25:55 +00:00			`An ordered list of bundle keys, one per line.`

extend manifest with keys to be deleted This will eventually be used to recover from an interrupted fullPush and drop the old bundle keys it was unable to delete. Sponsored-by: Luke T. Shumaker on Patreon 2024-05-13 13:03:43 +00:00			`Additionally, there may be bundle keys that are prefixed with "-".`
			`These keys are not part of the current content of the git remote`
			`and are in the process of being deleted.`

working toward git-remote-annex using a special remote Not quite there yet. Also, changed the format of GITBUNDLE keys to use only one '-' after the UUID. A sha256 does not contain that character, so can just split at the last one. Amusingly, the sha256 will probably not actually be verified. A git bundle contains its own checksums that git uses to verify it. And if someone wanted to replace the content of a GITBUNDLE object, they could just edit the manifest to use a new one whose sha256 does verify. Sponsored-by: Nicholas Golder-Manning 2024-05-06 20:25:55 +00:00			(Lines end with unix `"\n"`, not `"\r\n"`.)
added docs for git-remote-annex special remote contents Designed with the help of Timothy Sanders and Michael Hanke at Distribits 2024 2024-04-06 09:28:29 +00:00
git-remote-annex support exporttree=yes remotes Put the annex objects in .git/annex/objects/ inside the export remote. This way, when importing from the remote, they will be filtered out. Note that, when importtree=yes, content identifiers are used, and this means that pushing to a remote updates the git-annex branch. Urk. Will need to try to prevent that later, but I already had a todo about that for other reasons. Untested! Sponsored-By: Brock Spratlen on Patreon 2024-05-13 15:37:47 +00:00			`# exporttree=yes remotes`

			`In an exporttree=yes remote, the GITMANIFEST and GITBUNDLE objects are`
			stored in the remote, under the `.git/annex/objects/` path.

avoid interrupted push leaving remote without a manifest Added a backup manifest key, which is used if the main manifest key is not present. When uploading a new Manifest, it makes sure that it never drops one key except when the other key is present. It's entirely possible for the two manifest keys to get out of sync, due to races. The main one wins when it's present, it is possible for the main one being dropped to expose the backup one, which has a different push recorded. 2024-05-20 19:41:09 +00:00			`# multiple special remotes in the same place`
parameterize manifest on UUID and expand slightly 2024-04-06 12:30:51 +00:00
avoid interrupted push leaving remote without a manifest Added a backup manifest key, which is used if the main manifest key is not present. When uploading a new Manifest, it makes sure that it never drops one key except when the other key is present. It's entirely possible for the two manifest keys to get out of sync, due to races. The main one wins when it's present, it is possible for the main one being dropped to expose the backup one, which has a different push recorded. 2024-05-20 19:41:09 +00:00			`It's possible for multiple special remotes to point to the same`
			`object storage.`
parameterize manifest on UUID and expand slightly 2024-04-06 12:30:51 +00:00
preparing to merge git-remote-annex Update its todo with remaining items. Add changelog entry. Simplified internals document to no longer be notes to myself, but target users who want to understand how the data is stored and might want to extract these repos manually. Sponsored-by: Kevin Mueller on Patreon 2024-05-10 18:41:18 +00:00			`This is why the UUID of the special remote is included in the GITMANIFEST`
			`key, and in the annex:: uri.`

			`# manually cloning from these files`

			`If you are unable to use git-annex and need to clone a git repository`
			`stored in such a special remote, this procedure will work:`

			`* Find and download the GITMANIFEST`
			`* Download each listed GITBUNDLE`
			* `git fetch` from each new bundle in order.
			`(Note that later bundles can update refs from the versions in previous`
			`bundles.)`

only delete bundles on pushEmpty This avoids some apparently otherwise unsolveable problems involving races that resulted in the manifest listing bundles that were deleted. Removed the annex-max-git-bundles config because it can't actually result in deleting old bundles. It would still be possible to have a config that controls how often to do a full push, which would avoid needing to download too many bundles on clone, as well as needing to checkpresent too many bundles in verifyManifest. But it would need a different name and description. 2024-05-21 14:41:48 +00:00			`Note that, if a GITBUNDLE listed in the GITMANIFEST turns out not to exist,`
			`a clone should treat this the same as if the GITMANIFEST were empty.`
			`bundle objects are deleted when a push is made to the remote that`
			`deletes all refs from it, and in a race between such a push and another`
			`push of some refs, it is possible for the GITMANIFEST to refer to deleted`
			`bundles.`

			`When the special remote is encrypted, both the names and content of`
			`the GITMANIFEST and GITBUNDLE will also be encrypted. To`
			`decrypt those manually, see this [[fairly simple shell script using standard tools\|tips/Decrypting_files_in_special_remotes_without_git-annex]].`