138 lines
6 KiB
Markdown
138 lines
6 KiB
Markdown
In general, git-annex repositories that are "synchronized" (e.g. with
|
|
the [[git-annex-sync]] command, whatever the backend) have a global
|
|
namespace. Repositories will eventually converge to have very exactly
|
|
the same content, generally using git's push/pull/merge
|
|
mechanisms.
|
|
|
|
What if we do *not* wish to exactly have the same content across all
|
|
repositories, but still want to share some objects?
|
|
|
|
An example use case here is content (e.g. `.git/annex/objects` blobs)
|
|
sharing, without having to deliberately collaborate over a globally
|
|
consistent set of objects in the `master` branch. Think of a
|
|
decentralized [conference proceedings][] repository where each
|
|
conference could add their own content to a conference-specific
|
|
repository, while at the same time allowing a unified view in another,
|
|
more centralized repository, or allowing users to pick and choose
|
|
which conference they would want content from.
|
|
|
|
[conference proceedings]: https://github.com/RichiH/conference_proceedings
|
|
|
|
While each repository could have its own distinct branch, all
|
|
repositories will see all those branches and this may affect content
|
|
retention, as git-annex may consider files to be "in use" because they
|
|
are on some remote branch, for example. Furthermore, I consider git
|
|
branching to be a rather advanced topic in git usage. While git-annex
|
|
uses those mechanisms (e.g. the `git-annex` and `sync/*` branches),
|
|
those are generally hidden from the user until something goes
|
|
wrong. Therefore I looked into providing a more straightforward
|
|
approach to this problem for my users and myself.
|
|
|
|
In my use case, I have the following repositories:
|
|
|
|
* repoA: my own curated media collection
|
|
* repoB: a third-party media collection
|
|
|
|
I do not wish for my local curated collection (repoA) to be completely
|
|
synchronized with the third-party collection (repoB). This is because
|
|
we may have different tastes and retention policies: while I archive
|
|
everything, there are certain media I am not interested in. On the
|
|
other hand repoB might keep only (say) the last month of media and
|
|
disard older content but have a more varied collection, which only a
|
|
subset is interesting to me. Yet I still want to access some of that
|
|
content!
|
|
|
|
So I did the following to add the third party repository:
|
|
|
|
git remote add repoB example.net:repoB
|
|
git annex sync --no-push repoB
|
|
git annex get --from=repoB
|
|
|
|
This works well: I get the files from repoB locally. Of course, if
|
|
repoB expires some files, this will be impacted locally, but I can
|
|
always revert those choices without conflict, because I do not push
|
|
those back.
|
|
|
|
The downside of the `--no-push` option in [[git-annex-sync]] is that
|
|
it needs to be made explicit at each invocation of the
|
|
command. Furthermore, this option is not supported by the assistant,
|
|
which will happily sync the master branch to all remotes by default.
|
|
|
|
An alternative is to manually fetch and merge content:
|
|
|
|
git fetch repoB
|
|
git annex merge repoB
|
|
git reset HEAD^
|
|
# revert any possible changes upstream we don't want
|
|
git commit
|
|
|
|
Needless to say this quickly becomes quite messy, but it's the amazing
|
|
level of control git and git-annex provides, which obviously comes
|
|
with its price in complexity. Such a method will also be ignored by
|
|
the assistant and further `sync` commands.
|
|
|
|
To make sure those principles are respected in the assistant or a
|
|
plain `git annex sync` that may mistakenly be ran in that repository,
|
|
I need some special setting. There are the options I considered, in
|
|
[.gitconfig](https://manpages.debian.org/git-config.1.en.html) or [[git-annex]]'s config options:
|
|
|
|
* `remote.<name>.annex-ignore=true`: `sync` and `assistant` will not
|
|
sync *content* to the repository, but explicit `get --from=repoB`
|
|
will still work.
|
|
* `remote.<name>.annex-sync=false`: `sync` (and `assistant`?) will
|
|
not sync the git repository with the remote
|
|
* `remote.<name>.push=nothing`: git won't push by default, unless
|
|
branches are explicitly given, which may actually be the case for
|
|
git-annex, so unlikely to work.
|
|
* `remote.<name>.pushurl=/dev/null`: will completely disable any push
|
|
functionality to that remote. any sync will yield the following
|
|
error:
|
|
|
|
fatal: '/dev/null' does not appear to be a git repository
|
|
[...]
|
|
git-annex: sync: 1 failed
|
|
|
|
* `remote.<name>.pushurl=.`: will push to the local repo
|
|
instead. crude hack and may confuse the hell out of git-annex, but
|
|
at least doesn't yield errors.
|
|
|
|
A similar approach to hacking the `pushurl` is to make `repoB`
|
|
read-only to the user. This however, may trigger the activation of
|
|
`annex-ignore` by git-annex and will otherwise yield the same warnings
|
|
as the `pushurl=/dev/null` hack.
|
|
|
|
Right now, I am using `annex-sync = false` in `.git/config`. I have
|
|
also configured the repository to be in the "manual" [[standard
|
|
group|preferred_content/standard_groups]] which will avoid copying
|
|
files into that repository:
|
|
|
|
$ git annex group repoB manual
|
|
group repoB ok
|
|
(recording state in git...)
|
|
$ git annex wanted repoB standard
|
|
wanted repoB ok
|
|
(recording state in git...)
|
|
|
|
This is roughly equivalent to setting `annex-ignore = true`, yet it
|
|
allows for more flexibility. I could, for example, create custom
|
|
content expressions to sync certain folders automatically.
|
|
|
|
A disadvantage of the `annex-sync` settings is that it affects both
|
|
ways (push and pull), not just push, which is what I am interested
|
|
in. Although it could be argued that restricting both is fine here
|
|
because we want to manually review changes when we pull changes from
|
|
those remotes anyways.
|
|
|
|
The best approach may be to have git-annex respect the
|
|
`remote.<name>.push=nothing` setting. Another approach would be to add
|
|
`remote.<name>.annex-push` and `remote.<name>.annex-pull` settings
|
|
that would match the `sync --[no-]push --[no-]pull` flags.
|
|
|
|
Note that this is similar in concept to
|
|
[[todo/Bittorrent-like_features]], although here we assumes you
|
|
already have some transport to share anything you need, yet still have
|
|
to address the question of semi-synchronized git repositories in some
|
|
way.
|
|
|
|
I would obviously welcome additional comments and questions on this
|
|
approach. -- [[anarcat]]
|