mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/git-annex-migrate.mdwn

113 lines
3.3 KiB
Text
Raw Normal View History

finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
# NAME
git-annex migrate - switch data to different backend
# SYNOPSIS
git annex migrate `[path ...]`
started migrate --update This is most of the way there, but not quite working. The layout of migrate.tree/ needs to be changed to follow this approach. git log will list all the files in tree order, so the new layout needs to alternate old and new keys. Can that be done? git may not document tree order, or may not preserve it here. Alternatively, change to using git log --format=raw and extract the tree header from that, then use git diff --raw $tree:migrate.tree/old $tree:migrate.tree/new That will be a little more expensive, but only when there are lots of migrations. Sponsored-by: Joshua Antonishen on Patreon
2023-12-07 19:50:52 +00:00
git annex migrate --update
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
# DESCRIPTION
Changes the specified annexed files to use the default key-value backend
(or the one specified with `--backend`). Only files whose content
display filenames in migrate --update Have to go to a lot of bother to find them, but I think it's worth it for usability. Sponsored-by: Luke T. Shumaker on Patreon
2023-12-07 22:00:09 +00:00
is currently present are migrated.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
display filenames in migrate --update Have to go to a lot of bother to find them, but I think it's worth it for usability. Sponsored-by: Luke T. Shumaker on Patreon
2023-12-07 22:00:09 +00:00
Note that the content is also still stored using the old keys after
migration. When possible, hard links are used to avoid that taking up
extra disk space. Use `git annex unused` to find and remove the old keys.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
distributed migration for pull and sync --content pull, sync: When operating on content, automatically hard link objects that have been migrated. Added annex.syncmigrations config that can be set to false to prevent pull and sync from migrating object content. I think that true is a good default for this config, because it avoids users having to re-download migrated content or learning about migration. But, some users will surely not like it, whether because it does take some time (especially for the first git-annex branch scan when there is a long history), or because they want to deal with it manually, or because their filesystem doesn't support hard links and they don't want it to copy objects. Sponsored-by: k0ld on Patreon
2023-12-08 18:18:18 +00:00
Normally, nothing will be done to specified files that are already using
the new backend. However, if a backend changes the information it uses to
construct a key, this can also be used to migrate files to use the new key
format.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
# OPTIONS
started migrate --update This is most of the way there, but not quite working. The layout of migrate.tree/ needs to be changed to follow this approach. git log will list all the files in tree order, so the new layout needs to alternate old and new keys. Can that be done? git may not document tree order, or may not preserve it here. Alternatively, change to using git log --format=raw and extract the tree header from that, then use git diff --raw $tree:migrate.tree/old $tree:migrate.tree/new That will be a little more expensive, but only when there are lots of migrations. Sponsored-by: Joshua Antonishen on Patreon
2023-12-07 19:50:52 +00:00
* `--update`
This updates the local repository for migrations that were performed
migrate --apply And avoid migrate --update/--aply migrating when the new key was already present in the repository, and got dropped. Luckily, the location log allows distinguishing from the new key never having been present! That is mostly useful for --apply because otherwise dropped files would keep coming back until the old objects were reaped as unused. But it seemed to make sense to also do it for --update. for consistency in edge cases if nothing else. One case where --update can use it is when one branch got migrated earlier, and we dropped the file, and now another branch has migrated the same file. Sponsored-by: Jack Hill on Patreon
2023-12-08 17:23:03 +00:00
elsewhere. Only new migrations since the last time this was run will
be performed.
This does not modify the working tree, but only hard links
(or in some cases copies) annex objects to their new keys.
distributed migration for pull and sync --content pull, sync: When operating on content, automatically hard link objects that have been migrated. Added annex.syncmigrations config that can be set to false to prevent pull and sync from migrating object content. I think that true is a good default for this config, because it avoids users having to re-download migrated content or learning about migration. But, some users will surely not like it, whether because it does take some time (especially for the first git-annex branch scan when there is a long history), or because they want to deal with it manually, or because their filesystem doesn't support hard links and they don't want it to copy objects. Sponsored-by: k0ld on Patreon
2023-12-08 18:18:18 +00:00
`git-annex pull` and `git-annex sync --content` automatically do this,
unless the `annex.syncmigrations` config is set to false.
migrate --apply And avoid migrate --update/--aply migrating when the new key was already present in the repository, and got dropped. Luckily, the location log allows distinguishing from the new key never having been present! That is mostly useful for --apply because otherwise dropped files would keep coming back until the old objects were reaped as unused. But it seemed to make sense to also do it for --update. for consistency in edge cases if nothing else. One case where --update can use it is when one branch got migrated earlier, and we dropped the file, and now another branch has migrated the same file. Sponsored-by: Jack Hill on Patreon
2023-12-08 17:23:03 +00:00
Note that older versions of git-annex did not record migrations in a
way that this can use. Migrations performed with those older versions
had to be manually run in each clone of the repository.
* `--apply`
This applies all recorded migrations to the local repository. It is the
non-incremental form of `--update`.
One situation where this can be useful is when `git-annex migrate
--update` has been run, but since then un-migrated
objects have entered the repository. Using this option ensures that
any such objects get migrated.
started migrate --update This is most of the way there, but not quite working. The layout of migrate.tree/ needs to be changed to follow this approach. git log will list all the files in tree order, so the new layout needs to alternate old and new keys. Can that be done? git may not document tree order, or may not preserve it here. Alternatively, change to using git log --format=raw and extract the tree header from that, then use git diff --raw $tree:migrate.tree/old $tree:migrate.tree/new That will be a little more expensive, but only when there are lots of migrations. Sponsored-by: Joshua Antonishen on Patreon
2023-12-07 19:50:52 +00:00
Note that older versions of git-annex did not record migrations in a
way that this can use. Migrations performed with those older versions
had to be manually run in each clone of the repository.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
* `--backend`
Specify the new key-value backend to use for migrated data.
* `--force`
Force migration of keys that are already using the new backend.
* file matching options
The [[git-annex-matching-options]](1)
can be used to specify files to migrate.
split out common options to its own page and mention it on each subcommand page Sometimes users would get confused because an option they were looking for was not mentioned on a subcommand's man page, and they had not noticed that the main git-annex man page had a list of common options. This change lets each subcommand mention the common options, similarly to how the matching options are handled. This commit was sponsored by Svenne Krap on Patreon.
2021-05-10 19:00:13 +00:00
* Also the [[git-annex-common-options]](1) can be used.
migrate: New --remove-size option While intended for converting URL keys added by addurl --fast to be as if added by addurl --relaxed, it can also be used to remove size from other types of keys. Although that is not likely to be useful for checksummed keys, I suppose it could be used for WORM or other non-checksum keys. Specifying the --remove-size option does not prevent other migrations from taking effect if there's a key upgrade to perform, or if the backend has changed. So --backend=URL needs to be used to prevent migrating an URL key to the default backend. Note that it's not possible to use git-annex migrate to convert from a non-URL key to an URL key, as URL keys cannot be generated, except by addurl. So while this can get the same effect as --relaxed would have when addurl --fast was used, when --fast was not used, it won't work, or if --backend=URL is not used will remove the size but not prevent checksum verification, which is not useful. Due to this complexity, I decided not to mention it in the git-annex addurl man page. Sponsored-by: Jochen Bartl on Patreon
2021-11-12 16:59:30 +00:00
* `--remove-size`
Keys often include the size of their content, which is generally a useful
thing. In fact, this command defaults to adding missing size information
clarification
2024-03-01 21:03:40 +00:00
to keys in most migrations. With this option, the size information is
removed instead.
migrate: New --remove-size option While intended for converting URL keys added by addurl --fast to be as if added by addurl --relaxed, it can also be used to remove size from other types of keys. Although that is not likely to be useful for checksummed keys, I suppose it could be used for WORM or other non-checksum keys. Specifying the --remove-size option does not prevent other migrations from taking effect if there's a key upgrade to perform, or if the backend has changed. So --backend=URL needs to be used to prevent migrating an URL key to the default backend. Note that it's not possible to use git-annex migrate to convert from a non-URL key to an URL key, as URL keys cannot be generated, except by addurl. So while this can get the same effect as --relaxed would have when addurl --fast was used, when --fast was not used, it won't work, or if --backend=URL is not used will remove the size but not prevent checksum verification, which is not useful. Due to this complexity, I decided not to mention it in the git-annex addurl man page. Sponsored-by: Jochen Bartl on Patreon
2021-11-12 16:59:30 +00:00
One use of this option is to convert URL keys that were added
by `git-annex addurl --fast` to ones that would have been added if
that command was run with the `--relaxed` option. Eg:
git-annex migrate --remove-size --backend=URL somefile
migrate: support adding size back to URL keys migrate: Support adding size to URL keys that were added with --relaxed, by running eg: git-annex migrate --backend=URL foo Since url keys cannot be generated, that used to fail. Make it notice that the backend is not changed, and just get the size of the content. Sponsored-by: Brock Spratlen on Patreon
2023-12-08 20:22:14 +00:00
To add back the size to an URL key, use this:
git-annex migrate --backend=URL somefile
migrate, undo: Support --json and --json-error-messages Sponsored-By: the NIH-funded NICEMAN (ReproNim TR&D3) project
2023-05-04 20:34:35 +00:00
* `--json`
Enable JSON output. This is intended to be parsed by programs that use
git-annex. Each line of output is a JSON object.
* `--json-error-messages`
Messages that would normally be output to standard error are included in
the JSON instead.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
# SEE ALSO
[[git-annex]](1)
expand manpages cross-references significantly i found that most man pages only had references to the main git-annex manpage, which i stillfind pretty huge and hard to navigate through. i tried to sift through all the man pages and add cross-references between relevant pages. my general rule of thumb is that links should be both ways unless one of the pages is a more general page that would become ridiculously huge if all backlinks would be added (git-annex-preferred-content comes to mind). i have also make the links one per line as this is how it was done in the metadata pages so far. i did everything but the plumbing, utility and test commands, although some of those are linked from the other commands so cross-links were added there as well.
2015-05-29 16:12:55 +00:00
[[git-annex-upgrade]](1)
improve documentation about backends I noticed that, using just the man pages, there is no real description of what backends are, or what ones are available. Except for some examples. Added a git-annex-backends man page, that is just a stub, but at least describes what they basically are, and tells how to find the supported ons, and links to the backends web page. Sponsored-by: Brett Eisenberg on Patreon
2022-09-26 19:59:10 +00:00
[[git-annex-backend]](1)
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
# AUTHOR
Joey Hess <id@joeyh.name>
Warning: Automatically converted into a man page by mdwn2man. Edit with care.
Reference in a new issue Copy permalink