2017-08-29 17:25:48 +00:00
|
|
|
# NAME
|
|
|
|
|
2020-12-17 16:51:49 +00:00
|
|
|
git-annex export - export a tree of files to a special remote
|
2017-08-29 17:25:48 +00:00
|
|
|
|
|
|
|
# SYNOPSIS
|
|
|
|
|
|
|
|
git annex export `treeish --to remote`
|
|
|
|
|
|
|
|
# DESCRIPTION
|
|
|
|
|
|
|
|
Use this command to export a tree of files from a git-annex repository.
|
|
|
|
|
|
|
|
Normally files are stored on a git-annex special remote named by their
|
2017-09-04 20:39:56 +00:00
|
|
|
keys. That is great for reliable data storage, but your filenames are
|
|
|
|
obscured. Exporting replicates the tree to the special remote as-is.
|
2017-08-29 17:25:48 +00:00
|
|
|
|
2024-08-02 18:07:45 +00:00
|
|
|
To use this, you have to configure a special remote with
|
2017-09-06 19:46:35 +00:00
|
|
|
`exporttree=yes` when initially setting it up with
|
|
|
|
[[git-annex-initremote]](1).
|
2017-08-29 17:25:48 +00:00
|
|
|
|
2017-09-16 20:36:18 +00:00
|
|
|
The treeish to export can be the name of a git branch, or a tag, or any
|
|
|
|
other treeish accepted by git, including eg master:subdir to only export a
|
|
|
|
subdirectory from a branch.
|
|
|
|
|
2020-08-10 19:35:26 +00:00
|
|
|
When the remote has a preferred content expression set by
|
|
|
|
[[git-annex-wanted]](1), the treeish is
|
|
|
|
filtered through it, excluding annexed files it does not want from
|
|
|
|
being exported to it. (Note that things in the expression like
|
|
|
|
"include=" match relative to the top of the treeish being exported.)
|
2020-08-10 16:37:25 +00:00
|
|
|
|
|
|
|
Any files in the treeish that are stored on git will also be exported to
|
|
|
|
the special remote.
|
2019-05-20 16:01:37 +00:00
|
|
|
|
2017-08-29 17:25:48 +00:00
|
|
|
Repeated exports are done efficiently, by diffing the old and new tree,
|
2017-09-16 20:36:18 +00:00
|
|
|
and transferring only the changed files, and renaming files as necessary.
|
2017-08-29 17:25:48 +00:00
|
|
|
|
|
|
|
Exports can be interrupted and resumed. However, partially uploaded files
|
2018-02-28 16:09:03 +00:00
|
|
|
will be re-started from the beginning in most cases.
|
2017-08-29 17:25:48 +00:00
|
|
|
|
2017-09-04 20:39:56 +00:00
|
|
|
Once content has been exported to a remote, commands like `git annex get`
|
|
|
|
can download content from there the same as from other remotes. However,
|
|
|
|
since an export is not a key/value store, git-annex has to do more
|
|
|
|
verification of content downloaded from an export. Some types of keys,
|
|
|
|
that are not based on checksums, cannot be downloaded from an export.
|
|
|
|
And, git-annex will never trust an export to retain the content of a key.
|
|
|
|
|
2018-08-30 17:45:28 +00:00
|
|
|
However, some special remotes, notably S3, support keeping track of old
|
|
|
|
versions of files stored in them. If a special remote is set up to do
|
|
|
|
that, it can be used as a key/value store and the limitations in the above
|
2019-01-29 17:51:21 +00:00
|
|
|
paragraph do not apply. Note that dropping content from such a remote is
|
2018-08-30 17:45:28 +00:00
|
|
|
not supported. See individual special remotes' documentation for
|
2018-09-27 19:35:46 +00:00
|
|
|
details of how to enable such versioning.
|
2018-08-30 17:45:28 +00:00
|
|
|
|
git-annex pull and push
Split out two new commands, git-annex pull and git-annex push. Those plus a
git commit are equivilant to git-annex sync.
In a sense, git-annex sync conflates 3 things, and it would have been
better to have push and pull from the beginning and not sync. Although
note that git-annex sync --content is faster than a pull followed by a
push, because it only has to walk the tree once, look at preferred
content once, etc. So there is some value in git-annex sync in speed, as
well as user convenience.
And it would be hard to split out pull and push from sync, as far as the
implementaton goes. The implementation inside sync was easy, just adjust
SyncOptions so it does the right thing.
Note that the new commands default to syncing content, unless
annex.synccontent is explicitly set to false. I'd like sync to also do
that, but that's a hard transition to make. As a start to that
transition, I added a note to git-annex-sync.mdwn that it may start to
do so in a future version of git-annex. But a real transition would
necessarily involve displaying warnings when sync is used without
--content, and time.
Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
|
|
|
Commands like `git-annex push` can also be used to export a branch to a
|
|
|
|
special remote, updating the special remote whenever the branch is changed.
|
|
|
|
To do this, you need to configure "remote.<name>.annex-tracking-branch" to
|
|
|
|
tell it what branch to track. For example:
|
2019-02-23 19:46:03 +00:00
|
|
|
|
|
|
|
git config remote.myremote.annex-tracking-branch master
|
git-annex pull and push
Split out two new commands, git-annex pull and git-annex push. Those plus a
git commit are equivilant to git-annex sync.
In a sense, git-annex sync conflates 3 things, and it would have been
better to have push and pull from the beginning and not sync. Although
note that git-annex sync --content is faster than a pull followed by a
push, because it only has to walk the tree once, look at preferred
content once, etc. So there is some value in git-annex sync in speed, as
well as user convenience.
And it would be hard to split out pull and push from sync, as far as the
implementaton goes. The implementation inside sync was easy, just adjust
SyncOptions so it does the right thing.
Note that the new commands default to syncing content, unless
annex.synccontent is explicitly set to false. I'd like sync to also do
that, but that's a hard transition to make. As a start to that
transition, I added a note to git-annex-sync.mdwn that it may start to
do so in a future version of git-annex. But a real transition would
necessarily involve displaying warnings when sync is used without
--content, and time.
Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
|
|
|
git annex push myremote
|
2019-02-23 19:46:03 +00:00
|
|
|
|
2019-03-05 18:55:06 +00:00
|
|
|
You can combine using `git annex export` to send changes to a special
|
|
|
|
remote with `git annex import` to fetch changes from a special remote.
|
2019-04-23 17:16:25 +00:00
|
|
|
When a file on a special remote has been modified by software other than
|
|
|
|
git-annex, exporting to it will not overwrite the modified file, and the
|
|
|
|
export will not succeed. You can resolve this conflict by using
|
|
|
|
`git annex import`.
|
2019-03-05 18:55:06 +00:00
|
|
|
|
|
|
|
(Some types of special remotes such as S3 with versioning may instead
|
|
|
|
let an export overwrite the modified file; then `git annex import`
|
|
|
|
will create a sequence of commits that includes the modified file,
|
|
|
|
so the overwritten modification is not lost.)
|
|
|
|
|
2017-09-19 17:05:43 +00:00
|
|
|
# OPTIONS
|
|
|
|
|
|
|
|
* `--to=remote`
|
|
|
|
|
|
|
|
Specify the special remote to export to.
|
|
|
|
|
export: Added --from option
This is similar to git-annex copy --from --to, in that it downloads a
local copy, locks it for removal, uploads it, and drops it. Removal of
the temporary local copy is done without verifying numcopies for the
same reason as that command.
I do wonder, looking at this, if there's a race where the local copy
gets used as a copy to allow some other drop in the narrow window after
it is downloaded and before it gets locked for removal. That would need
some other repository to have an out of date location log that says the
repository contains a copy of the key, in order for it to try to use it
as a copy. If there is such a race, git-annex copy/move would also be
vulnerable to it. It would be better to lock it for removal before
starting to download it! That is possible in v10 repositories, which do
use a separate content lock file.
Note that, when the exported tree contains several files that use the
same key, it will be downloaded repeatedly, once per time needed to
upload it. It would be possible to avoid that extra work, but it would
complicate this since the local copy would need to be preserved, locked
for removal, until the end. Also, that would mean that interrupting the
export would leave possibly a lot of temporarily downloaded keys in the
local repository, while currently it can only leave one.
2024-08-08 16:04:39 +00:00
|
|
|
* `--from=remote`
|
|
|
|
|
|
|
|
When the content of a file is not available in the local repository,
|
|
|
|
this option lets it be downloaded from another remote, and sent on to the
|
|
|
|
destination remote. The file will be temporarily stored on local disk,
|
|
|
|
but will never enter the local repository.
|
|
|
|
|
|
|
|
This option can be repeated multiple times.
|
|
|
|
|
|
|
|
It is possible to use --from with the same remote as --to. If the tree
|
|
|
|
contains several files with the same content, and the remote being
|
|
|
|
exported to already contains one copy of the content, this allows making
|
|
|
|
a copy by downloading the content from it.
|
|
|
|
|
2017-09-19 17:05:43 +00:00
|
|
|
* `--tracking`
|
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
This is a deprecated way to set "remote.<name>.annex-tracking-branch".
|
|
|
|
Instead of using this option, you should just set the git configuration
|
|
|
|
yourself.
|
2017-09-19 17:05:43 +00:00
|
|
|
|
2017-09-19 18:26:03 +00:00
|
|
|
* `--fast`
|
|
|
|
|
|
|
|
This sets up an export of a tree, but avoids any expensive file uploads to
|
git-annex pull and push
Split out two new commands, git-annex pull and git-annex push. Those plus a
git commit are equivilant to git-annex sync.
In a sense, git-annex sync conflates 3 things, and it would have been
better to have push and pull from the beginning and not sync. Although
note that git-annex sync --content is faster than a pull followed by a
push, because it only has to walk the tree once, look at preferred
content once, etc. So there is some value in git-annex sync in speed, as
well as user convenience.
And it would be hard to split out pull and push from sync, as far as the
implementaton goes. The implementation inside sync was easy, just adjust
SyncOptions so it does the right thing.
Note that the new commands default to syncing content, unless
annex.synccontent is explicitly set to false. I'd like sync to also do
that, but that's a hard transition to make. As a start to that
transition, I added a note to git-annex-sync.mdwn that it may start to
do so in a future version of git-annex. But a real transition would
necessarily involve displaying warnings when sync is used without
--content, and time.
Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
|
|
|
the remote. You can later run `git annex push` to upload
|
2017-09-19 18:26:03 +00:00
|
|
|
the files to the export.
|
|
|
|
|
2020-05-26 15:43:58 +00:00
|
|
|
* `--jobs=N` `-JN`
|
|
|
|
|
|
|
|
Exports multiple files in parallel. This may be faster.
|
|
|
|
For example: `-J4`
|
|
|
|
|
|
|
|
Setting this to "cpus" will run one job per CPU core.
|
|
|
|
|
2020-05-26 14:31:10 +00:00
|
|
|
* `--json`
|
|
|
|
|
|
|
|
Enable JSON output. This is intended to be parsed by programs that use
|
2023-04-25 21:37:34 +00:00
|
|
|
git-annex. Each line of output is a JSON object.
|
2020-05-26 14:31:10 +00:00
|
|
|
|
|
|
|
* `--json-progress`
|
|
|
|
|
|
|
|
Include progress objects in JSON output.
|
|
|
|
|
|
|
|
* `--json-error-messages`
|
|
|
|
|
2023-04-25 21:37:34 +00:00
|
|
|
Messages that would normally be output to standard error are included in
|
|
|
|
the JSON instead.
|
2020-05-26 14:31:10 +00:00
|
|
|
|
2021-05-10 19:00:13 +00:00
|
|
|
* Also the [[git-annex-common-options]](1) can be used.
|
|
|
|
|
2017-09-16 20:36:18 +00:00
|
|
|
# EXAMPLE
|
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
git annex initremote myremote type=directory directory=/mnt/myremote \
|
2017-11-01 14:08:54 +00:00
|
|
|
exporttree=yes encryption=none
|
2019-02-23 19:46:03 +00:00
|
|
|
git annex export master --to myremote
|
2017-09-16 20:36:18 +00:00
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
After that, /mnt/myremote will contain the same tree of files as the master
|
2017-09-16 20:36:18 +00:00
|
|
|
branch does.
|
|
|
|
|
|
|
|
git mv myfile subdir/myfile
|
|
|
|
git commit -m renamed
|
2019-02-23 19:46:03 +00:00
|
|
|
git annex export master --to myremote
|
2017-09-16 20:36:18 +00:00
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
That updates /mnt/myremote to reflect the renamed file.
|
2017-09-16 20:36:18 +00:00
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
git annex export master:subdir --to myremote
|
2017-09-16 20:36:18 +00:00
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
That updates /mnt/myremote, to contain only the files in the "subdir"
|
2017-09-16 20:36:18 +00:00
|
|
|
directory of the master branch.
|
|
|
|
|
2017-09-06 19:33:40 +00:00
|
|
|
# EXPORT CONFLICTS
|
|
|
|
|
|
|
|
If two different git-annex repositories are both exporting different trees
|
|
|
|
to the same special remote, it's possible for an export conflict to occur.
|
|
|
|
This leaves the special remote with some files from one tree, and some
|
|
|
|
files from the other. Files in the special remote may have entirely the
|
|
|
|
wrong content as well.
|
|
|
|
|
|
|
|
It's not possible for git-annex to detect when making an export will result
|
|
|
|
in an export conflict. The best way to avoid export conflicts is to either
|
|
|
|
only ever export to a special remote from a single repository, or to have a
|
|
|
|
rule about the tree that you export to the special remote. For example, if
|
|
|
|
you always export origin/master after pushing to origin, then an export
|
|
|
|
conflict can't happen.
|
|
|
|
|
|
|
|
An export conflict can only be detected after the two git repositories
|
|
|
|
that produced it get back in sync. Then the next time you run `git annex
|
|
|
|
export`, it will detect the export conflict, and resolve it.
|
|
|
|
|
2017-08-29 17:25:48 +00:00
|
|
|
# SEE ALSO
|
|
|
|
|
|
|
|
[[git-annex]](1)
|
|
|
|
|
2017-09-06 19:46:35 +00:00
|
|
|
[[git-annex-initremote]](1)
|
2017-08-29 17:25:48 +00:00
|
|
|
|
2019-02-23 19:46:03 +00:00
|
|
|
[[git-annex-import]](1)
|
|
|
|
|
git-annex pull and push
Split out two new commands, git-annex pull and git-annex push. Those plus a
git commit are equivilant to git-annex sync.
In a sense, git-annex sync conflates 3 things, and it would have been
better to have push and pull from the beginning and not sync. Although
note that git-annex sync --content is faster than a pull followed by a
push, because it only has to walk the tree once, look at preferred
content once, etc. So there is some value in git-annex sync in speed, as
well as user convenience.
And it would be hard to split out pull and push from sync, as far as the
implementaton goes. The implementation inside sync was easy, just adjust
SyncOptions so it does the right thing.
Note that the new commands default to syncing content, unless
annex.synccontent is explicitly set to false. I'd like sync to also do
that, but that's a hard transition to make. As a start to that
transition, I added a note to git-annex-sync.mdwn that it may start to
do so in a future version of git-annex. But a real transition would
necessarily involve displaying warnings when sync is used without
--content, and time.
Sponsored-by: Kevin Mueller on Patreon
2023-05-16 20:37:30 +00:00
|
|
|
[[git-annex-push]](1)
|
2017-09-19 17:05:43 +00:00
|
|
|
|
2019-05-20 16:01:37 +00:00
|
|
|
[[git-annex-preferred-content]](1)
|
|
|
|
|
2017-12-02 00:26:29 +00:00
|
|
|
# HISTORY
|
|
|
|
|
|
|
|
The `export` command was introduced in git-annex version 6.20170925.
|
|
|
|
|
2017-08-29 17:25:48 +00:00
|
|
|
# AUTHOR
|
|
|
|
|
|
|
|
Joey Hess <id@joeyh.name>
|
|
|
|
|
|
|
|
Warning: Automatically converted into a man page by mdwn2man. Edit with care.
|