mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/git-annex-copy.mdwn

152 lines
3.7 KiB
Text
Raw Normal View History

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# NAME
git-annex copy - copy content of files to/from another repository
# SYNOPSIS
git annex copy `[path ...] [--from=remote|--to=remote]`
# DESCRIPTION
Copies the content of files from or to another remote.
improve documentation
2020-04-13 16:33:35 +00:00
With no parameters, operates on all annexed files in the current directory.
Paths of files or directories to operate on can be specified.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# OPTIONS
* `--from=remote`
move --to=here * move --to=here moves from all reachable remotes to the local repository. The output of move --from remote is changed slightly, when the remote and local both have the content. It used to say: move foo ok Now: move foo (from theremote...) ok That was done so that, when move --to=here is used and the content is locally present and also in several remotes, it's clear which remotes the content gets dropped from. Note that move --to=here will report an error if a non-reachable remote contains the file, even if the local repository also contains the file. I think that's reasonable; the user may be intending to move all other copies of the file from remotes. OTOH, if a copy of the file is believed to be present in some repository that is not a configured remote, move --to=here does not report an error. So a little bit inconsistent, but erroring in this case feels wrong. copy --to=here came along for free, but it's basically the same behavior as git-annex get, and probably with not as good messages in edge cases (especially on failure), so I've not documented it. This commit was sponsored by Anthony DeRobertis on Patreon.
2017-05-31 20:57:27 +00:00
Copy the content of files from the specified
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
remote to the local repository.
fix thru documentation
2015-05-01 18:13:30 +00:00
Any files that are not available on the remote will be silently skipped.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
* `--to=remote`
move --to=here * move --to=here moves from all reachable remotes to the local repository. The output of move --from remote is changed slightly, when the remote and local both have the content. It used to say: move foo ok Now: move foo (from theremote...) ok That was done so that, when move --to=here is used and the content is locally present and also in several remotes, it's clear which remotes the content gets dropped from. Note that move --to=here will report an error if a non-reachable remote contains the file, even if the local repository also contains the file. I think that's reasonable; the user may be intending to move all other copies of the file from remotes. OTOH, if a copy of the file is believed to be present in some repository that is not a configured remote, move --to=here does not report an error. So a little bit inconsistent, but erroring in this case feels wrong. copy --to=here came along for free, but it's basically the same behavior as git-annex get, and probably with not as good messages in edge cases (especially on failure), so I've not documented it. This commit was sponsored by Anthony DeRobertis on Patreon.
2017-05-31 20:57:27 +00:00
Copy the content of files from the local repository
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
to the specified remote.
document copy --to=here This happened for free due to move supporting it but it's certianly useful and worth documenting.
2018-04-09 18:17:31 +00:00
* `--to=here`
Copy the content of files from all reachable remotes to the local
repository.
move/copy: option parsing for --from with --to Allowing --from and --to as an alternative to --from or --to is hard to do with optparse-applicative! The obvious approach of (pfrom <|> pto <|> pfromandto) does not work when pfromandto uses the same option names as pfrom and pto do. It compiles but the generated parser does not work for all desired combinations. Instead, have to parse optionally from and optionally to. When neither is provided, the parser succeeds, but it's a result that can't be handled. So, have to giveup after option parsing. There does not seem to be a way to make an optparse-applicative Parser give up internally either. Also, need seek' because I first tried making fto be a where binding, but that resulted in a hang when git-annex move was run without --from or --to. I think because startConcurrency was not expecting the stages value to contain an exception and so ended up blocking. Sponsored-by: Dartmouth College's DANDI project
2023-01-18 18:42:39 +00:00
* `--from=remote1 --to=remote2`
fix behavior of copy --from --to Sponsored-by: Dartmouth College's DANDI project
2023-01-23 21:53:57 +00:00
Copy the content of files that are in remote1 to remote2.
move/copy: option parsing for --from with --to Allowing --from and --to as an alternative to --from or --to is hard to do with optparse-applicative! The obvious approach of (pfrom <|> pto <|> pfromandto) does not work when pfromandto uses the same option names as pfrom and pto do. It compiles but the generated parser does not work for all desired combinations. Instead, have to parse optionally from and optionally to. When neither is provided, the parser succeeds, but it's a result that can't be handled. So, have to giveup after option parsing. There does not seem to be a way to make an optparse-applicative Parser give up internally either. Also, need seek' because I first tried making fto be a where binding, but that resulted in a hang when git-annex move was run without --from or --to. I think because startConcurrency was not expecting the stages value to contain an exception and so ended up blocking. Sponsored-by: Dartmouth College's DANDI project
2023-01-18 18:42:39 +00:00
fix behavior of copy --from --to Sponsored-by: Dartmouth College's DANDI project
2023-01-23 21:53:57 +00:00
This is implemented by first downloading the content from remote1 to the
local repository (if not already present), then sending it to remote2, and
then deleting the content from the local repository (if it was not present
to start with).
move/copy: option parsing for --from with --to Allowing --from and --to as an alternative to --from or --to is hard to do with optparse-applicative! The obvious approach of (pfrom <|> pto <|> pfromandto) does not work when pfromandto uses the same option names as pfrom and pto do. It compiles but the generated parser does not work for all desired combinations. Instead, have to parse optionally from and optionally to. When neither is provided, the parser succeeds, but it's a result that can't be handled. So, have to giveup after option parsing. There does not seem to be a way to make an optparse-applicative Parser give up internally either. Also, need seek' because I first tried making fto be a where binding, but that resulted in a hang when git-annex move was run without --from or --to. I think because startConcurrency was not expecting the stages value to contain an exception and so ended up blocking. Sponsored-by: Dartmouth College's DANDI project
2023-01-18 18:42:39 +00:00
copy/move --from-anywhere --to remote Implementation was simple because it's equivilant to --from=foo --to remote for each other remote, followed by --to remote when there's a local copy. (Or, in the edge case of --from-anywhere --to=here, it's the same as --to=here.) Note that, when the local repo does not have a copy, fromToPerform gets it from a remote, sends it to the destination, and drops the local copy. Another call to that for a second remote will notice that the dest now has a copy, and simply drop from the second remote, avoiding a second transfer. Also note that, when numcopies doesn't allow dropping it from everywhere, it will drop it from the cheapest remotes first (maybe not ideal) up to more expensive remotes, and finally from the local repo. So the local repo will generally end up holding a copy. Maybe not ideal in all cases either, but it seems no worse to do that than to end up with a copy undropped from a remote. And I'm not entirely happy with the output, eg: copy bigfile (from r3...) ok copy bigfile ok That makes sense if you think of the second line as being the same as what is output by `git-annex copy bigfile --to bar`, but it's less clear in this context. Maybe add "(from here...)"? Also the --json output doesn't have a machine-readable field for the "from" uuid, and maybe it should? Sponsored-by: Dartmouth College's DANDI project
2023-11-30 20:32:32 +00:00
* `--from-anywhere --to=remote`
Copy to the remote files from the local repository as well as from any reachable
remotes.
get, move, copy, mirror: Concurrent downloads and uploads are now supported! This works, and seems fairly robust. Clean get of 20 files at -J3. At -J10, there are some messages about ssh multiplexing, probably due to a race spinning up the ssh connection cacher. But, it manages to get all the files ok regardless. The progress bars are a scrambled mess though, due to bugs in ascii-progress, which I've already filed. Particularly this one: https://github.com/yamadapc/haskell-ascii-progress/issues/8
2015-04-10 21:08:07 +00:00
* `--jobs=N` `-JN`
Enables parallel transfers with up to the specified number of jobs
running at once. For example: `-J10`
annex.jobs=cpus etc Added the ability to run one job per CPU (core), by setting annex.jobs=cpus, or using option --jobs=cpus or -Jcpus. Built with future expansion in mind, including not defaulting matching on Concurrency so more constructors can later be added, and using "cpu" instead of "0".
2019-05-10 17:24:31 +00:00
Setting this to "cpus" will run one job per CPU core.
improve concurrency of move/copy --from --to Use separate stages for download and upload. In the common case where it downloads the file from one remote and then uploads to the other, those are by far the most expensive operations, and there's a decent chance the two remotes bottleneck on different resources. Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads will be started both downloading from the src remote. They will probably finish at the same time. Then two threads will be started uploading to the dst remote. They will probably take the same time as well. Before this change, it would alternate back and forth, bottlenecking on src and dst. With this change, as soon as the two threads start uploading to dst, two more threads are able to start, downloading from src. So bandwidth to both remotes is saturated more often. Other commands that use transferStages only send in one direction at a time. So the worker threads for the other direction will sit idle, and there will be no change in their behavior. Sponsored-by: Dartmouth College's DANDI project
2023-01-24 17:45:01 +00:00
Note that when using --from with --to, twice this many jobs will
run at once, evenly split between the two remotes.
--auto is no longer a global option; only get, drop, and copy accept it. Not a behavior change unless you were passing it to a command that ignored it.
2015-03-25 21:06:14 +00:00
* `--auto`
--auto doc wording try to make more clear that --auto is not --all
2020-04-13 20:12:48 +00:00
Rather than copying all specified files, only copy those that don't yet have
--auto is no longer a global option; only get, drop, and copy accept it. Not a behavior change unless you were passing it to a command that ignored it.
2015-03-25 21:06:14 +00:00
the desired number of copies, or that are preferred content of the
destination repository. See [[git-annex-preferred-content]](1)
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
* `--fast`
document move/copy --fast and --force There was documentation in 66285ca3d1abb21dc1e17d54c7215e96ba46aba2, but it was lost in the man page split. I don't know if this --force is very useful. Considered removing it instead..
2017-06-01 00:01:21 +00:00
When copying content to a remote, avoid a round trip to check if the remote
already has content. This can be faster, but might skip copying content
to the remote in some cases.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
document the -A shorthand for --all
2018-04-04 03:12:04 +00:00
* `--all` `-A`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
Rather than specifying a filename or path to copy, this option can be
used to copy all available versions of all files.
finished splitting out man pages for all commands
2015-03-25 16:09:49 +00:00
This is the default behavior when running git-annex in a bare repository.
--branch, stage 1 Added --branch option to copy, drop, fsck, get, metadata, mirror, move, and whereis commands. This option makes git-annex operate on files that are included in a specified branch (or other treeish). The names of the files from the branch that are being operated on are not displayed yet; only the keys. Displaying the filenames will need changes to every affected command. Also, note that --branch can be specified repeatedly. This is not really documented, but seemed worth supporting, especially since we may later want the ability to operate on all branches matching a refspec. However, when operating on two branches that contain the same key, that key will be operated on twice.
2016-07-20 16:05:22 +00:00
* `--branch=ref`
Operate on files in the specified branch or treeish.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
* `--unused`
Operate on files found by last run of git-annex unused.
get, move, copy, mirror: Added --failed switch which retries failed copies/moves Note that get --from foo --failed will get things that a previous get --from bar tried and failed to get, etc. I considered making --failed only retry transfers from the same remote, but it was easier, and seems more useful, to not have the same remote requirement. Noisy due to some refactoring into Types/
2016-08-03 16:37:12 +00:00
* `--failed`
Operate on files that have recently failed to be transferred.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
* `--key=keyname`
fix typo
2019-09-18 17:11:53 +00:00
Use this option to copy a specified key.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
--batch-keys New --batch-keys option added to these commands: get, drop, move, copy, whereis git-annex-matching-options had to be reworded since some of its options can be used to match on keys, not only files. Sponsored-by: Luke Shumaker on Patreon
2021-08-25 18:20:33 +00:00
* matching options
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
The [[git-annex-matching-options]](1)
--batch-keys New --batch-keys option added to these commands: get, drop, move, copy, whereis git-annex-matching-options had to be reworded since some of its options can be used to match on keys, not only files. Sponsored-by: Luke Shumaker on Patreon
2021-08-25 18:20:33 +00:00
can be used to specify what to copy.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
move, copy: Support --batch.
2017-08-15 16:39:10 +00:00
* `--batch`
Enables batch mode, in which lines containing names of files to copy
are read from stdin.
As each specified file is processed, the usual progress output is
make --batch honor matching options When --batch is used with matching options like --in, --metadata, etc, only operate on the provided files when they match those options. Otherwise, a blank line is output in the batch protocol. Affected commands: find, add, whereis, drop, copy, move, get In the case of find, the documentation for --batch already said it honored the matching options. The docs for the rest didn't, but it makes sense to have them honor them. While this is a behavior change, why specify the matching options with --batch if you didn't want them to apply? Note that the batch output for all of the affected commands could already output a blank line in other cases, so batch users should already be prepared to deal with it. git-annex metadata didn't seem worth making support the matching options, since all it does is output metadata or set metadata, the use cases for using it in combination with the martching options seem small. Made it refuse to run when they're combined, leaving open the possibility for later support if a use case develops. This commit was sponsored by Brett Eisenberg on Patreon.
2018-08-08 16:03:30 +00:00
displayed. If a file's content does not need to be copied, or it does not
match specified matching options, or it is not an annexed file,
a blank line is output in response instead.
move, copy: Support --batch.
2017-08-15 16:39:10 +00:00
Since the usual output while copying a file is verbose and not
machine-parseable, you may want to use --json in combination with
--batch.
--batch-keys New --batch-keys option added to these commands: get, drop, move, copy, whereis git-annex-matching-options had to be reworded since some of its options can be used to match on keys, not only files. Sponsored-by: Luke Shumaker on Patreon
2021-08-25 18:20:33 +00:00
* `--batch-keys`
This is like `--batch` but the lines read from stdin are parsed as keys.
added -z Added -z option to git-annex commands that use --batch, useful for supporting filenames containing newlines. It only controls input to --batch, the output will still be line delimited unless --json or etc is used to get some other output. While git often makes -z affect both input and output, I don't like trying them together, and making it affect output would have been a significant complication, and also git-annex output is generally not intended to be machine parsed, unless using --json or a format option. Commands that take pairs like "file key" still separate them with a space in --batch mode. All such commands take care to support filenames with spaces when parsing that, so there was no need to change it, and it would have needed significant changes to the batch machinery to separate tose with a null. To make fromkey and registerurl support -z, I had to give them a --batch option. The implicit batch mode they enter when not provided with input parameters does not support -z as that would have complicated option parsing. Seemed better to move these toward using the same --batch as everything else, though the implicit batch mode can still be used. This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-09-20 20:09:21 +00:00
* `-z`
--batch-keys New --batch-keys option added to these commands: get, drop, move, copy, whereis git-annex-matching-options had to be reworded since some of its options can be used to match on keys, not only files. Sponsored-by: Luke Shumaker on Patreon
2021-08-25 18:20:33 +00:00
Makes batch input be delimited by nulls instead of the usual newlines.
Revert "--json-exceptions" This reverts commit a325524454fdc45a61f91844efb3065f6e34b318. Turns out this was predicated on an incorrect belief that json output didn't already sometimes lack the "key" field. Since json output already can when `giveup` was used, it seems unncessary to add a whole new option for this.
2023-04-25 21:37:34 +00:00
copy, move, mirror: Support --json and --json-progress.
2016-09-09 20:24:26 +00:00
* `--json`
Enable JSON output. This is intended to be parsed by programs that use
Revert "--json-exceptions" This reverts commit a325524454fdc45a61f91844efb3065f6e34b318. Turns out this was predicated on an incorrect belief that json output didn't already sometimes lack the "key" field. Since json output already can when `giveup` was used, it seems unncessary to add a whole new option for this.
2023-04-25 21:37:34 +00:00
git-annex. Each line of output is a JSON object.
copy, move, mirror: Support --json and --json-progress.
2016-09-09 20:24:26 +00:00
* `--json-progress`
Include progress objects in JSON output.
add --json-error-messages (not yet implemented) Added --json-error-messages option, which includes error messages in the json output, rather than outputting them to stderr. The actual rediretion of errors is not implemented yet, this is only the docs and option plumbing. This commit was supported by the NSF-funded DataLad project.
2018-02-19 18:28:17 +00:00
* `--json-error-messages`
Revert "--json-exceptions" This reverts commit a325524454fdc45a61f91844efb3065f6e34b318. Turns out this was predicated on an incorrect belief that json output didn't already sometimes lack the "key" field. Since json output already can when `giveup` was used, it seems unncessary to add a whole new option for this.
2023-04-25 21:37:34 +00:00
Messages that would normally be output to standard error are included in
the JSON instead.
add --json-error-messages (not yet implemented) Added --json-error-messages option, which includes error messages in the json output, rather than outputting them to stderr. The actual rediretion of errors is not implemented yet, this is only the docs and option plumbing. This commit was supported by the NSF-funded DataLad project.
2018-02-19 18:28:17 +00:00
split out common options to its own page and mention it on each subcommand page Sometimes users would get confused because an option they were looking for was not mentioned on a subcommand's man page, and they had not noticed that the main git-annex man page had a list of common options. This change lets each subcommand mention the common options, similarly to how the matching options are handled. This commit was sponsored by Svenne Krap on Patreon.
2021-05-10 19:00:13 +00:00
* Also the [[git-annex-common-options]](1) can be used.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# SEE ALSO
[[git-annex]](1)
expand manpages cross-references significantly i found that most man pages only had references to the main git-annex manpage, which i stillfind pretty huge and hard to navigate through. i tried to sift through all the man pages and add cross-references between relevant pages. my general rule of thumb is that links should be both ways unless one of the pages is a more general page that would become ridiculously huge if all backlinks would be added (git-annex-preferred-content comes to mind). i have also make the links one per line as this is how it was done in the metadata pages so far. i did everything but the plumbing, utility and test commands, although some of those are linked from the other commands so cross-links were added there as well.
2015-05-29 16:12:55 +00:00
[[git-annex-get]](1)
[[git-annex-move]](1)
[[git-annex-drop]](1)
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# AUTHOR
Joey Hess <id@joeyh.name>
Warning: Automatically converted into a man page by mdwn2man. Edit with care.
Reference in a new issue Copy permalink