579d9b60c1
Use separate stages for download and upload. In the common case where it downloads the file from one remote and then uploads to the other, those are by far the most expensive operations, and there's a decent chance the two remotes bottleneck on different resources. Suppose it's being run with -J2 and a bunch of 10 mb files. Two threads will be started both downloading from the src remote. They will probably finish at the same time. Then two threads will be started uploading to the dst remote. They will probably take the same time as well. Before this change, it would alternate back and forth, bottlenecking on src and dst. With this change, as soon as the two threads start uploading to dst, two more threads are able to start, downloading from src. So bandwidth to both remotes is saturated more often. Other commands that use transferStages only send in one direction at a time. So the worker threads for the other direction will sit idle, and there will be no change in their behavior. Sponsored-by: Dartmouth College's DANDI project
146 lines
3.6 KiB
Markdown
146 lines
3.6 KiB
Markdown
# NAME
|
|
|
|
git-annex copy - copy content of files to/from another repository
|
|
|
|
# SYNOPSIS
|
|
|
|
git annex copy `[path ...] [--from=remote|--to=remote]`
|
|
|
|
# DESCRIPTION
|
|
|
|
Copies the content of files from or to another remote.
|
|
|
|
With no parameters, operates on all annexed files in the current directory.
|
|
Paths of files or directories to operate on can be specified.
|
|
|
|
# OPTIONS
|
|
|
|
* `--from=remote`
|
|
|
|
Copy the content of files from the specified
|
|
remote to the local repository.
|
|
|
|
Any files that are not available on the remote will be silently skipped.
|
|
|
|
* `--to=remote`
|
|
|
|
Copy the content of files from the local repository
|
|
to the specified remote.
|
|
|
|
* `--to=here`
|
|
|
|
Copy the content of files from all reachable remotes to the local
|
|
repository.
|
|
|
|
* `--from=remote1 --to=remote2`
|
|
|
|
Copy the content of files that are in remote1 to remote2.
|
|
|
|
This is implemented by first downloading the content from remote1 to the
|
|
local repository (if not already present), then sending it to remote2, and
|
|
then deleting the content from the local repository (if it was not present
|
|
to start with).
|
|
|
|
* `--jobs=N` `-JN`
|
|
|
|
Enables parallel transfers with up to the specified number of jobs
|
|
running at once. For example: `-J10`
|
|
|
|
Setting this to "cpus" will run one job per CPU core.
|
|
|
|
Note that when using --from with --to, twice this many jobs will
|
|
run at once, evenly split between the two remotes.
|
|
|
|
* `--auto`
|
|
|
|
Rather than copying all specified files, only copy those that don't yet have
|
|
the desired number of copies, or that are preferred content of the
|
|
destination repository. See [[git-annex-preferred-content]](1)
|
|
|
|
* `--fast`
|
|
|
|
When copying content to a remote, avoid a round trip to check if the remote
|
|
already has content. This can be faster, but might skip copying content
|
|
to the remote in some cases.
|
|
|
|
* `--all` `-A`
|
|
|
|
Rather than specifying a filename or path to copy, this option can be
|
|
used to copy all available versions of all files.
|
|
|
|
This is the default behavior when running git-annex in a bare repository.
|
|
|
|
* `--branch=ref`
|
|
|
|
Operate on files in the specified branch or treeish.
|
|
|
|
* `--unused`
|
|
|
|
Operate on files found by last run of git-annex unused.
|
|
|
|
* `--failed`
|
|
|
|
Operate on files that have recently failed to be transferred.
|
|
|
|
* `--key=keyname`
|
|
|
|
Use this option to copy a specified key.
|
|
|
|
* matching options
|
|
|
|
The [[git-annex-matching-options]](1)
|
|
can be used to specify what to copy.
|
|
|
|
* `--batch`
|
|
|
|
Enables batch mode, in which lines containing names of files to copy
|
|
are read from stdin.
|
|
|
|
As each specified file is processed, the usual progress output is
|
|
displayed. If a file's content does not need to be copied, or it does not
|
|
match specified matching options, or it is not an annexed file,
|
|
a blank line is output in response instead.
|
|
|
|
Since the usual output while copying a file is verbose and not
|
|
machine-parseable, you may want to use --json in combination with
|
|
--batch.
|
|
|
|
* `--batch-keys`
|
|
|
|
This is like `--batch` but the lines read from stdin are parsed as keys.
|
|
|
|
* `-z`
|
|
|
|
Makes batch input be delimited by nulls instead of the usual newlines.
|
|
|
|
* `--json`
|
|
|
|
Enable JSON output. This is intended to be parsed by programs that use
|
|
git-annex. Each line of output is a JSON object.
|
|
|
|
* `--json-progress`
|
|
|
|
Include progress objects in JSON output.
|
|
|
|
* `--json-error-messages`
|
|
|
|
Messages that would normally be output to standard error are included in
|
|
the json instead.
|
|
|
|
* Also the [[git-annex-common-options]](1) can be used.
|
|
|
|
# SEE ALSO
|
|
|
|
[[git-annex]](1)
|
|
|
|
[[git-annex-get]](1)
|
|
|
|
[[git-annex-move]](1)
|
|
|
|
[[git-annex-drop]](1)
|
|
|
|
# AUTHOR
|
|
|
|
Joey Hess <id@joeyh.name>
|
|
|
|
Warning: Automatically converted into a man page by mdwn2man. Edit with care.
|