diff --git a/doc/design/compute_special_remote_interface.mdwn b/doc/design/compute_special_remote_interface.mdwn new file mode 100644 index 0000000000..63bc253493 --- /dev/null +++ b/doc/design/compute_special_remote_interface.mdwn @@ -0,0 +1,80 @@ +**draft** + +The [[special_remotes/compute]] special remote uses this interface to run +compute programs. + +When an compute special remote is initremoted, a program is specified: + + git-annex initremote myremote type=compute program=foo + +That causes `git-annex-compute-foo` to be run to get files from that +compute special remote. + +The environment variable `ANNEX_COMPUTE_KEY` is the key that the program +is requested to compute. + +The program is run in a temporary directory, which will be cleaned up after it +exits. When it generates the content of a key, it should write it to a file +with the same name as the key, in that directory. Then it should +output the key in a line to stdout. + +While usually this will be the requested key, the program can output any +number of other keys as well, all of which will be stored in the git-annex +repository when getting files from the compute special remote. When a +computation generates several files, this allows running it a single time +to get them all. + +The program is passed environment variables to provide inputs to the +computation. These are all prefixed with `"ANNEX_COMPUTE_"`. + +The names are taken from the `git-annex addcomputed` command that was used to +add a computed file to the repository. + +For example, this command: + + git-annex addcomputed file.gen --to foo \ + --input raw=file.raw --value passes=10 + +Will result in this environment: + + ANNEX_COMPUTE_KEY=SHA256--... + ANNEX_COMPUTE_raw=file.in + ANNEX_COMPUTE_INPUT_raw=/path/.git/annex/objects/.. + ANNEX_COMPUTE_passes=10 + +For security, the program should avoid exposing values from `ANNEX_COMPUTE_*` +variables to the shell unprotected, or otherwise executing them. + +The program will also inherit other environment variables +that were set when git-annex was run, like PATH. + +Anything that the program outputs to stderr will be displayed to the user. +This stderr should be used for error messages, and possibly computation +output, but not for progress displays, since git-annex has its own progress +displays. + +If possible, the program should write the content of the key it is +generating directly to the file, rather than writing to somewhere else and +renaming it at the end. If git-annex sees that the file corresponding to +the key it requested be computed is growing, it will use the file size when +displaying progress to the user. + +Alternatively, if the program outputs a number on a line to stdout, this is +taken to be the number of bytes of the requested key that have been computed +so far. Or, the program can output a percentage eg "50%" on a line to stdout +to indicate what percent of the computation has been performed so far. + +If the program exits nonzero, nothing it computed will be stored in the +git-annex repository. + +An example `git-annex-compute-foo` shell script follows: + + #!/bin/sh + set -e + if [ -z "$ANNEX_COMPUTE_passes" || -z "$ANNEX_COMPUTE_INPUT_raw" ]; then + echo "Missing expected inputs" >&2 + exit 1 + fi + frobnicate --passes="$ANNEX_COMPUTE_passes" \ + <"$ANNEX_COMPUTE_INPUT_raw" >"$ANNEX_COMPUTE_KEY" + echo "$ANNEX_COMPUTE_KEY" diff --git a/doc/todo/compute_special_remote/comment_17_8bc2bd7f93ac98d55b69ec1e4e6fa487._comment b/doc/todo/compute_special_remote/comment_17_8bc2bd7f93ac98d55b69ec1e4e6fa487._comment new file mode 100644 index 0000000000..5735dfa1c6 --- /dev/null +++ b/doc/todo/compute_special_remote/comment_17_8bc2bd7f93ac98d55b69ec1e4e6fa487._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 17""" + date="2025-02-13T20:10:52Z" + content=""" +I've written up a draft interface for programs used by a compute special +remote: [[design/compute_special_remote_interface]] +"""]]