git-annex/doc/tips/computing_annexed_files.mdwn

Do you ever check in original versions of files to `git-annex`, but then
convert them in some way? Maybe you check in original photos from a camera,
but then change them to a more useful file format, or smaller resolution. 
Or you clip a video file. Or you crunch some data to a result.

If you check the computed file into `git-annex` too, and store it on
your remotes along with the original, that's a waste of disk space.
But it is so convenient to be able to `git-annex get` the computed file.

The [[compute special remote|special_remotes/compute]] is the solution to
this. It "stores" the computed file by remembering how to compute it from
input files. When you `git-annex get` the computed file from it, it re-runs
the computation on the original input file to produced the computed file.

[[!toc ]]

## using the compute special remote

There are many compute programs that each handle some type of computation,
and it's pretty easy to write your own compute program too. In this tip,
we'll use [[special_remotes/compute/git-annex-compute-imageconvert]], 
which uses imagemagick to convert between image formats.

To follow along, install that program in PATH (and remember to make it
executable!) and make sure you have
[imagemagick](https://www.imagemagick.org/) installed.

First, initialize a compute remote:

    # git-annex initremote imageconvert type=compute program=git-annex-compute-imageconvert

Now suppose you have a file `foo.jpeg`, and you want to add a computed
`foo.gif` to the git-annex repository.

    # git-annex addcomputed --to=imageconvert foo.jpeg foo.gif

(The syntax of the `git-annex addcomputed` command will vary depending on the
program that a compute remote uses. Some may have multiple input files, or
multiple ouput files, or other options to control the computation. See
the documentation of each compute program for details.)

Now you have `foo.gif` and can use it as usual, including copying it to
other remotes. But it's already "stored" in the imageconvert remote,
as a computation. So to free up space, you can drop it:

    # git-annex drop foo.gif
    drop foo.gif ok

By the way, you can also add a computed file to the repository 
without bothering to compute it yet! Just use `--fast`:

    # git-annex addcomputed --fast --to=imageconvert bar.jpeg bar.gif

Now suppose you're in another clone of this same repository, and you want
these gifs.

    # git-annex get foo.gif
    get foo.gif (not available)
	  Maybe enable some of these special remotes (git annex enableremote ...):
	  	8332f7ad-d54e-435e-803b-138c1cfa7b71 -- imageconvert
	failed

With [[special_remotes/compute/git-annex-compute-imageconvert]] and
imagemagic installed, all you need to do is enable the special remote to
get the computed files from it:

	# git-annex enableremote imageconvert
	# git-annex get foo.gif
	get foo.gif (from imageconvert...)
    (getting input foo.jpeg from origin...)
	ok

Notice that, when the input file is not present in the repository, getting
a file from a compute remote will first get the input file.

That's the basics of using the compute special remote.

## recomputation

What happens if the input file `foo.gif` is changed to a new version?
Will getting `foo.jpeg` from the compute remote base it on the new version
too? No. `foo.gif` is stuck on the original version of the input file that
was used to compute it.

But, it's easy to recompute the file with a new version of the input file.
Just `git-annex add` the new version of the input file, and then:

    # git-annex recompute foo.gif
    recompute foo.gif (foo.jpeg changed)
    ok

You can use commands like `git diff` and `git status` to see the
change that made to `foo.gif`.

    # git status --short foo.gif
     M foo.gif

Now both the new and old versions of `foo.gif` are stored in the
imageconvert remote, and it can compute either as needed.

## reproducibility

You might be wondering, what happens if a computed file, such as `foo.gif`
isn't exactly the same identical file each time it's computed? For example,
what if there's a timestamp in there.

The answer is that, by default, files computed by a compute special remote
are not required, or guaranteed to be bit-for-bit reproducible. One gif
converted from a jpeg is much like any other converted from the same jpeg.

So git-annex usually treats all files computed in the same way from the
same input as interchangeable. (Unless the compute program indicates
that it produces reproducible files.)

Sometimes though, it's important that a file be bit-for-bit reproducible. And
you can ask git-annex to enforce this for computed files.
There is a `--reproducible` option for this, which you can pass to
`git-annex addcomputed` or to `git-annex recompute`.

Let's switch the computed `foo.gif` to a reproducible file:

    # git-annex recompute --original --reproducible foo.gif
    recompute foo.gif
    ok

You can `git commit foo.gif` to store this change.

But first, let's check if that computation actually *is* reproducible.
This is easy, just drop it and get it from the compute remote again:

    # git-annex drop foo.gif
    drop foo.gif ok
    # git-annex get foo.gif --from imageconvert
    get foo.gif (from imageconvert...)
    ok

If it turned out that the computation was not reproducible, getting the
file would fail, like this:

    # git-annex get foo.gif --from imageconvert
    get foo.gif (from imageconvert...)
    Verification of content failed

This is because a reproducible file uses a regular [[backend]], which
by default uses a hash to verify the content of the file.

If it does turn out that a file that was expected to be reproducible isn't,
you can always convert it to an unreproducible file:

    # git-annex recompute --original --unreproducible foo.gif
    recompute foo.gif
    ok

## writing your own compute programs

There is a whole little protocol that compute programs use to
communicate with git-annex. It's all documented at
[[design/compute_special_remote_interface]].

But it's really easy to write simple ones, and you don't need to
dive into all the details to do it. Let's walk through the code
to [[special_remotes/compute/git-annex-compute-imageconvert]],
which at 14 lines, is about as simple as one can be.

    #!/bin/sh

It's a shell script.

    set -e

If it fails to read input from standard input, or if a command fails, it
will exit nonzero.

	if [ -z "$1" ] || [ -z "$2" ]; then
		echo "Specify the input image file, followed by the output image file." >&2
		echo "Example: foo.jpg foo.gif" >&2
		exit 1
	fi

It expects to be passed two parameters, which were "foo.jpeg" and "foo.gif" in
the examples above. And it outputs some usage to stderr otherwise. That is
displayed if the user runs `git-annex addcomputed` without the necessary
filenames.

	echo "INPUT $1"
	read input

It tells git-annex that the first filename is the input file. And git-annex
replies by telling it *where* the content of the input file is. This is the
path to a git-annex object file.

	echo "OUTPUT $2"
	read output

It tells git-annex that the second filename is the output file. And git-annex
replies by telling it the path it should write the output file to.

	if [ -n "$input" ]; then

When `git-annex addcomputed --fast` is used, the program shouldn't actually
read the input file or compute the output file. git-annex indicates this by
not giving it a path to the input file. That's checked here.

		convert "$input" "$output" >&2

This uses `convert` from imagemagick, and just converts the input file to
the output file.

Notice that stdout from `convert` is redirected to stderr. This is done
because the compute program is speaking this protocol with git-annex over
stdin and stdout, and we don't want random program output to mess that up.

	fi

Closing the `if` above.

And that's all!

Now you know almost enough to write your own compute program. Editing this one
will be a good start.

**But first, a word about security.**

A user who enables a compute special remote and runs `git pull` followed by
`git-annex get` is running the compute program with inputs under the control
of anyone who has commit access to the repository.

So, it's important that your compute program be secure. Please see
the section on security in [[design/compute_special_remote_interface]]
for security considerations.

If you write a nice secure compute program, you can add it to the list
in [[special_remotes/compute]] so other people can use it.