git-annex/doc/todo/import_tree.mdwn

When `git annex export treeish` is used to export to a remote, and the
remote allows files to somehow be edited on it, then there ought to be a
way to import the changes back from the remote into the git repository.

The command could be `git annex import treeish` or something like that.

It would ask the special remote to list changed/new files, and deleted
files. Download the changed/new files and inject into the annex. 
Generate a new treeish, with parent the treeish that was exported,
that has the modifications in it.

Updating the working copy is then done by merging the import treeish.
This way, conflicts will be detected and handled as normal by git.

The remote interface needs one new method, to list the changed/new and
deleted files. It will be up to remotes to implement that if they can
support importing.

----

One way for a remote to do it, assuming it has mtimes, is to export
files to the remote with their mtime set to the date of the treeish
being exported (when the treeish is a commit, which has dates, and not
a raw tree). Then the remote can simply enumerate all files,
with their mtimes, and look for files that have mtimes
newer than the last exported treeish's date.

> But: If files on the remote are being changed at around the time
> of the export, they could have older mtimes than the exported treeish's
> date, and so be missed.
> 
> Also, a rename that swaps two files would be missed if mtimes
> are only compared to the treeish's date.

A perhaps better way is for the remote to keep track of the mtime,
size, etc of all exported files, and use that state to find changes.
Where to store that data?

The data could be stored in a file/files on the remote, or perhaps
the remote has a way to store some arbitrary metadata about a file
that could be used. Note that's basically the same as implementing the git
index, on a per-remote basis.

It could be stored in git-annex branch per-remote state. However,
that state is per-key, not per-file. The export database could be
used to convert a ExportLocation to a Key, which could be used
to access the per-remote state. Querying the database for each file
in the export could be a bottleneck without the right interface.

If only one repository will ever access the remote, it could be stored
in eg a local database. But access from only one repository is a 
hard invariant to guarantee.

Would local storage pose a problem when multiple repositories import from
the same remote? In that case, perhaps different trees would be imported,
and merged into master. So the two repositories then have differing
masters, which can be reconciled as usual. It would mean extra downloads
of content from the remote, since each import would download its own copy.
Perhaps this is acceptable?

----

Following the thoughts above, how about this design: The remote
is responsible for collecting a list of files currently in it, along with
some content identifier. That data is sent to git-annex. git-annex stores
the content identifiers locally, and compares old and new lists to determine
when a file on the remote has changed or is new.

This way, each special remote doesn't have to reimplement the equivilant of
the git index, or comparing lists of files, it only needs a way to list
files, and a good content identifier.

A good content identifier needs to:

* Be stable, so when a file has not changed, the content identifier
  remains the same.
* Change when a file is modified.
* Be reasonably unique, but not necessarily fully unique.  
  For example, if the mtime of a file is used as the content identifier, then
  a rename that swaps two files would be noticed, except for in the
  unusual case where they have the same mtime. If a new file
  is added with the same mtime as some other file in the tree though,
  git-annex will see that the filename is new, and so can still import it,
  even though it's seen that content identifier before. Of course, that might
  result in unncessary downloads (eg of a renamed file), so a more unique
  content identifer would be better.

A (size, mtime, inode) tuple is as good a content identifier as git uses in
its index. That or a hash of the content would be ideal. 

Do remotes need to tell git-annex about the properties of content
identifiers they use, or does git-annex assume a minimum bar, and pay the
price with some unncessary transfers of renamed files etc?

Note that git-annex will need a way to get the content identifiers of files
that it stores on the remote when exporting a tree to it. There's a race
here, since a file could be modified on the remote while it's being
exported, and if the remote then uses its mtime in the content identifier,
the modification would never be noticed. (Does git have this same race when
updating the work tree after a merge?)

Some remotes could avoid that race, if they sent back the content
identifier in response to the TRANSFEREXPORT message, and kept the file
quarentined until they had generated the content identifier. Other remotes
probably can't avoid the race. Is it worth changing the TRANSFEREXPORT
interface to include the content identifier in the reply?

----

If multiple repos can access the remote at the same time, then there's a
potential problem when one is exporting a new tree, and the other one is
importing from the remote.

> This can be reduced to the same problem as exports of two
> different trees to the same remote, which is already handled with the
> export log.
> 
> Once a tree has been imported from the remote, it's
> in the same state as exporting that same tree to the remote, so
> update the export log to say that the remote has that treeish exported
> to it. A conflict between two export log entries will be handled as
> usual, with the user being prompted to re-export the tree they want
> to be on the remote. (May need to reword that prompt.)
> --[[Joey]]

----

See also, [[adb_special_remote]]