add preliminary design

doc/design/assistant.mdwn

@@ -0,0 +1,19 @@
+The git-annex assistant is being
+[crowd funded on Kickstarter](http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own/).
+
+This is my design and plan for developing it.
+Still being fleshed out, still many ideas and use cases to add. --[[Joey]]
+
+## roadmap
+
+### Month 1 "like dropbox": [[!traillink inotify]] [[!traillink syncing]]
+
+### Month 2 "shiny webapp": [[!traillink webapp]] [[!traillink progressbars||
+
+### Month 3 "easy setup": [[!traillink configurators]]
+
+### Month 4: polishing and overflow
+
+### Month 5 & 6 "9k bonus round": [[!traillink Android]]
+
+### In my overfunded nighmares: [[!traillink Windows]]

doc/design/assistant/android.mdwn

@@ -0,0 +1,31 @@
+Porting git-annex to Android will use the Android native SDK.
+
+A hopefully small Java app will be developed, which runs the webapp
+daemon, and a web browser to display it.
+
+### programs to port
+
+These will probably need to be bundled into the Android app, unless already
+available in the App Store.
+
+* ssh (native ssh needed for scp, not a client like ConnectBot)
+* rsync
+* gpg
+* git (not all git commands are needed, but a core plumbing and a few like `git-add` are.)
+
+### FAT sucks
+
+The main media partition will use some awful FAT filesystem format from
+1982 that cannot support git-annex's symlinks. Hopefully it can at least
+handle all of git's filenames. Possible approaches to this:
+
+* Keep only a bare git repo on Android. The app would then need to include
+  a file browser to access the files in there, and adding a file would move
+  it into the repo. Not ideal.
+* Implement [[smudge]] filters to avoid needing symlinks. Difficult.
+* Use a bare git repo but don't keep files in `annex/objects`, instead
+  leave them outside the repo, and add some local mapping to find them.
+  Seems best?
+* Use a `LD_PRELOAD` wrapper to do Something Crazy.
+
+(May want to consider which of these would make a Windows port easier too.)

doc/design/assistant/configurators.mdwn

@@ -0,0 +1,21 @@
+Add to the [[webapp]] some configuration of git-annex.
+
+There are some basic settings that pass through to `git config`, things
+like how much disk space to leave free, how many copies to ensure are kept
+of files, etc.
+
+The meat of the configuration will be in configuration assistants that walk
+through setting up common use cases.
+
+* Clone this repo to a USB drive.
+* Clone this repo to another host:
+  1. Prompt for the hostname (or do avahi local machine discovery). 
+  2. Enable the two hosts to ssh to one-another and run git-annex shell.
+     (A tricky problem, if ssh keys need to be added to do that.)
+  3. Push over a clone of the repository. (Using git-annex-shell?)
+  4. Start [[syncing]].
+* Set up Amazon S3.
+* Set up rsync remote.
+* Set up encryption.
+* I lost my USB drive!
+* etc -- many more possibilities

doc/design/assistant/inotify.mdwn

@@ -0,0 +1,18 @@
+Finish "git annex watch" command, which runs, in the background, watching via
+inotify for changes, and automatically annexing new files, etc.
+
+There is a `watch` branch in git that adds such a command, although currently
+it only handles adding new files, and nothing else. To make this really
+useful, it needs to:
+
+- notice deleted files and stage the deletion
+  (tricky; there's a race with add..)
+- notice renamed files, auto-fix the symlink, and stage the new file location
+- periodically auto-commit staged changes
+- honor .gitignore, not adding files it excludesa
+
+Also to do:
+
+- Support OSes other than Linux; it only uses inotify currently.
+  OSX and FreeBSD use the same mechanism, and there is a Haskell interface
+  for it,

doc/design/assistant/progressbars.mdwn

@@ -0,0 +1,14 @@
+Currently, git-annex takes a very lazy approch to displaying
+progress into. It just lets rsync or whatever display the progress
+for it, in the terminal.
+
+Something better is needed for the [[webapp]]. There needs to be a
+way for the web app to know what the current progress is of all transfers.
+
+To get this info for downloads, git-annex can watch the file as it arrives
+and use its size.
+
+TODO: What about uploads? Will i have to parse rsync's progresss output?
+Ugh.
+
+This is one of those potentially hidden but time consuming problems.

doc/design/assistant/syncing.mdwn

@@ -0,0 +1,31 @@
+Once files are added (or removed or moved), need to send those changes to
+all the other git clones, at both the git level and the key/value level.
+
+## git syncing
+
+1. At regular intervals, just run `git annex sync`, which already handles
+   bidirectional syncing.
+2. Investigate the XMPP approach like dvcs-autosync does, or other ways of
+   signaling a change out of band.
+3. Add a hook, so when there's a change to sync, a program can be run.
+
+## data syncing
+
+There are two parts to data syncing. First, map the network and second,
+decide what to sync when.
+
+Mapping the network can reuse code in `git annex map`. Once the map is
+built, we want to find paths through the network that reach all nodes
+eventually, with the least cost. This is a minimum spanning tree problem,
+except with a directed graph, so really a Arborescence problem.
+
+With the map, we can determine which nodes to push new content to. Then we
+need to control those data transfers, sending to the cheapest nodes first,
+and with appropriate rate limiting and control facilities.
+
+This probably will need lots of refinements to get working well.
+
+## other considerations
+
+This assumes the network is connected. It's often not, so the
+cloud needs to be used to bridge between LANs.

doc/design/assistant/webapp.mdwn

@@ -0,0 +1,27 @@
+The webapp is a web server that displays a shiny interface.
+
+## security
+
+* Listen only to localhost.
+* Instruct the user's web browser to open an url that contains a secret
+  token. This guards against other users on the same system.
+* I would like to avoid passwords or other authentication methods,
+  it's your local system.
+
+## interface
+
+* list of files uploading and downloading
+* progress bars for each file
+* drag and drop to reorder
+* cancel and pause
+
+## implementation
+
+Hope to use Yesod.
+
+TODO: Ensure that Yesod will work on arm. Necessary for later Android port.
+Will its template haskell cause a problem? Does new GHC support TH on ARM?
+Will it use too much memory or be too slow?
+
+Hopefully Yesod comes with some good UI widgets. Otherwise, need to use
+Jquery or similar.

doc/design/assistant/windows.mdwn

@@ -0,0 +1,17 @@
+See [[todo/windows_support]]..
+
+## symlinks
+
+Apparently new versions of Windows have something very like symlinks.
+(Or really, 3 or so things not entirely unlike symlinks and all different.)
+Stackoverflow has some details.
+
+Make git use them, as it (apparently) does not yet.
+
+(What **does** git do on Windows when it clones a repo with symlinks?)
+
+## POSIX
+
+Lots of ifdefs and pain to deal with POSIX calls in the code base.
+
+Or I could try to use Cywin.