From 45a01db6add3399ff6ca93f2e7c7d83dbf59992d Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sat, 26 May 2012 21:11:19 -0400 Subject: [PATCH] add preliminary design --- doc/design/assistant.mdwn | 19 +++++++++++++++ doc/design/assistant/android.mdwn | 31 +++++++++++++++++++++++++ doc/design/assistant/configurators.mdwn | 21 +++++++++++++++++ doc/design/assistant/inotify.mdwn | 18 ++++++++++++++ doc/design/assistant/progressbars.mdwn | 14 +++++++++++ doc/design/assistant/syncing.mdwn | 31 +++++++++++++++++++++++++ doc/design/assistant/webapp.mdwn | 27 +++++++++++++++++++++ doc/design/assistant/windows.mdwn | 17 ++++++++++++++ 8 files changed, 178 insertions(+) create mode 100644 doc/design/assistant.mdwn create mode 100644 doc/design/assistant/android.mdwn create mode 100644 doc/design/assistant/configurators.mdwn create mode 100644 doc/design/assistant/inotify.mdwn create mode 100644 doc/design/assistant/progressbars.mdwn create mode 100644 doc/design/assistant/syncing.mdwn create mode 100644 doc/design/assistant/webapp.mdwn create mode 100644 doc/design/assistant/windows.mdwn diff --git a/doc/design/assistant.mdwn b/doc/design/assistant.mdwn new file mode 100644 index 0000000000..a9444c3673 --- /dev/null +++ b/doc/design/assistant.mdwn @@ -0,0 +1,19 @@ +The git-annex assistant is being +[crowd funded on Kickstarter](http://www.kickstarter.com/projects/joeyh/git-annex-assistant-like-dropbox-but-with-your-own/). + +This is my design and plan for developing it. +Still being fleshed out, still many ideas and use cases to add. --[[Joey]] + +## roadmap + +### Month 1 "like dropbox": [[!traillink inotify]] [[!traillink syncing]] + +### Month 2 "shiny webapp": [[!traillink webapp]] [[!traillink progressbars|| + +### Month 3 "easy setup": [[!traillink configurators]] + +### Month 4: polishing and overflow + +### Month 5 & 6 "9k bonus round": [[!traillink Android]] + +### In my overfunded nighmares: [[!traillink Windows]] diff --git a/doc/design/assistant/android.mdwn b/doc/design/assistant/android.mdwn new file mode 100644 index 0000000000..24791501da --- /dev/null +++ b/doc/design/assistant/android.mdwn @@ -0,0 +1,31 @@ +Porting git-annex to Android will use the Android native SDK. + +A hopefully small Java app will be developed, which runs the webapp +daemon, and a web browser to display it. + +### programs to port + +These will probably need to be bundled into the Android app, unless already +available in the App Store. + +* ssh (native ssh needed for scp, not a client like ConnectBot) +* rsync +* gpg +* git (not all git commands are needed, but a core plumbing and a few like `git-add` are.) + +### FAT sucks + +The main media partition will use some awful FAT filesystem format from +1982 that cannot support git-annex's symlinks. Hopefully it can at least +handle all of git's filenames. Possible approaches to this: + +* Keep only a bare git repo on Android. The app would then need to include + a file browser to access the files in there, and adding a file would move + it into the repo. Not ideal. +* Implement [[smudge]] filters to avoid needing symlinks. Difficult. +* Use a bare git repo but don't keep files in `annex/objects`, instead + leave them outside the repo, and add some local mapping to find them. + Seems best? +* Use a `LD_PRELOAD` wrapper to do Something Crazy. + +(May want to consider which of these would make a Windows port easier too.) diff --git a/doc/design/assistant/configurators.mdwn b/doc/design/assistant/configurators.mdwn new file mode 100644 index 0000000000..e03723e946 --- /dev/null +++ b/doc/design/assistant/configurators.mdwn @@ -0,0 +1,21 @@ +Add to the [[webapp]] some configuration of git-annex. + +There are some basic settings that pass through to `git config`, things +like how much disk space to leave free, how many copies to ensure are kept +of files, etc. + +The meat of the configuration will be in configuration assistants that walk +through setting up common use cases. + +* Clone this repo to a USB drive. +* Clone this repo to another host: + 1. Prompt for the hostname (or do avahi local machine discovery). + 2. Enable the two hosts to ssh to one-another and run git-annex shell. + (A tricky problem, if ssh keys need to be added to do that.) + 3. Push over a clone of the repository. (Using git-annex-shell?) + 4. Start [[syncing]]. +* Set up Amazon S3. +* Set up rsync remote. +* Set up encryption. +* I lost my USB drive! +* etc -- many more possibilities diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn new file mode 100644 index 0000000000..be38ef13cd --- /dev/null +++ b/doc/design/assistant/inotify.mdwn @@ -0,0 +1,18 @@ +Finish "git annex watch" command, which runs, in the background, watching via +inotify for changes, and automatically annexing new files, etc. + +There is a `watch` branch in git that adds such a command, although currently +it only handles adding new files, and nothing else. To make this really +useful, it needs to: + +- notice deleted files and stage the deletion + (tricky; there's a race with add..) +- notice renamed files, auto-fix the symlink, and stage the new file location +- periodically auto-commit staged changes +- honor .gitignore, not adding files it excludesa + +Also to do: + +- Support OSes other than Linux; it only uses inotify currently. + OSX and FreeBSD use the same mechanism, and there is a Haskell interface + for it, diff --git a/doc/design/assistant/progressbars.mdwn b/doc/design/assistant/progressbars.mdwn new file mode 100644 index 0000000000..f76b42d73a --- /dev/null +++ b/doc/design/assistant/progressbars.mdwn @@ -0,0 +1,14 @@ +Currently, git-annex takes a very lazy approch to displaying +progress into. It just lets rsync or whatever display the progress +for it, in the terminal. + +Something better is needed for the [[webapp]]. There needs to be a +way for the web app to know what the current progress is of all transfers. + +To get this info for downloads, git-annex can watch the file as it arrives +and use its size. + +TODO: What about uploads? Will i have to parse rsync's progresss output? +Ugh. + +This is one of those potentially hidden but time consuming problems. diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn new file mode 100644 index 0000000000..0a081c1010 --- /dev/null +++ b/doc/design/assistant/syncing.mdwn @@ -0,0 +1,31 @@ +Once files are added (or removed or moved), need to send those changes to +all the other git clones, at both the git level and the key/value level. + +## git syncing + +1. At regular intervals, just run `git annex sync`, which already handles + bidirectional syncing. +2. Investigate the XMPP approach like dvcs-autosync does, or other ways of + signaling a change out of band. +3. Add a hook, so when there's a change to sync, a program can be run. + +## data syncing + +There are two parts to data syncing. First, map the network and second, +decide what to sync when. + +Mapping the network can reuse code in `git annex map`. Once the map is +built, we want to find paths through the network that reach all nodes +eventually, with the least cost. This is a minimum spanning tree problem, +except with a directed graph, so really a Arborescence problem. + +With the map, we can determine which nodes to push new content to. Then we +need to control those data transfers, sending to the cheapest nodes first, +and with appropriate rate limiting and control facilities. + +This probably will need lots of refinements to get working well. + +## other considerations + +This assumes the network is connected. It's often not, so the +cloud needs to be used to bridge between LANs. diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn new file mode 100644 index 0000000000..c0f6b893e1 --- /dev/null +++ b/doc/design/assistant/webapp.mdwn @@ -0,0 +1,27 @@ +The webapp is a web server that displays a shiny interface. + +## security + +* Listen only to localhost. +* Instruct the user's web browser to open an url that contains a secret + token. This guards against other users on the same system. +* I would like to avoid passwords or other authentication methods, + it's your local system. + +## interface + +* list of files uploading and downloading +* progress bars for each file +* drag and drop to reorder +* cancel and pause + +## implementation + +Hope to use Yesod. + +TODO: Ensure that Yesod will work on arm. Necessary for later Android port. +Will its template haskell cause a problem? Does new GHC support TH on ARM? +Will it use too much memory or be too slow? + +Hopefully Yesod comes with some good UI widgets. Otherwise, need to use +Jquery or similar. diff --git a/doc/design/assistant/windows.mdwn b/doc/design/assistant/windows.mdwn new file mode 100644 index 0000000000..0b176934b9 --- /dev/null +++ b/doc/design/assistant/windows.mdwn @@ -0,0 +1,17 @@ +See [[todo/windows_support]].. + +## symlinks + +Apparently new versions of Windows have something very like symlinks. +(Or really, 3 or so things not entirely unlike symlinks and all different.) +Stackoverflow has some details. + +Make git use them, as it (apparently) does not yet. + +(What **does** git do on Windows when it clones a repo with symlinks?) + +## POSIX + +Lots of ifdefs and pain to deal with POSIX calls in the code base. + +Or I could try to use Cywin.