blog for the day

doc/bugs/More_sync__39__ing_weirdness_with_the_assistant_branch_on_OSX.mdwn

@@ -10,3 +10,6 @@ hint: See the 'Note about fast-forwards' in 'git push --help' for details.
 (Recording state in git...)
 
 manually running a 'git annex sync' usually fixes it, I guess once the sync command runs periodically this problem will go away, is this even OSX specific? I don't quite get the behaviour that is described in [[design/assistant/blog/day_15__its_aliiive]].
+
+> With my changes today, I've seen it successfully recover from this
+> situation. [[done]] --[[Joey]] 

doc/design/assistant/blog/day_16__more_robust_syncing.mdwn

@@ -0,0 +1,44 @@
+I released a version of git-annex over the weekend that includes the `git
+annex watch` command. There's a minor issue installing it from cabal on
+OSX, which I've fixed in my tree. Nice timing: At least the watch command
+should be shipped in the next Debian release, which freezes at the end of
+the month.
+
+Jimmy found out how kqueue [[blows
+up|bugs/Issue_on_OSX_with_some_system_limits]] when there are too many
+directories to keep all open. I'm not surprised this happens, but it's nice
+to see exactly how. Odd that it happened to him at just 512 directories;
+I'd have guessed more. I have plans to fork watcher programs that each
+watch 512 directories (or whatever the ulimit is), to deal with this. What
+a pitiful interface is kqueue.. I have not thought yet about how the watcher
+programs would communicate back to the main program.
+
+----
+
+Back on the assistant front, I've worked today on making git syncing more
+robust. Now when a push fails, it tries a pull, and a merge, and repushes.
+That ensures that the push is, almost always, a fast-forward. Unless
+something else gets in a push first, anyway!
+
+If a push still fails, there's Yet Another Thread, added today, that will
+wake up after 30 minutes and retry the push. It currently keeps retrying
+every 30 minutes until the push finally gets though. This will deal, to
+some degree, with those situations where a remote is only sometimes
+available.
+
+I need to refine the code a bit, to avoid it keeping an ever-growing queue
+of failed pushes, if a remote is just dead. And to clear old failed pushes
+from the queue when a later push succeeds.
+
+I also need to write a git merge driver that handles conflicts in the tree.
+If two conflicting versions of a file `foo` are saved, this would merge
+them, renaming them to `foo.X` and `foo.Y`. Probably X and Y are the
+git-annex keys for the content of the files; this way all clones will
+resolve the conflict in a way that leads to the same tree. It's also
+possible to get a conflict by one repo deleting a file, and another
+modifying it. In this case, renaming the deleted file to `foo.Y` may
+be the right approach, I am not sure.
+
+I glanced through some Haskell dbus bindings today. I belive there are dbus
+events available to detect when drives are mounted, and on Linux this would
+let git-annex notice and sync to usb drives, etc.

doc/design/assistant/syncing.mdwn

@@ -13,8 +13,9 @@ all the other git clones, at both the git level and the key/value level.
    [The watching can be done with the existing inotify code! This avoids needing
    any special mechanism to notify a remote that it's been synced to.]  
    **done**
-1. Periodically retry pushes that failed. Also, detect if a push failed
-   due to not being up-to-date, pull, and repush.
+1. Periodically retry pushes that failed.  **done** (every half an hour)
+1. Also, detect if a push failed due to not being up-to-date, pull,
+   and repush. **done**
 2. Use a git merge driver that adds both conflicting files,
    so conflicts never break a sync.
 3. Investigate the XMPP approach like dvcs-autosync does, or other ways of
@@ -41,7 +42,7 @@ This probably will need lots of refinements to get working well.
 ## other considerations
 
 It would be nice if, when a USB drive is connected,
-syncing starts automatically.
+syncing starts automatically. Use dbus on Linux?
 
 This assumes the network is connected. It's often not, so the
 [[cloud]] needs to be used to bridge between LANs.