blog for the day

doc/design/assistant/blog/day_64__syncing_robustly.mdwn

@@ -0,0 +1,33 @@
+Working toward getting the data syncing to happen robustly,
+so a bunch of improvements.
+
+* Got unmount events to be noticed, so unplugging and replugging
+  a removable drive will resume the syncing to it. There's really no
+  good unmount event available on dbus in kde, so it uses a heuristic
+  there.
+* Avoid requeuing a download from a remote that no longer has a key.
+* Run a full scan on startup, for multiple reasons, including dealing with
+  crashes.
+
+Ran into a strange issue: Occasionally the assistant will run `git-annex
+copy` and it will not transfer the requested file. It seems that
+when the copy command runs `git ls-files`, it does not see the file
+it's supposed to act on in its output.
+
+Eventually I figured out what's going on: When updating the git-annex
+branch, it sets `GIT_INDEX_FILE`, and of course environment settings are
+not thread-safe! So there's a race between threads that access
+the git-annex branch, and the Transferrer thread, or any other thread
+that might expect to look at the normal git index.
+
+Unfortunatly, I don't have a fix for this yet.. Git's only interface for
+using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of
+code to tear apart, to push back the setenv until after forking every git
+command. :(
+
+Before I figured out the root problem, I developed a workaround for the
+symptom I was seeing. I added a `git-annex transferkey`, which is
+optimised to be run by the assistant, and avoids running `git ls-files`, so
+avoids the problem. While I plan to fix this environment variable problem
+properly, `transferkey` turns out to be so much faster than how it was
+using `copy` that I'm going to keep it.

doc/design/assistant/syncing.mdwn

@@ -3,11 +3,16 @@ all the other git clones, at both the git level and the key/value level.
 
 ## immediate action items
 
-* Run transfer scan on startup.
 * The syncing code currently doesn't run for special remotes. While
   transfering the git info about special remotes could be a complication,
   if we assume that's synced between existing git remotes, it should be
   possible for them to do file transfers to/from special remotes.
+* Often several remotes will be queued for full TransferScanner scans,
+  and the scan does the same thing for each .. so it would be better to
+  combine them into one scan in such a case.
+* Sometimes a Download gets queued from a slow remote, and then a fast
+  remote becomes available, and a Download is queued from it. Would be
+  good to sort the transfer queue to run fast Downloads (and Uploads) first.
 * Ensure that when a remote receives content, and updates its location log,
   it syncs that update back out. Prerequisite for:
 * After git sync, identify new content that we don't have that is now available
@@ -200,3 +205,4 @@ redone to check it.
   may get content we were uploading from elsewhere, so check the location
   log when queuing a failed Upload too.) **done**
 * Fix MountWatcher to notice umounts and remounts of drives. **done**
+* Run transfer scan on startup. **done**