mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

blog for the day

Browse source
This commit is contained in:
Joey Hess 2012-08-24 17:40:38 -04:00
commit dda61b1b80
2 changed files with 40 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

33
doc/design/assistant/blog/day_64__syncing_robustly.mdwn Normal file
View file

@ -0,0 +1,33 @@
Working toward getting the data syncing to happen robustly,
so a bunch of improvements.
* Got unmount events to be noticed, so unplugging and replugging
a removable drive will resume the syncing to it. There's really no
good unmount event available on dbus in kde, so it uses a heuristic
there.
* Avoid requeuing a download from a remote that no longer has a key.
* Run a full scan on startup, for multiple reasons, including dealing with
crashes.
Ran into a strange issue: Occasionally the assistant will run `git-annex
copy` and it will not transfer the requested file. It seems that
when the copy command runs `git ls-files`, it does not see the file
it's supposed to act on in its output.
Eventually I figured out what's going on: When updating the git-annex
branch, it sets `GIT_INDEX_FILE`, and of course environment settings are
not thread-safe! So there's a race between threads that access
the git-annex branch, and the Transferrer thread, or any other thread
that might expect to look at the normal git index.
Unfortunatly, I don't have a fix for this yet.. Git's only interface for
using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of
code to tear apart, to push back the setenv until after forking every git
command. :(
Before I figured out the root problem, I developed a workaround for the
symptom I was seeing. I added a `git-annex transferkey`, which is
optimised to be run by the assistant, and avoids running `git ls-files`, so
avoids the problem. While I plan to fix this environment variable problem
properly, `transferkey` turns out to be so much faster than how it was
using `copy` that I'm going to keep it.

8
doc/design/assistant/syncing.mdwn
View file

@ -3,11 +3,16 @@ all the other git clones, at both the git level and the key/value level.
## immediate action items
* Run transfer scan on startup.
* The syncing code currently doesn't run for special remotes. While
transfering the git info about special remotes could be a complication,
if we assume that's synced between existing git remotes, it should be
possible for them to do file transfers to/from special remotes.
* Often several remotes will be queued for full TransferScanner scans,
and the scan does the same thing for each .. so it would be better to
combine them into one scan in such a case.
* Sometimes a Download gets queued from a slow remote, and then a fast
remote becomes available, and a Download is queued from it. Would be
good to sort the transfer queue to run fast Downloads (and Uploads) first.
* Ensure that when a remote receives content, and updates its location log,
it syncs that update back out. Prerequisite for:
* After git sync, identify new content that we don't have that is now available
@ -200,3 +205,4 @@ redone to check it.
may get content we were uploading from elsewhere, so check the location
log when queuing a failed Upload too.) **done**
* Fix MountWatcher to notice umounts and remounts of drives. **done**
* Run transfer scan on startup. **done**