blog for the day
This commit is contained in:
parent
47875e9b94
commit
dda61b1b80
2 changed files with 40 additions and 1 deletions
33
doc/design/assistant/blog/day_64__syncing_robustly.mdwn
Normal file
33
doc/design/assistant/blog/day_64__syncing_robustly.mdwn
Normal file
|
@ -0,0 +1,33 @@
|
|||
Working toward getting the data syncing to happen robustly,
|
||||
so a bunch of improvements.
|
||||
|
||||
* Got unmount events to be noticed, so unplugging and replugging
|
||||
a removable drive will resume the syncing to it. There's really no
|
||||
good unmount event available on dbus in kde, so it uses a heuristic
|
||||
there.
|
||||
* Avoid requeuing a download from a remote that no longer has a key.
|
||||
* Run a full scan on startup, for multiple reasons, including dealing with
|
||||
crashes.
|
||||
|
||||
Ran into a strange issue: Occasionally the assistant will run `git-annex
|
||||
copy` and it will not transfer the requested file. It seems that
|
||||
when the copy command runs `git ls-files`, it does not see the file
|
||||
it's supposed to act on in its output.
|
||||
|
||||
Eventually I figured out what's going on: When updating the git-annex
|
||||
branch, it sets `GIT_INDEX_FILE`, and of course environment settings are
|
||||
not thread-safe! So there's a race between threads that access
|
||||
the git-annex branch, and the Transferrer thread, or any other thread
|
||||
that might expect to look at the normal git index.
|
||||
|
||||
Unfortunatly, I don't have a fix for this yet.. Git's only interface for
|
||||
using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of
|
||||
code to tear apart, to push back the setenv until after forking every git
|
||||
command. :(
|
||||
|
||||
Before I figured out the root problem, I developed a workaround for the
|
||||
symptom I was seeing. I added a `git-annex transferkey`, which is
|
||||
optimised to be run by the assistant, and avoids running `git ls-files`, so
|
||||
avoids the problem. While I plan to fix this environment variable problem
|
||||
properly, `transferkey` turns out to be so much faster than how it was
|
||||
using `copy` that I'm going to keep it.
|
|
@ -3,11 +3,16 @@ all the other git clones, at both the git level and the key/value level.
|
|||
|
||||
## immediate action items
|
||||
|
||||
* Run transfer scan on startup.
|
||||
* The syncing code currently doesn't run for special remotes. While
|
||||
transfering the git info about special remotes could be a complication,
|
||||
if we assume that's synced between existing git remotes, it should be
|
||||
possible for them to do file transfers to/from special remotes.
|
||||
* Often several remotes will be queued for full TransferScanner scans,
|
||||
and the scan does the same thing for each .. so it would be better to
|
||||
combine them into one scan in such a case.
|
||||
* Sometimes a Download gets queued from a slow remote, and then a fast
|
||||
remote becomes available, and a Download is queued from it. Would be
|
||||
good to sort the transfer queue to run fast Downloads (and Uploads) first.
|
||||
* Ensure that when a remote receives content, and updates its location log,
|
||||
it syncs that update back out. Prerequisite for:
|
||||
* After git sync, identify new content that we don't have that is now available
|
||||
|
@ -200,3 +205,4 @@ redone to check it.
|
|||
may get content we were uploading from elsewhere, so check the location
|
||||
log when queuing a failed Upload too.) **done**
|
||||
* Fix MountWatcher to notice umounts and remounts of drives. **done**
|
||||
* Run transfer scan on startup. **done**
|
||||
|
|
Loading…
Add table
Reference in a new issue