mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

blog for the day

Browse source
This commit is contained in:
Joey Hess 2012-08-21 20:25:20 -04:00
parent 625330ae78
commit 6873ca0c1b
2 changed files with 50 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

36
doc/design/assistant/blog/day_61__network_connection_detection.mdwn Normal file
View file

@ -0,0 +1,36 @@
Today, added a thread that deals with recovering when there's been a loss
of network connectivity. When the network's down, the normal immediate
syncing of changes of course doesn't work. So this thread detects when the
network comes back up, and does a pull+push to network remotes, and
triggers scanning for file content that needs to be transferred.
I used dbus again, to detect events generated by both network-manager and
wicd when they've sucessfully brought an interface up. Or, if they're not
available, it polls every 30 minutes.
When the network comes up, in addition to the git pull+push, it also
currently does a full scan of the repo to find files whose contents
need to be transferred to get fully back into sync.
I think it'll be ok for some git pulls and pushes to happen when
moving to a new network, or resuming a laptop (or every 30 minutes when
resorting to polling). But the transfer scan is currently really too heavy
to be appropriate to do every time in those situations. I have an idea for
avoiding that scan when the remote's git-annex branch has not changed. But
I need to refine it, to handle cases like this:
1. a new remote is added
2. file contents start being transferred to (or from it)
3. the network is taken down
4. all the queued transfers fail
5. the network comes back up
6. the transfer scan needs to know the remote was not all in sync
before #3, and so should do a full scan despite the git-annex branch
not having changed
---
Doubled the ram in my netbook, which I use for all development. Yesod needs
rather a lot of ram to compile and link, and this should make me quite a
lot more productive. I was struggling with OOM killing bits of chromium
during my last week of development.

17
doc/design/assistant/syncing.mdwn
View file

@ -3,9 +3,15 @@ all the other git clones, at both the git level and the key/value level.
## immediate action items
* At startup, and possibly periodically, or when the network connection
changes, or some heuristic suggests that a remote was disconnected from
us for a while, queue remotes for processing by the TransferScanner.
* Sync with all available remotes on startup.
* TransferScanner should avoid unnecessary scanning of remotes.
This is paricilarly important for scans queued by the NetWatcher,
which can be polling, or could be after a momentary blip in network
connectivity. The TransferScanner could check the remote's git-annex
branch; if it is not ahead of the local git-annex branch, then
there's nothing to transfer. **Except** if the tree was not already
up-to-date before the loss of connectivity. So doing this needs
tracking of when the tree is not yet fully up-to-date.
* Ensure that when a remote receives content, and updates its location log,
it syncs that update back out. Prerequisite for:
* After git sync, identify new content that we don't have that is now available
@ -157,3 +163,8 @@ redone to check it.
finishes. **done**
* Test MountWatcher on KDE, and add whatever dbus events KDE emits when
drives are mounted. **done**
* Possibly periodically, or when the network connection
changes, or some heuristic suggests that a remote was disconnected from
us for a while, queue remotes for processing by the TransferScanner.
**done**; both network-manager and wicd connection events are supported,
and it falls back to polling every 30 minutes when neither is available.