mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' into assistant

Browse source 
Conflicts:
	doc/design/assistant/syncing.mdwn
This commit is contained in:
Joey Hess 2012-07-06 18:26:40 -06:00
commit d6f65aed16
8 changed files with 140 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

10
doc/design/assistant/blog/day_25__transfer_queueing/comment_1_59fd4f1ffe96c412f613dc86276e7dbd._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E"
nickname="Matt"
subject="Source code"
date="2012-07-06T00:12:15Z"
content="""
Hi Joey,
Is the source code for git-annex assistant available somewhere?
"""]]

7
doc/design/assistant/blog/day_25__transfer_queueing/comment_2_93bf768a67117e873af5732ecf08dc78._comment Normal file
View file

@ -0,0 +1,7 @@
[[!comment format=mdwn
username="http://joeyh.name/"
subject="comment 2"
date="2012-07-06T00:21:43Z"
content="""
It's in the `assistant` branch of git://git-annex.branchable.com/
"""]]

28
doc/design/assistant/blog/day_26__dying_drives.mdwn Normal file
View file

@ -0,0 +1,28 @@
My laptop's SSD died this morning. I had some work from yesterday
committed to the git repo on it, but not pushed as it didn't build.
Luckily I was able to get that off the SSD, which is now a read-only
drive -- even mounting it fails with fsck write errors.
Wish I'd realized the SSD was dying before the day before my trip to
Nicaragua..
Getting back to a useful laptop used most of my time and energy today.
I did manage to fix transfers to not block the rest of the assistant's
threads. Problem was that, without Haskell's threaded runtime, waiting
on something like a rsync command blocks all threads. To fix this,
transfers now are run in separate processes.
Also added code to allow multiple transfers to run at once. Each transfer
takes up a slot, with the number of free slots tracked by a `QSemN`.
This allows the transfer starting thread to block until a slot frees up,
and then run the transfer.
This needs to be extended to be aware of transfers initiated by remotes.
The transfer watcher thread should detect those starting and stopping
and update the `QSemN` accordingly. It would also be nice if transfers
initiated by remotes would be delayed when there are no free slots for them
... but I have not thought of a good way to do that.
There's a bug somewhere in the new transfer code, when two transfers are
queued close together, the second one is lost and doesn't happen.
Would debug this, but I'm spent for the day.

94
doc/design/assistant/syncing.mdwn
View file

@ -3,27 +3,6 @@ all the other git clones, at both the git level and the key/value level.
## action items
* on-disk transfers in progress information files (read/write/enumerate)
**done**
* locking for the files, so redundant transfer races can be detected,
and failed transfers noticed **done**
* transfer info for git-annex-shell **done**
* update files as transfers proceed. See [[progressbars]]
(updating for downloads is easy; for uploads is hard)
* add Transfer queue TChan **done**
* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
**done**
* Poll transfer in progress info files for changes (use inotify again!
wow! hammer, meet nail..), and update the TransferInfo Map **done**
* enqueue Transfers (Uploads) as new files are added to the annex by
Watcher. **done**
* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
Watcher. **done**
* Write basic Transfer handling thread. Multiple such threads need to be
able to be run at once. Each will need its own independant copy of the
Annex state monad. **done**
* Write transfer control thread, which decides when to launch transfers.
**done**
* Check that download transfer triggering code works (when a symlink appears
and the remote does *not* upload to us.
* Investigate why transfers seem to block other git-annex assistant work.
@ -35,31 +14,23 @@ all the other git clones, at both the git level and the key/value level.
* git-annex needs a simple speed control knob, which can be plumbed
through to, at least, rsync. A good job for an hour in an
airport somewhere.
* file transfer processes are not waited for, contain the zombies.
## git syncing
## longer-term TODO
1. Can use `git annex sync`, which already handles bidirectional syncing.
When a change is committed, launch the part of `git annex sync` that pushes
out changes. **done**; changes are pushed out to all remotes in parallel
1. Watch `.git/refs/remotes/` for changes (which would be pushed in from
another node via `git annex sync`), and run the part of `git annex sync`
that merges in received changes, and follow it by the part that pushes out
changes (sending them to any other remotes).
[The watching can be done with the existing inotify code! This avoids needing
any special mechanism to notify a remote that it's been synced to.]
**done**
1. Periodically retry pushes that failed. **done** (every half an hour)
1. Also, detect if a push failed due to not being up-to-date, pull,
and repush. **done**
2. Use a git merge driver that adds both conflicting files,
so conflicts never break a sync. **done**
3. Investigate the XMPP approach like dvcs-autosync does, or other ways of
* Investigate the XMPP approach like dvcs-autosync does, or other ways of
signaling a change out of band.
4. Add a hook, so when there's a change to sync, a program can be run
* Add a hook, so when there's a change to sync, a program can be run
and do its own signaling.
5. --debug will show often unnecessary work being done. Optimise.
6. It would be nice if, when a USB drive is connected,
* --debug will show often unnecessary work being done. Optimise.
* It would be nice if, when a USB drive is connected,
syncing starts automatically. Use dbus on Linux?
* This assumes the network is connected. It's often not, so the
[[cloud]] needs to be used to bridge between LANs.
* Configurablity, including only enabling git syncing but not data transfer;
only uploading new files but not downloading, and only downloading
files in some directories and not others. See for use cases:
[[forum/Wishlist:_options_for_syncing_meta-data_and_data]]
## misc todo
@ -90,7 +61,42 @@ reachable remote. This is worth doing first, since it's the simplest way to
get the basic functionality of the assistant to work. And we'll need this
anyway.
## other considerations
## done
This assumes the network is connected. It's often not, so the
[[cloud]] needs to be used to bridge between LANs.
1. Can use `git annex sync`, which already handles bidirectional syncing.
When a change is committed, launch the part of `git annex sync` that pushes
out changes. **done**; changes are pushed out to all remotes in parallel
1. Watch `.git/refs/remotes/` for changes (which would be pushed in from
another node via `git annex sync`), and run the part of `git annex sync`
that merges in received changes, and follow it by the part that pushes out
changes (sending them to any other remotes).
[The watching can be done with the existing inotify code! This avoids needing
any special mechanism to notify a remote that it's been synced to.]
**done**
1. Periodically retry pushes that failed. **done** (every half an hour)
1. Also, detect if a push failed due to not being up-to-date, pull,
and repush. **done**
2. Use a git merge driver that adds both conflicting files,
so conflicts never break a sync. **done**
* on-disk transfers in progress information files (read/write/enumerate)
**done**
* locking for the files, so redundant transfer races can be detected,
and failed transfers noticed **done**
* transfer info for git-annex-shell **done**
* update files as transfers proceed. See [[progressbars]]
(updating for downloads is easy; for uploads is hard)
* add Transfer queue TChan **done**
* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
**done**
* Poll transfer in progress info files for changes (use inotify again!
wow! hammer, meet nail..), and update the TransferInfo Map **done**
* enqueue Transfers (Uploads) as new files are added to the annex by
Watcher. **done**
* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
Watcher. **done**
* Write basic Transfer handling thread. Multiple such threads need to be
able to be run at once. Each will need its own independant copy of the
Annex state monad. **done**
* Write transfer control thread, which decides when to launch transfers.
**done**