Merge branch 'master' into assistant
Conflicts: doc/design/assistant/syncing.mdwn
This commit is contained in:
commit
d6f65aed16
8 changed files with 140 additions and 46 deletions
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E"
|
||||
nickname="Matt"
|
||||
subject="Source code"
|
||||
date="2012-07-06T00:12:15Z"
|
||||
content="""
|
||||
Hi Joey,
|
||||
|
||||
Is the source code for git-annex assistant available somewhere?
|
||||
"""]]
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="http://joeyh.name/"
|
||||
subject="comment 2"
|
||||
date="2012-07-06T00:21:43Z"
|
||||
content="""
|
||||
It's in the `assistant` branch of git://git-annex.branchable.com/
|
||||
"""]]
|
28
doc/design/assistant/blog/day_26__dying_drives.mdwn
Normal file
28
doc/design/assistant/blog/day_26__dying_drives.mdwn
Normal file
|
@ -0,0 +1,28 @@
|
|||
My laptop's SSD died this morning. I had some work from yesterday
|
||||
committed to the git repo on it, but not pushed as it didn't build.
|
||||
Luckily I was able to get that off the SSD, which is now a read-only
|
||||
drive -- even mounting it fails with fsck write errors.
|
||||
|
||||
Wish I'd realized the SSD was dying before the day before my trip to
|
||||
Nicaragua..
|
||||
Getting back to a useful laptop used most of my time and energy today.
|
||||
|
||||
I did manage to fix transfers to not block the rest of the assistant's
|
||||
threads. Problem was that, without Haskell's threaded runtime, waiting
|
||||
on something like a rsync command blocks all threads. To fix this,
|
||||
transfers now are run in separate processes.
|
||||
|
||||
Also added code to allow multiple transfers to run at once. Each transfer
|
||||
takes up a slot, with the number of free slots tracked by a `QSemN`.
|
||||
This allows the transfer starting thread to block until a slot frees up,
|
||||
and then run the transfer.
|
||||
|
||||
This needs to be extended to be aware of transfers initiated by remotes.
|
||||
The transfer watcher thread should detect those starting and stopping
|
||||
and update the `QSemN` accordingly. It would also be nice if transfers
|
||||
initiated by remotes would be delayed when there are no free slots for them
|
||||
... but I have not thought of a good way to do that.
|
||||
|
||||
There's a bug somewhere in the new transfer code, when two transfers are
|
||||
queued close together, the second one is lost and doesn't happen.
|
||||
Would debug this, but I'm spent for the day.
|
|
@ -3,27 +3,6 @@ all the other git clones, at both the git level and the key/value level.
|
|||
|
||||
## action items
|
||||
|
||||
* on-disk transfers in progress information files (read/write/enumerate)
|
||||
**done**
|
||||
* locking for the files, so redundant transfer races can be detected,
|
||||
and failed transfers noticed **done**
|
||||
* transfer info for git-annex-shell **done**
|
||||
* update files as transfers proceed. See [[progressbars]]
|
||||
(updating for downloads is easy; for uploads is hard)
|
||||
* add Transfer queue TChan **done**
|
||||
* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
|
||||
**done**
|
||||
* Poll transfer in progress info files for changes (use inotify again!
|
||||
wow! hammer, meet nail..), and update the TransferInfo Map **done**
|
||||
* enqueue Transfers (Uploads) as new files are added to the annex by
|
||||
Watcher. **done**
|
||||
* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
|
||||
Watcher. **done**
|
||||
* Write basic Transfer handling thread. Multiple such threads need to be
|
||||
able to be run at once. Each will need its own independant copy of the
|
||||
Annex state monad. **done**
|
||||
* Write transfer control thread, which decides when to launch transfers.
|
||||
**done**
|
||||
* Check that download transfer triggering code works (when a symlink appears
|
||||
and the remote does *not* upload to us.
|
||||
* Investigate why transfers seem to block other git-annex assistant work.
|
||||
|
@ -35,31 +14,23 @@ all the other git clones, at both the git level and the key/value level.
|
|||
* git-annex needs a simple speed control knob, which can be plumbed
|
||||
through to, at least, rsync. A good job for an hour in an
|
||||
airport somewhere.
|
||||
* file transfer processes are not waited for, contain the zombies.
|
||||
|
||||
## git syncing
|
||||
## longer-term TODO
|
||||
|
||||
1. Can use `git annex sync`, which already handles bidirectional syncing.
|
||||
When a change is committed, launch the part of `git annex sync` that pushes
|
||||
out changes. **done**; changes are pushed out to all remotes in parallel
|
||||
1. Watch `.git/refs/remotes/` for changes (which would be pushed in from
|
||||
another node via `git annex sync`), and run the part of `git annex sync`
|
||||
that merges in received changes, and follow it by the part that pushes out
|
||||
changes (sending them to any other remotes).
|
||||
[The watching can be done with the existing inotify code! This avoids needing
|
||||
any special mechanism to notify a remote that it's been synced to.]
|
||||
**done**
|
||||
1. Periodically retry pushes that failed. **done** (every half an hour)
|
||||
1. Also, detect if a push failed due to not being up-to-date, pull,
|
||||
and repush. **done**
|
||||
2. Use a git merge driver that adds both conflicting files,
|
||||
so conflicts never break a sync. **done**
|
||||
3. Investigate the XMPP approach like dvcs-autosync does, or other ways of
|
||||
* Investigate the XMPP approach like dvcs-autosync does, or other ways of
|
||||
signaling a change out of band.
|
||||
4. Add a hook, so when there's a change to sync, a program can be run
|
||||
* Add a hook, so when there's a change to sync, a program can be run
|
||||
and do its own signaling.
|
||||
5. --debug will show often unnecessary work being done. Optimise.
|
||||
6. It would be nice if, when a USB drive is connected,
|
||||
* --debug will show often unnecessary work being done. Optimise.
|
||||
* It would be nice if, when a USB drive is connected,
|
||||
syncing starts automatically. Use dbus on Linux?
|
||||
* This assumes the network is connected. It's often not, so the
|
||||
[[cloud]] needs to be used to bridge between LANs.
|
||||
* Configurablity, including only enabling git syncing but not data transfer;
|
||||
only uploading new files but not downloading, and only downloading
|
||||
files in some directories and not others. See for use cases:
|
||||
[[forum/Wishlist:_options_for_syncing_meta-data_and_data]]
|
||||
|
||||
## misc todo
|
||||
|
||||
|
@ -90,7 +61,42 @@ reachable remote. This is worth doing first, since it's the simplest way to
|
|||
get the basic functionality of the assistant to work. And we'll need this
|
||||
anyway.
|
||||
|
||||
## other considerations
|
||||
## done
|
||||
|
||||
This assumes the network is connected. It's often not, so the
|
||||
[[cloud]] needs to be used to bridge between LANs.
|
||||
1. Can use `git annex sync`, which already handles bidirectional syncing.
|
||||
When a change is committed, launch the part of `git annex sync` that pushes
|
||||
out changes. **done**; changes are pushed out to all remotes in parallel
|
||||
1. Watch `.git/refs/remotes/` for changes (which would be pushed in from
|
||||
another node via `git annex sync`), and run the part of `git annex sync`
|
||||
that merges in received changes, and follow it by the part that pushes out
|
||||
changes (sending them to any other remotes).
|
||||
[The watching can be done with the existing inotify code! This avoids needing
|
||||
any special mechanism to notify a remote that it's been synced to.]
|
||||
**done**
|
||||
1. Periodically retry pushes that failed. **done** (every half an hour)
|
||||
1. Also, detect if a push failed due to not being up-to-date, pull,
|
||||
and repush. **done**
|
||||
2. Use a git merge driver that adds both conflicting files,
|
||||
so conflicts never break a sync. **done**
|
||||
|
||||
* on-disk transfers in progress information files (read/write/enumerate)
|
||||
**done**
|
||||
* locking for the files, so redundant transfer races can be detected,
|
||||
and failed transfers noticed **done**
|
||||
* transfer info for git-annex-shell **done**
|
||||
* update files as transfers proceed. See [[progressbars]]
|
||||
(updating for downloads is easy; for uploads is hard)
|
||||
* add Transfer queue TChan **done**
|
||||
* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
|
||||
**done**
|
||||
* Poll transfer in progress info files for changes (use inotify again!
|
||||
wow! hammer, meet nail..), and update the TransferInfo Map **done**
|
||||
* enqueue Transfers (Uploads) as new files are added to the annex by
|
||||
Watcher. **done**
|
||||
* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
|
||||
Watcher. **done**
|
||||
* Write basic Transfer handling thread. Multiple such threads need to be
|
||||
able to be run at once. Each will need its own independant copy of the
|
||||
Annex state monad. **done**
|
||||
* Write transfer control thread, which decides when to launch transfers.
|
||||
**done**
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue