Maybe the spec allows it, but broadcasting self-directed presence info to
all buddies is just insane.
I had to bring back the IQ messages for self-pairing, while still using
directed presence for other pairing. Ugly.
Testing between Google Talk and prosody, the directed IQ messages
were not received. Google Talk probably only relays them between
clients using the same account.
I first tried even more directed presence, with each client JID being sent
a separate presence, but that didn't work on Google Talk, particularly
it was ignored when one client sent it to another client using the same
account.
So, presence directed at the user@host of the client to pair with. Tested
working between Google Talk and prosody (in both directions), as well
as between two clients with the same account on Google Talk, and
two clients with the same account on prosody.
Only problem with this form of directed presence is that if I also use it
for git pushes, more clients than are interested in a push's data will
receive it. So I may need some better approach, or a hybrid between
directed IQ and directed presence.
Amusingly, I am not really using xmpp ping for pairing. I forgot to put in
the ping tag! And when I did, it stopped working, on Google Talk. Seems
it handles client to client pings, at least using the same JID, without
actually sending them to the end client. My mistake avoided this,
and seems to work, so I've left it as-is for now, with just the git-annex
tag in an IQ message. Also tested on prosody.
Wrote a better git remote name sanitizer. Git blows up on lots of weird
stuff, especially if it starts the remote name, but I managed to get
some common punctuation working.
Still wait 1 minute after a change before waiting on the next change, but don't
wait at the start, when we might get a pull that contains config changes
right away.
Currently have three old versions of functions that more reworking is
needed to remove: getDaemonStatusOld, modifyDaemonStatusOld_, and
modifyDaemonStatusOld
This is a nice win; much less code runs in Annex, so other threads have
more chances to run concurrently.
I do notice that renaming a file has gone from 1 to 2 commits. I think this
is due to the above improvement letting the committer run more frequently,
so it commits the rm first.
Converted several threads to run in the monad.
Added a lot of useful combinators for working with the monad.
Now the monad includes the name of the thread.
Some debugging messages are disabled pending converting other threads.
I now have this topology working:
assistant ---> {bare repo, special remote} <--- assistant
And, I think, also this one:
+----------- bare repo --------+
v v
assistant ---> special remote <--- assistant
While before with assistant <---> assistant connections, both sides got
location info updated after a transfer, in this topology, the bare repo
*might* get its location info updated, but the other assistant has no way to
know that it did. And a special remote doesn't record location info,
so transfers to it won't propigate out location log changes at all.
So, for these to work, after a transfer succeeds, the git-annex branch
needs to be pushed. This is done by recording a synthetic commit has
occurred, which lets the pusher handle pushing out the change (which will
include actually committing any still journalled changes to the git-annex
branch).
Of course, this means rather a lot more syncing action than happened
before. At least the pusher bundles together very close together pushes,
somewhat. Currently it just waits 2 seconds between each push.
I am befuddled that Twitter Bootstrap has no built-in Icon for The Cloud,
and also that Chromium's depiction of CLOUD (U+2601) has an uncanny
resemblance to PILE OF POO (U+1F4A9) when rendered small, and looks like a
looming Frankenstorm when rendered large, and not a sweet, sunny, nothing
can go wrong The Cloud.
<http://www.fileformat.info/info/unicode/char/2601/browsertest.htm>
So, I must resort to irony in my choice of icons.
Adjust build deps to ensure that only a fixed version of the library will
be used.
Also, removed the bound thread stuff, which I now think was (probably)
a red herring.
MountWatcher can't do this, because it uses the session dbus,
and won't have access to the new DBUS_SESSION_BUS_ADDRESS if a new session
is started.
Bumped dbus library version, FD leak in it is fixed.
Currently relies on SRV being set, or the JID's hostname being the server
hostname and the port being default. Future work: Allow manual
configuration of user name, hostname, and port.
Now when the dbus connection is dropped, it'll fall back to polling.
I could make it try to reconnect, but there's a FD leak in the dbus
library, so not yet.
This *may* solve the segfault I was seeing when the XMPP library called
startTLS. My hypothesis is as follows:
* TLS is documented
(http://www.gnu.org/software/gnutls/manual/gnutls.html#Thread-safety)
thread safe, but only when a single thread accesses it.
* forkIO threads are not bound to an OS thread, so it was possible for
the threaded runtime to run part of the XMPP code on one thread, and
then switch to another thread later.
So, forkOS, with its bound threads, should be used for the XMPP thread.
Since the crash doesn't happen reliably, I am not yet sure about this fix.
Note that I kept all the other threads in the assistant unbound, because
bound threads have significantly higher overhead.
Seems presence notifications are not sent to clients that have marked
themselves unavailable. (Testing with google talk.)
This is the death knell for the presence hack, because it has to stay
available, and even the toggle to unavailable and back could cause it to
miss a notification. Still, flipped it so it basically works, for now.
Lacking error handling, reconnection, credentials configuration,
and doesn't actually do anything when it receives an incoming notification.
Other than that, it might work! :)
Hooked up everything that needs to notify on pushes. Note that
syncNewRemote does not notify. This is probably ok, and I'd need to thread
more state through to make it do so.
This is only set up to support a single push notification method; I didn't
use a NotificationBroadcaster. Partly because I don't yet know what info
about pushes needs to be communicated, so my data types are only
preliminary.
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.
Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.
A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.
Also added a uuid.log cache. All other config files already had caches.
This can result in the file being dropped, or being downloaded, or even
being dropped from some other repo.
It's even possible to create a file in a directory where content is not
wanted, which will make the assistant immediately send it elsewhere, and
then drop it.
This was complicated quite a bit by needing to check numcopies. I optimised
that, so it only looks up numcopies once per file, no matter how many
remotes it checks to drop from. Although it did just occur to me that
it might be better to first check if it wants to drop content, and only
then check numcopies..
None-bare removable drive repos don't have the assistant running in them,
so don't get their master branch updated as syncs come in. This will
probably change later, but for now, set up something that works.
Also, set the description of a newly added drive's repo locally. This
ensures that the repo edit form has the description in it.
This avoids the expensive transfer scan relying on its list of remotes
to scan being accurate throughout, which it will not be when the user
pauses syncing to a remote.
I feel it's ok to queue transfers to *any* known remote, not just the ones
being scanned.
Note that there are still small races where after syncing to a remote is
paused, a transfer can be queued for it. Not just in the expensive transfer
scan, but in the cheap failed transfer scan, and elsewhere.
Although I observe that these toggles don't always prevent syncing.
When a transfer scan is active, it will still queue items from the disabled
remote.
Also, transfers from a disabled remote show up as from "unknown", which is
not ideal.
This reserves annex.ignore for repos that should not be visible at all;
repos with syncing disabled are now skipped by the assistant, but are
displayed in the list and can be configured.
I noticed this while offline (so that lack of solar power is good for something).
Apparently it tries to bind multicast to lo, and that fails.
If this happens, catch it, and retry until a real network interface becomes
available.
It may be that this should tie into the NetWatcher, and rebind whenever
an interface comes up. Needs testing..
I'm using transfer for most things, both removable drives and cloud
storage, because it's the safest choice. We'll see if it makes sense
to prompt for the group when setting this up, or let the user pick
something else after the fact.
webapp: Adds newly created repositories to one of these groups:
clients, drives, servers
This is heuristic, but it's a pretty good heuristic, and can always be
configured.
Both when queueing downloads, and uploads, consults the preferred content
settings.
I didn't make it check yet when requeing failed transfers or queuing
deferred downloads; dealing with the preferred content settings (or indeed,
other settings) changing while the assistant is running still needs work.
Makes it safe to use git annex unlock with the watcher/assistant.
And also to mix use of the watcher/assistant with regular files stored in git.
Long ago, I had avoided doing this check, except during the startup scan,
because it would be slow to run ls-files repeatedly.
But then I added the lsof check, and to make that fast, got it to detect
batch file adds. So let's move the ls-files check to also occur when it'll
have a batch, and can check them all with one call.
This does slow down adding a single file by just a bit, but really only
a little bit. (The lsof check is probably more expensive.) It also
speeds up the startup scan, especially when there are lots of new files
found by the scan.
Also, fixed the sleep for annex.delayadd to not run while the threadstate
lock is held, so it doesn't unnecessarily freeze everything else.
Also, --force no longer makes it skip the lsof check, which was not
documented, and seems never a good idea.
This was needed for the OSX self-contained app, but is a generally good
idea. It avoids needing perl; is probably faster; and could eventually
be replaced by something faster yet.
I put it in ~/.ssh/ because there's no reliable way to get it into PATH,
and OSX ssh doesn't even honor user's PATH by default.
authorized_keys generators will need to check if it's there. Not done yet.
