Commit graph

641 commits

Author SHA1 Message Date
Joey Hess
cd7055631f batch commit every 5 thousand changes, not 10 thousand
There's a tradeoff between making less frequent commits, and
needing to use memory to store all the changes that are coming
in. At 10 thousand, it needs 150 mb of memory. 5 thousand drops
that down to 90 mb or so.

This also turns out to have significant imact on total run time.
I benchmarked 10k changes taking 27 minutes. But two 5k batches
took only 21 minutes.
2013-04-24 16:40:35 -04:00
Joey Hess
bda237f14a convert PendingAddChange back to Change when an add fails
If an add failed, we should lose the KeySource, since it, presumably,
differs due to a change that was made to the file.

(The locked down file is already deleted.)
2013-04-24 16:29:25 -04:00
Joey Hess
a929e6641a show one alert when bulk adding files
Turns out that a lot of the time spent in a bulk add was just updating the
add alert to rotate through each file that was added. Showing one alert
makes for a significant speedup.

Also, when the webapp is open, this makes it take quite a lot less cpu
during bulk adds.

Also, it lets the user know when a bulk add happened, which is sorta
nice..
2013-04-24 13:04:46 -04:00
Joey Hess
ca72b1ac7b assistant: when an add fails, requeue it for later
See analysis in bug report for one way this could happen.
2013-04-23 18:23:04 -04:00
Joey Hess
2c42e70f6c rename module 2013-04-23 11:38:52 -04:00
Joey Hess
8861e270be sync, assistant: Sync with remotes that have annex-ignore set
This is so git remotes on servers without git-annex installed can be used
to keep clients' git repos in sync.

This is a behavior change, but since annex-sync can be set to disable
syncing with a remote, I think it's acceptable.
2013-04-22 14:57:09 -04:00
Joey Hess
780ed8ebf1 fix warning 2013-04-20 18:52:46 -04:00
Joey Hess
090a69f00c assistant: Work around misfeature in git 1.8.2 that makes git commit --alow-empty -m "" run an editor.
See http://git-annex.branchable.com/bugs/assistant_hangs_during_commit/
2013-04-18 16:27:17 -04:00
Joey Hess
824c95376a refactor static route definition -- android webapp build fix
Incidentially should work around the last problem that prevented the webapp
building on Android. (Except for a few places I need to clean up after
hand-fixing the spliced TH code.)
2013-04-17 01:37:08 -04:00
Joey Hess
2c365b8b74 remove debug 2013-04-11 16:36:45 -04:00
Joey Hess
1eb3fff787 addurl: Register transfer so the webapp can see it.
* addurl: Register transfer so the webapp can see it.
* addurl: Automatically retry downloads that fail, as long as some
  additional content was downloaded.
2013-04-11 16:14:17 -04:00
Joey Hess
04a27ad926 assistant: Bug fix to avoid annexing the files that git uses to stand in for symlinks on FAT and other filesystem not supporting symlinks.
also, blog for the day..
2013-04-10 19:57:26 -04:00
Joey Hess
5e2e4347a3 webapp: New --listen= option allows running the webapp on one computer and connecting to it from another.
Does not yet use HTTPS. I'd need to generate a certificate, and I'm not
sure what's the best way to do that.
2013-04-08 15:04:35 -04:00
Joey Hess
602baae12e Bugfix: Direct mode no longer repeatedly checksums duplicated files.
Fixed by storing a list of cached inodes for a key, instead of just one.

Backwards compatability note: An old git-annex version will fail to parse
an inode cache file that has been written by a new version, and has
multiple items. It will succees if just one. So old git-annexes will have
even worse behavior when there are duplicated files, if that is possible.
I don't think it will be a problem. (Famous last words.)

Also, note that it doesn't expire old and unused inode caches for a key.
It would be possible to add this if needed; just look through the
associated files for a key and if there are more cached inodes, throw out
any not corresponding to associated files. Unless a file is being copied
repeatedly and the old copy deleted, this lack of expiry should not be a
problem.
2013-04-06 16:07:25 -04:00
Joey Hess
f1b0a4b404 Use lower case hash directories for storing files on crippled filesystems, same as is already done for bare repositories.
* since this is a crippled filesystem anyway, git-annex doesn't use
  symlinks on it
* so there's no reason to use the mixed case hash directories that we're
  stuck using to avoid breaking everyone's symlinks to the content
* so we can do what is already done for all bare repos, and make non-bare
  repos on crippled filesystems use the all-lower case hash directories
* which are, happily, all 3 letters long, so they cannot conflict with
  mixed case hash directories
* so I was able to 100% fix this and even resuming `git annex add` in the
  test case will recover and it will all just work.
2013-04-04 15:46:33 -04:00
Joey Hess
8b329c0317 refactor alert button creation code 2013-04-04 01:48:26 -04:00
Joey Hess
2c3aeec19b check for unused keys on an unwanted remote, and move them off, before deleting it 2013-04-03 19:03:16 -04:00
Joey Hess
021c564319 clean up urlrenderer handling when the webapp is not built 2013-04-03 17:48:54 -04:00
Joey Hess
9a5f421768 detect when unwanted remote is empty and remove it
Needs fixes to build when the webapp is disabled.
2013-04-03 17:01:40 -04:00
Joey Hess
47950cdf31 more efficient uuid to remote lookup 2013-04-02 16:39:11 -04:00
Joey Hess
6e7842475b convert "./file" from inotify to just "file"
This just prettifies some display.
2013-04-02 16:20:23 -04:00
Joey Hess
38d61f934d Update working tree files fully atomically
This avoids commit churn by the assistant when eg,
replacing a file with a symlink.

But, just as importantly, it prevents the working tree being left with a
deleted file if git-annex, or perhaps the whole system, crashes at the
wrong time.

(It also probably avoids confusing displays in file managers.)
2013-04-02 15:02:00 -04:00
Joey Hess
8c52b20cc7 optimise last commit
Rather than re-adding a direct mode file unnecessarily when it's not
changed, just re-stage the symlink.
2013-04-02 12:58:56 -04:00
Joey Hess
31cbde8190 assistant: Fix bug that could cause direct mode files to be unstaged from git.
My test case for this bug is to have the assistant running and syncing to
a remote, and create a file in the annex. Then at the command line run
git annex drop. The assistant sees that the file is gone, sees it's a wanted
file, and downloads it from the remote.

With a directory special remote and a small file, I was seeing around 1
time in 3, a race where the file got unstaged from git after it got
downloaded.

Looking at what direct mode content managing code does in this case, it
deletes the symlink, and then adds the file content back. It would be
possible, sometimes, to avoid removing the symlink and do this atomically.
And I probably should.. but in some cases, particularly where the file
needs to be run through `cp` (multiple direct mode files with same
content), there's no way to atomically replace the symlink with the
content.

Anyway, the bug turns out to be something that the watcher does right for
indirect mode, but not for direct mode. When it got an add event, it
checked to see if this was a new file, or one we've already added. In the
latter case, no add event was queued. But that means that only the rm event
is queued, and so it unstages the file.

Fixed by queueing an add event even when the file is already in git.

Tested by running hundreds of drops in a loop; file remained staged.
2013-04-02 12:45:31 -04:00
Joey Hess
c57baaaa30 webapp: Added UI to delete repositories. Closes: #689847 2013-03-31 16:38:05 -04:00
Joey Hess
5771cfce02 assistant: Check small files into git directly. 2013-03-29 16:54:59 -04:00
Joey Hess
67e817c6a1 New annex.largefiles setting, which configures which files git annex add and the assistant add to the annex.
I would have sort of liked to put this in .gitattributes, but it seems
it does not support multi-word attribute values. Also, making this a single
config setting makes it easy to only parse the expression once.

A natural next step would be to make the assistant `git add` files that
are not annex.largefiles. OTOH, I don't think `git annex add` should
`git add` such files, because git-annex command line tools are
not in the business of wrapping git command line tools.
2013-03-29 16:17:13 -04:00
Joey Hess
577128e9b8 avoid removing a transfer being made by another process
When another process is running a transfer, if we try to run a dup it'll
fail, and we should not remove the transfer from the transfer display.
2013-03-28 15:16:45 -04:00
Joey Hess
1d0b692198 webapp: Fix a race that sometimes caused alerts or other notifications to be missed if they occurred while a page was loading.
When a page is loaded, the javascript requests an notification url, and
does long polling on the url to be informed of changes. But if a change
occured before the notification url was requested, it would not be notified
of that change, and so the page display would not update.

I fixed this by *always* updating the page display after it gets
the notification url. This is extra work, but the overhead is not noticable
in the other overhead of loading a page.

(A nicer way would be to somehow record the version of a page initially
loaded, and then compare it with the current version when getting the
notification url, and only force an update if it's changed. But getting
the "version" of the different parts of the page that use long polling
is difficult.)
2013-03-27 14:56:20 -04:00
Joey Hess
3ef6b6a200 fix build with xmpp and w/o webapp 2013-03-24 18:55:19 -04:00
Joey Hess
b6d691aff7 maintain pools of running transferkeys processes (untested) 2013-03-19 18:46:29 -04:00
Joey Hess
bb284cd980 move display of transfer scan in progress to transfers section of dashboard
This way it's only visible when transfers are not running, which is about
what a user would expect.
2013-03-19 13:03:41 -04:00
Joey Hess
a30768cf7f new alert while scanning
Like the old one, but does not mention which remotes are scanned.
I think this is less confusing, as it does not imply the remotes
were somehow accessed (which they are not; inaccessible remotes
can be scanned.)
2013-03-18 23:15:48 -04:00
Joey Hess
aadb9069b3 deal with transferkey crashing
If transferkey crashes or even fails to run, the TransferWatcher will not
see the transfer info file be created, so will not remove the transfer
from the list of active transfers. This causes the list to grow
continually, and all active transfers are displayed in the webapp. So, put
in a guard.

I assume that transferkey will not exit 0 while neglecting to clean up.
2013-03-18 22:58:14 -04:00
Joey Hess
29d603b72a ensure 2013-03-18 22:41:16 -04:00
Joey Hess
b543842a7f optimisation for transfers to drives that are not plugged in
Rather than forking a git-annex transferkey only to have it fail,
just immediately record the failed transfer (so when the drive is plugged
in, the scan will retry it).
2013-03-18 20:40:24 -04:00
Joey Hess
cdb21649d0 webapp: Improved alerts displayed when syncing with remotes, and when syncing with a remote fails. 2013-03-18 17:23:47 -04:00
Joey Hess
35a0ae334c assistant: Fix OSX bug that prevented committing changed files to a repository when in indirect mode. 2013-03-17 17:01:43 -04:00
Joey Hess
746ffa773a assistant: Avoid syncing with annex-ignored remotes when reconnecting to the network, or connecting a drive. 2013-03-17 15:59:03 -04:00
Joey Hess
db2fe522ba xmpp: Re-enable XA flag, since disabling it did not turn out to help with the problems Google Talk has with not always sending presence messages to clients. 2013-03-16 16:00:37 -04:00
Joey Hess
55f20ae099 xmpp: send a presence query when there's an important message to send
This may work around google talk's horrible presence handling, in which
clients often don't learn about other clients, at least when using the same
account. This way, every time we start a git push over xmpp, we'll waste
bandwidth asking clients to please try again to identify themselves.
2013-03-16 15:36:47 -04:00
Joey Hess
e3354cf19c xmpp: --debug now enables a sanitized dump of the XMPP protocol
So I can debug these damn google talk presence issues.
2013-03-16 15:29:51 -04:00
Joey Hess
c94c99942b make liftAnnex and liftAssistant polymorphic, like liftIO 2013-03-16 00:19:56 -04:00
Joey Hess
77c82de4ea webapp: Display an alert when there are XMPP remotes, and a cloud transfer repository needs to be configured. 2013-03-15 17:52:41 -04:00
Joey Hess
39e979fb65 webapp: Improved UI for pairing your own devices together using XMPP. 2013-03-15 15:19:07 -04:00
Joey Hess
9cf4701a8f no longer need webapp state storage! excellent 2013-03-15 01:01:25 -04:00
Joey Hess
02facde154 assistant: Be smarter about avoiding unncessary transfers.
Just before starting a transfer, do one last check that it's still
preferred content.

I was just doing this for uploads, as part of the smarter flood filling
bug, but realized it's also possible for a download that was preferred
content to change to not be before the download begins, so check that too.
2013-03-13 13:36:02 -04:00
Joey Hess
60760cb430 tweak 2013-03-13 13:11:49 -04:00
Joey Hess
0ef8d806ac gratuitous rename HomeR -> DashboardR 2013-03-12 22:18:36 -04:00
Joey Hess
8221c2b4ed split repolist out of configuration, into its own tab (temporarily) 2013-03-12 21:51:03 -04:00
Joey Hess
393340dc3b better handling of batch renames
Rather than wait a full second, which may be longer than needed, or too
short to get all the rename events, we start a mode where we wait 1/10th of
a second, and if there are Changes received, wait again. Basically we're
back in batch mode when this happens.
2013-03-11 15:46:09 -04:00
Joey Hess
14fcfced48 detect directory rename and wait up to 1 second to get all the changes 2013-03-11 15:24:13 -04:00
Joey Hess
f340fd324c synthesize RmChange when a directory is deleted
This gets directory renames closer to being fully detected.
There's close to no extra overhead to doing it this way.
2013-03-11 15:14:42 -04:00
Joey Hess
06046a0d2b finish fast direct mode rename handling. wow, it's fast 2013-03-11 14:14:45 -04:00
Joey Hess
87cba71d5a fix changeFile to not be partial
That led to runtime crashes, without even a warning from -Wall. Yipes!
2013-03-11 13:55:36 -04:00
Joey Hess
61c5e8736c detect renames during commit, and .. um, do nothing special because it's lunch time
But I'm well set up to fast-track direct mode adds for renames now.
2013-03-11 12:56:47 -04:00
Joey Hess
74f723bb50 let's put type modules under the parent module, not in a Types directory 2013-03-10 22:24:13 -04:00
Joey Hess
2762ab03b4 assistant: generate better commits for renames 2013-03-10 22:10:26 -04:00
Joey Hess
b2c7ee5551 tweak 2013-03-10 20:20:58 -04:00
Joey Hess
f27c21eb0c avoid ugly alert caused by trying to push to unavailable removable drive 2013-03-10 18:42:28 -04:00
Joey Hess
65a4c7966f moved transfer queueing out of watcher and into committer
This cleaned up the code quite a bit; now the committer just looks at the
Change to see if it's a change that needs to have a transfer queued for it.
If I later want to add dropping keys for files that were removed, or
something like that, this should make it straightforward.

This also fixes a bug. In direct mode, moving a file out of an archive
directory failed to start a transfer to get its content. The problem
was that the file had not been committed to git yet, and so the transfer
code didn't want to touch it, since fileKey failed to get its key.
Only starting transfers after a commit avoids this problem.
2013-03-10 18:16:03 -04:00
Joey Hess
0e508f860a assistant: Sync with all git remotes on startup. 2013-03-08 13:48:27 -04:00
Joey Hess
243e99717d empty buddy list when client is connecting
This is not perfect, because on loss of connection, we do not currently
immediately detect it and stop the client. It has to time out, and then
the buddy list will clear.

The NetWatcher should detect disconnects too..
2013-03-07 03:50:21 -04:00
Joey Hess
f8c2dc82d8 show when not connected to xmpp server 2013-03-06 22:02:47 -04:00
Joey Hess
c16adc25c4 assistant: XMPP git pull and push requests are cached and sent when presence of a new client is detected.
Noticed that, At startup or network reconnect, git push messages were sent,
often before presence info has been gathered, so were not sent to any
buddies.

To fix this, keep track of which buddies have seen such messages,
and when new presence is received from a buddy that has not yet seen it,
resend.

This is done only for push initiation messages, so very little data needs
to be stored.
2013-03-06 21:38:01 -04:00
Joey Hess
060119fdc4 better xmpp debugging 2013-03-06 18:28:34 -04:00
Joey Hess
aaec2cbf03 avoid false alert about syncing with xmpp remote 2013-03-06 17:54:45 -04:00
Joey Hess
cbb6e1fae4 tag xmpp pushes with jid
This fixes the issue mentioned in the last commit.

Turns out just collecting UUID of clients behind a XMPP remote is
insufficient (although I should probably still do it for other reasons),
because a single remote repo might be connected via both XMPP and local
pairing. So a way is needed to know when a push was received from any
client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
Joey Hess
c23ea9e311 assistant: Get back in sync with XMPP remotes after network reconnection, and on startup.
Make manualPull send push requests over XMPP.

When reconnecting with remotes, those that are XMPP remotes cannot
immediately be pulled from and scanned, so instead maintain a set of
(probably) desynced remotes, and put XMPP remotes on it. (This set could be
used in other ways later, if we can detect we're out of sync with other
types of remotes.)

The merger handles detecting when a XMPP push is received from a desynced
remote, and triggers a scan then, if they have in fact diverged.

This has one known bug: A single XMPP remote can have multiple clients
behind it. When this happens, only the UUID of one client is recorded
as the UUID of the XMPP remote. Pushes from the other XMPP clients will not
trigger a scan. If the client whose UUID is expected responds to the push
request, it'll work, but when that client is offline, we're SOL.
2013-03-06 15:09:31 -04:00
Joey Hess
907b0c0d78 better liftAnnex, avoid using runAnnex undefined 2013-03-04 16:36:38 -04:00
Joey Hess
c908672f3d fix another potential race with the watcher and direct mode
Watcher wants to rewrite symlink to fix it. But in direct mode, the symlink
could be replaced at any time with file content that has finished being
transferred by some other process. So, just don't touch it.

FWIW, I audited the rest of the assistant for places where it removes
files, and the rest is ok. I have not audited the rest of git-annex.
2013-03-04 15:09:32 -04:00
Joey Hess
1d388d5579 fixed the race breaking moving files from archive in direct mode
assistant: Fix bug in direct mode that could occur when a symlink is moved
out of an archive directory, and resulted in the file not being set to
direct mode when it was transferred.

The bug was that the direct mode mapping was not up-to-date when the
transferrer finished. So, finding no direct mode place to store the object,
it was put into .git/annex in indirect mode.

To fix this, just make the watcher update the direct mode mapping to
include the new file before it starts the transfer. (Seems we don't need to
update it to remove the old file if the link was moved, because the direct
mode code will notice it's not present and the mapping gets updated for its
removal later.)

The reason this was a race, and was probably not seen often is because
the committer came along and updated the direct mode mapping as part of
adding the moved symlink. But when the file was sufficiently small or
the remote sufficiently fast, this could happen after the transfer
finished.
2013-03-04 14:25:22 -04:00
Joey Hess
08bdea7e52 webapp: New preferences page allows enabling/disabling debug logging at runtime, as well as configuring numcopies and diskreserve. 2013-03-03 17:07:27 -04:00
Joey Hess
724711e4b7 fix 2013-03-03 15:18:24 -04:00
Joey Hess
789ca15012 better prevention of auto repack
Looking through the git sources (documentation is unclear),
it seems commit doesn't ever trigger git-gc, mostly fetching and merging
seems to. I cannot easily override the setting in all those places, so
instead set gc.auto in git config when initializing a repository with
the assistant.

This does mean that the user cannot set gc.auto=0 and completely avoid
repacks, as the assistant does it daily. But, it only does it after there
are 100x the default number of loose objects, so this is probably not going
to be too annoying.
2013-03-03 14:20:07 -04:00
Joey Hess
6dea43831e assistant: Prevent automatic commits from causing git-gc runs, as that can make things quite slow. Instead, git-gc --auto is run once a day. (This can be disabled by the usual gc.auto=0 setting.) 2013-03-03 13:44:35 -04:00
Joey Hess
0c13d3065e git subcommand cleanup
Pass subcommand as a regular param, which allows passing git parameters
like -c before it. This was already done in the pipeing set of functions,
but not the command running set.
2013-03-03 13:39:07 -04:00
Joey Hess
7a8833c81d remove excess log rotation; openLog rotates 2013-03-01 16:56:56 -04:00
Joey Hess
0d4b513ec2 assistant: Fix dropping content when a file is moved to an archive directory.
A transfer is queued, but if the file has already been transferred to the
remote before, the transfer is skipped. In this case, it needs to perform
any actions it would normally take after finishing the transfer, like
dropping the local object.
2013-03-01 16:46:36 -04:00
Joey Hess
4d33423067 assistant: Avoid noise in logs from git commit about typechanged files in direct mode repositories. 2013-03-01 16:21:29 -04:00
Joey Hess
a733271a9c add additional debug info about reasons for drops 2013-03-01 15:58:44 -04:00
Joey Hess
46c9cbeb1e add additional debug info about reasons for transfers 2013-03-01 15:23:59 -04:00
Joey Hess
1865b28094 assistant: Logs are rotated to avoid them using too much disk space.
This cannot completely guard against a runaway log event, and only runs
every hour anyway, but it should avoid most problems with very
long-running, active assistants using up too much space.
2013-03-01 13:30:48 -04:00
Joey Hess
08854afa10 fix inverted logic 2013-02-22 17:01:48 -04:00
Joey Hess
2a4dad8bd4 remove debug prints 2013-02-19 23:18:15 -04:00
Joey Hess
d7c93b8913 fully support core.symlinks=false in all relevant symlink handling code
Refactored annex link code into nice clean new library.

Audited and dealt with calls to createSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:  ( liftIO $ createSymbolicLink linktarget file
  only when core.symlinks=true
Assistant/WebApp/Configurators/Local.hs:                createSymbolicLink link link
  test if symlinks can be made
Command/Fix.hs: liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/FromKey.hs:     liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/Indirect.hs:                    liftIO $ createSymbolicLink l f
  refuses to run if core.symlinks=false
Init.hs:                createSymbolicLink f f2
  test if symlinks can be made
Remote/Directory.hs:    go [file] = catchBoolIO $ createSymbolicLink file f >> return True
  fast key linking; catches failure to make symlink and falls back to copy
Remote/Git.hs:          liftIO $ catchBoolIO $ createSymbolicLink loc file >> return True
  ditto
Upgrade/V1.hs:                          liftIO $ createSymbolicLink link f
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to readSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:		( liftIO $ catchMaybeIO $ readSymbolicLink file
  only when core.symlinks=true
Assistant/Threads/Watcher.hs:		ifM ((==) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file))
  code that fixes real symlinks when inotify sees them
  It's ok to not fix psdueo-symlinks.
Assistant/Threads/Watcher.hs:		mlink <- liftIO (catchMaybeIO $ readSymbolicLink file)
  ditto
Command/Fix.hs:	stopUnless ((/=) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file)) $ do
  command only works in indirect mode
Upgrade/V1.hs:	getsymlink = takeFileName <$> readSymbolicLink file
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to isSymbolicLink.
(Typically used with getSymbolicLinkStatus, but that is just used because
getFileStatus is not as robust; it also works on pseudolinks.)
Remaining calls are all safe, because:

Assistant/Threads/SanityChecker.hs:                             | isSymbolicLink s -> addsymlink file ms
  only handles staging of symlinks that were somehow not staged
  (might need to be updated to support pseudolinks, but this is
  only a belt-and-suspenders check anyway, and I've never seen the code run)
Command/Add.hs:         if isSymbolicLink s || not (isRegularFile s)
  avoids adding symlinks to the annex, so not relevant
Command/Indirect.hs:                            | isSymbolicLink s -> void $ flip whenAnnexed f $
  only allowed on systems that support symlinks
Command/Indirect.hs:            whenM (liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f) $ do
  ditto
Seek.hs:notSymlink f = liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f
  used to find unlocked files, only relevant in indirect mode
Utility/FSEvents.hs:                    | Files.isSymbolicLink s = runhook addSymlinkHook $ Just s
Utility/FSEvents.hs:                                            | Files.isSymbolicLink s ->
Utility/INotify.hs:                             | Files.isSymbolicLink s ->
Utility/INotify.hs:                     checkfiletype Files.isSymbolicLink addSymlinkHook f
Utility/Kqueue.hs:              | Files.isSymbolicLink s = callhook addSymlinkHook (Just s) change
  all above are lower-level, not relevant

Audited and dealt with calls to isSymLink.
Remaining calls are all safe, because:

Annex/Direct.hs:			| isSymLink (getmode item) =
  This is looking at git diff-tree objects, not files on disk
Command/Unused.hs:		| isSymLink (LsTree.mode l) = do
  This is looking at git ls-tree, not file on disk
Utility/FileMode.hs:isSymLink :: FileMode -> Bool
Utility/FileMode.hs:isSymLink = checkMode symbolicLinkMode
  low-level

Done!!
2013-02-17 16:43:14 -04:00
Joey Hess
630f4531a7 fix assistant's use of lsof in crippled filesystem mode 2013-02-15 13:08:22 -04:00
Joey Hess
47477b2807 crippled filesystem support, probing and initial support
git annex init probes for crippled filesystems, and sets direct mode, as
well as `annex.crippledfilesystem`.

Avoid manipulating permissions of files on crippled filesystems.
That would likely cause an exception to be thrown.

Very basic support in Command.Add for cripped filesystems; avoids the lock
down entirely since doing it needs both permissions and hard links.
Will make this better soon.
2013-02-14 14:15:26 -04:00
Joey Hess
5737c49804 support Android's crippled lsof 2013-02-11 17:33:45 -04:00
Joey Hess
547d7745fb pre-commit: Update direct mode mappings.
Making the pre-commit hook look at git diff-index to find changed direct
mode files and update the mappings works pretty well.

One case where it does not work is when a file is git annex added, and then
git rmed, and then this is committed. That's a no-op commit, so the hook
probably doesn't even run, and it certianly never notices that the file
was deleted, so the mapping will still have the original filename in it.

For this and other reasons, it's important that the mappings still be
treated as possibly inconsistent.

Also, the assistant now allows the pre-commit hook to run when in direct
mode, so the mappings also get updated there.
2013-02-06 12:44:19 -04:00
Joey Hess
b19c2e6122 assistant: Fix location log when adding new file in direct mode. 2013-02-05 13:41:48 -04:00
Joey Hess
a261412c25 close 2013-01-28 15:39:51 +11:00
Joey Hess
a8bb2749b2 assistant: Ignore .DS_Store on OSX. 2013-01-28 15:13:22 +11:00
Joey Hess
5cd152b8a9 annex.autocommit
New setting, can be used to disable autocommit of changed files by the
assistant, while it still does data syncing and other tasks.

Also wired into webapp UI
2013-01-27 22:43:05 +11:00
Joey Hess
76ddf9b6d3 webapp: Now allows restarting any threads that crash. 2013-01-26 17:09:33 +11:00
Joey Hess
1713ed95f7 use async to track and manage threads 2013-01-26 14:14:32 +11:00
Joey Hess
d7ca6fb856 webapp: Now always logs to .git/annex/daemon.log
It used to not log to daemon.log when a repository was first created, and
when starting the webapp. Now both do. Redirecting stdout and stderr to the
log is tricky when starting the webapp, because the web browser may want to
communicate with the user. (Either a console web browser, or web.browser = echo)
This is handled by restoring the original fds when running the browser.
2013-01-15 13:34:59 -04:00
Joey Hess
f51ad2a00c assistant: Avoid committer crashing if a file is deleted at the wrong instant. 2013-01-14 15:02:13 -04:00
Joey Hess
fb9ab8b2b5 catch failure to start dbus service 2013-01-14 13:22:19 -04:00
Joey Hess
6ec2802228 fix 2013-01-10 16:43:22 -04:00
Joey Hess
f99563ad14 fix fix to names 2013-01-10 16:18:26 -04:00
Joey Hess
7e6fbd3507 seems I got the name wrong 2013-01-10 16:08:40 -04:00
Joey Hess
d22d06a84a assistant: Support new gvfs dbus names used in Gnome 3.6. (untested) 2013-01-10 15:06:08 -04:00
Joey Hess
6f7ae84650 webapp: Use IP address, rather than localhost
since some systems may have configuration problems or other issues that
prevent web browsers from connecting to the right localhost IP for the
webapp.

Tested on both ipv4 and ipv6 localhost. Url for the latter looks like:
http://[::1]:50676
2013-01-09 23:18:00 -04:00
Joey Hess
aedfcde969 guard readSymbolicLink
throws an exception if the file is not a symlink
2013-01-05 16:07:27 -04:00
Joey Hess
1cdf2b923d assistant: Make expensive transfer scan work fully in direct mode.
The expensive scan uses lookupFile, but in direct mode, that doesn't work
for files that are present. So the scan was not finding things that are
present that need to be uploaded. (It did find things not present that
needed to be downloaded.)

Now lookupFile also works in direct mode. Note that it still prefers
symlinks on disk to info committed to git, in direct mode. This is
necessary to make things like Assistant.Threads.Watcher.onAddSymlink
work correctly, when given a new symlink not yet checked into git (or
replacing a file checked into git).
2013-01-05 15:57:53 -04:00
Joey Hess
bad9b6761d restart UI
Browser behavior is not ideal; a new tab is opened on restart.
Browsers won't let me redirect to a file:// so I cannot use the old tab.
2013-01-03 18:50:30 -04:00
Joey Hess
de2e287133 webapp: Add UI to stop assistant.
Would like to also have restart UI, but that's rather harder to do,
seems it'd need to start another copy of the webapp, and redirect the
browser to its new url, but running two assistants in the same repo at
the same time isn't good.
2013-01-03 15:24:21 -04:00
Joey Hess
4008590c68 type based git config handling for remotes
Still a couple of places that use git config ad-hoc, but this is most of it
done.
2013-01-01 13:58:14 -04:00
Joey Hess
7f7c31df1c type based git config handling
Now there's a Config type, that's extracted from the git config at startup.
Note that laziness means that individual config values are only looked up
and parsed on demand, and so we get implicit memoization for all of them.
So this is not only prettier and more type safe, it optimises several
places that didn't have explicit memoization before. As well as getting rid
of the ugly explicit memoization code.

Not yet done for annex.<remote>.* configuration settings.
2012-12-29 23:10:18 -04:00
Joey Hess
8cc27b8afc avoid double commits with inotify when direct mode file is created 2012-12-29 14:58:13 -04:00
Joey Hess
c0f9810f0b OSX assistant: Uses direct mode by default when setting up a new local repository. 2012-12-28 16:42:11 -04:00
Joey Hess
bf3270c5b7 add missing modifyHook for watcher
Needed for FSEvents, which calls that hook for modified files.
inotify seems to call the add hook, so I didn't notice it before.
2012-12-28 16:00:45 -04:00
Joey Hess
eb40227d15 assistant direct mode file add/change bookkeeping
When a file is changed in direct mode, the old content is probably lost
(at least from the local repo), and bookeeping needs to be updated to
reflect this.

Also, synthetic add events are generated at assistant startup, so
make it detect when the file has not really changed, and avoid re-adding
it.

This does add the overhead of querying the runing git cat-file for the
key that's recorded in git for the file, each time a file is added or
modified in direct mode.
2012-12-25 15:48:15 -04:00
Joey Hess
8a8380f1b7 use sync command merge engine in assistant
To handle direct mode merging.
2012-12-25 14:10:07 -04:00
Joey Hess
cc5140d295 assistant adding of modified files in direct mode
Works with inotify, but I think in kqueue we don't get events
existing files that get modified.
2012-12-24 14:42:19 -04:00
Joey Hess
95db595e91 make startup scan for deleted files work in direct mode
git add --update cannot be used, because it'll stage typechanged direct
mode files. Intead, use ls-files to find deleted files, and stage them
ourselves.

It seems that no commit was made before when the scan staged deleted files.
(Probably masked since if files were added, a commit happened then..)
Now that I'm doing the staging, I was also able to fix that bug.
2012-12-24 14:24:13 -04:00
Joey Hess
c6d2bbe402 assistant adding of files in direct mode 2012-12-24 13:37:29 -04:00
Joey Hess
82617b92e9 move thirdparty program installation for standalone bundle into haskell program
This allows it to use Build.SysConfig to always install the programs
configure detected. Amoung other fixes, this ensures the right uuid
generator and checksum programs are installed.

I also cleaned up the handling of lsof's path; configure now checks for
it in PATH, but falls back to looking for it in sbin directories.
2012-12-14 16:07:59 -04:00
Joey Hess
0d50a6105b whitespace fixes 2012-12-13 00:45:27 -04:00
Joey Hess
99a8a5297c --auto fixes
* get/copy --auto: Transfer data even if it would exceed numcopies,
  when preferred content settings want it.
* drop --auto: Fix dropping content when there are no preferred content
  settings.
2012-12-06 13:22:16 -04:00
Joey Hess
5de0215986 remove dead code 2012-12-01 15:21:24 -04:00
Joey Hess
2f50af5273 better function name 2012-12-01 15:00:03 -04:00
Joey Hess
d2df2e52b4 remove hard link when sanity check failed
See http://git-annex.branchable.com/forum/dot_git_slash_annex_slash_tmp/
2012-11-29 16:54:51 -04:00
Joey Hess
3b35cde0e8 assistant: Retrival from glacier now handled. 2012-11-29 15:23:33 -04:00
Joey Hess
18fe34222a allow building webapp w/o webdav 2012-11-25 14:36:24 -04:00
Joey Hess
463cf58140 webapp and assistant glacier support 2012-11-24 16:30:15 -04:00
Joey Hess
c282c8b492 queue uploads when a new or renamed symlink is handled 2012-11-24 15:38:24 -04:00
Joey Hess
7addb89dc1 webapp: support box.com 2012-11-17 15:30:11 -04:00
Joey Hess
d468e37f46 automatic repolist updating 2012-11-13 17:50:54 -04:00
Joey Hess
8b22bd8831 add a yes/no confirm for xmpp pairing requests 2012-11-11 22:29:16 -04:00
Joey Hess
522e228aef auto-accept pair requests from JIDs already paired with 2012-11-11 18:16:11 -04:00
Joey Hess
5e44ab177c don't try to transfer data to/from XMPP remotes
Partition syncRemotes into ones needing git sync (ie, non-special remotes),
and ones needing data sync (ie, non-XMPP remotes).
2012-11-11 16:23:16 -04:00
Joey Hess
217eeede43 allow both one push and one receive-pack to run at the same time
Noticed that when pairing, sometimes both sides start to push, and the other
side sends a PushRequest, and the two deadlock, neither doing anything.
(Timeout eventually breaks this.) So, let both run at the same time.
2012-11-11 15:42:03 -04:00
Joey Hess
41085cdc8c set negative XMPP presence priority
This should help prevent git-annex clients receiving messages that
were intended for normal clients they're sharing the account with.

Changed XMPP protocol use to always send chat messages directed at the
specific client, as the negative priority blocks less directed messages.
2012-11-10 15:33:12 -04:00
Joey Hess
7a20b3f1d5 more refactoring 2012-11-10 14:01:24 -04:00
Joey Hess
0cf4c3ba9c more nice refactoring 2012-11-10 13:00:13 -04:00
Joey Hess
81953c2131 separate data type for push stages
This improves type safety.
2012-11-10 12:18:00 -04:00
Joey Hess
8b8964b523 refactor 2012-11-10 02:35:54 -04:00
Joey Hess
235f2ecb91 use a lookup table for speed 2012-11-10 02:18:19 -04:00
Joey Hess
16b2472574 refactor 2012-11-10 01:40:50 -04:00
Joey Hess
6acc4fa83f avoid excessive debugging 2012-11-09 17:51:26 -04:00
Joey Hess
47c032a748 run xmpp push actions in separate thread from xmpp client
Took me a while to figure out why the xmpp client was not receiving git xmpp
push messages after a push started.
2012-11-09 16:04:55 -04:00
Joey Hess
13547aa659 add canpush xmpp command 2012-11-09 14:34:06 -04:00
Joey Hess
fb3b9412e4 xmpp push control flow
It might even work, although nothing yet triggers XMPP pushes.

Also added a set of deferred push messages. Only one push can run at a
time, and unrelated push messages get deferred. The set will never grow
very large, because it only puts two types of messages in there, that
can only vary in the client doing the push.
2012-11-08 16:46:29 -04:00
Joey Hess
0238e4ba07 hooked up XMPP git push send/receive (but not yet control flow) 2012-11-08 14:04:41 -04:00
Joey Hess
b827afba03 rename another 'close' 2012-11-07 14:56:32 -04:00
Joey Hess
b8585a9452 NetManager is only currently used with dbus 2012-11-07 14:56:32 -04:00
Joey Hess
8f08aa3f45 better handling of lifting from XMPP -> Assistant 2012-11-05 19:39:08 -04:00
Joey Hess
fdb6a88877 ignore Messages from our client's own jid
shouldn't ever happen, but hey
2012-11-05 17:54:21 -04:00
Joey Hess
da6fb44446 finished XMPP pairing!
This includes keeping track of which buddies we're pairing with, to know
which PairAck are legitimate.
2012-11-05 17:43:17 -04:00
Joey Hess
da65c5c1d1 tweak 2012-11-05 16:00:55 -04:00
Joey Hess
db36b11e28 switch to silent chat messages for XMPP pairing
Along the way, significantly cleaned up Assistant.XMPP, and made XMPP
message decoding more efficient.
2012-11-05 15:40:56 -04:00
Joey Hess
0b8027e527 webapp: When setting up authorized_keys, use GIT_ANNEX_SHELL_DIRECTORY. 2012-11-05 12:35:05 -04:00
Joey Hess
a6cecfcf33 workaround for Google Talk's insane handling of self-directed presence
Maybe the spec allows it, but broadcasting self-directed presence info to
all buddies is just insane.

I had to bring back the IQ messages for self-pairing, while still using
directed presence for other pairing. Ugly.
2012-11-03 22:52:41 -04:00
Joey Hess
9cff286ea3 avoid repeated pairing alerts 2012-11-03 21:38:27 -04:00
Joey Hess
17e84a8096 switch to directed presence for pair requests
Testing between Google Talk and prosody, the directed IQ messages
were not received. Google Talk probably only relays them between
clients using the same account.

I first tried even more directed presence, with each client JID being sent
a separate presence, but that didn't work on Google Talk, particularly
it was ignored when one client sent it to another client using the same
account.

So, presence directed at the user@host of the client to pair with. Tested
working between Google Talk and prosody (in both directions), as well
as between two clients with the same account on Google Talk, and
two clients with the same account on prosody.

Only problem with this form of directed presence is that if I also use it
for git pushes, more clients than are interested in a push's data will
receive it. So I may need some better approach, or a hybrid between
directed IQ and directed presence.
2012-11-03 21:19:59 -04:00
Joey Hess
42f030c905 XMPP pair requests are now received, and an alert displayed 2012-11-03 17:46:22 -04:00
Joey Hess
b95c255b6d XMPP pairing notifications are now sent
Rest of pairing process still to do.
2012-11-03 16:01:09 -04:00
Joey Hess
cbbfd4d00b bugfix 2012-11-03 14:32:59 -04:00
Joey Hess
9a8d0be866 send a presence query when the buddy list is displayed
This ensures that clients that have not sent presence in a while will show
up in the list.
2012-11-03 14:25:06 -04:00
Joey Hess
a1228e27ed refactor XMPP client 2012-11-03 14:16:17 -04:00
Joey Hess
bf17262986 ignore our own client presence 2012-11-02 21:23:58 -04:00
Joey Hess
6a61829e2d add buddy list to pairing UI 2012-11-02 21:13:06 -04:00
Joey Hess
2ae43e71e0 xmpp buddy list tracking 2012-11-02 14:28:36 -04:00
Joey Hess
88d1907278 where indentation 2012-10-31 02:34:03 -04:00
Joey Hess
a1bd5b1fa0 don't delay at startup
Still wait 1 minute after a change before waiting on the next change, but don't
wait at the start, when we might get a pull that contains config changes
right away.
2012-10-30 17:23:42 -04:00
Joey Hess
93ffd47d76 finished pushing Assistant monad into all relevant files
All temporary and old functions are removed.
2012-10-30 17:14:51 -04:00
Joey Hess
47d94eb9a4 pushed Assistant monad down into DaemonStatus code
Currently have three old versions of functions that more reworking is
needed to remove: getDaemonStatusOld, modifyDaemonStatusOld_, and
modifyDaemonStatusOld
2012-10-30 15:39:15 -04:00
Joey Hess
ea8df8fe9f cleanup daemonStatus accessors 2012-10-30 14:44:18 -04:00
Joey Hess
68118b8986 split remaining assistant types 2012-10-30 14:34:48 -04:00
Joey Hess
f78ca9bc58 split out daemonstatus types 2012-10-30 14:11:14 -04:00
Joey Hess
42babf5012 split Commits and lifted 2012-10-29 19:35:18 -04:00
Joey Hess
d2294f0dfa split Changes and lifted 2012-10-29 19:30:23 -04:00
Joey Hess
39a3adf434 split BranchChange and lifted 2012-10-29 19:20:54 -04:00
Joey Hess
0c584bf70d split ScanRemotes and lifted 2012-10-29 19:14:30 -04:00
Joey Hess
86cb3faf51 moved ThreadedMonad to Types 2012-10-29 19:07:10 -04:00
Joey Hess
87ba4f8677 split and lift Assistant.Pushes 2012-10-29 17:52:43 -04:00
Joey Hess
d5a5c05a79 tweak 2012-10-29 17:02:50 -04:00
Joey Hess
1852eddce6 lift alertWhile 2012-10-29 16:49:47 -04:00
Joey Hess
e18b733c81 move alert display functions 2012-10-29 16:34:11 -04:00
Joey Hess
94ae5d14e5 more lifting 2012-10-29 16:28:45 -04:00
Joey Hess
5d57b28a34 lifted Assistant.Sync into Assistant monad
lots of nice cleanups
2012-10-29 16:22:14 -04:00
Joey Hess
0ffce4a942 tweak 2012-10-29 14:30:10 -04:00
Joey Hess
67ce7929a5 Assistant monad, stage 3
All toplevel named threads are converted to the Assistant monad.

Some utility functions still need to be converted.
2012-10-29 14:07:12 -04:00
Joey Hess
3eecb5b7bb converted 2 more threads.. only 2 more to go 2012-10-29 13:09:58 -04:00
Joey Hess
0ba4df3c1a tweak 2012-10-29 11:58:29 -04:00
Joey Hess
76768ad977 converted 6 more threads 2012-10-29 11:40:22 -04:00
Joey Hess
bad88e404a convert Watcher thread to Assistant monad
This is a nice win; much less code runs in Annex, so other threads have
more chances to run concurrently.

I do notice that renaming a file has gone from 1 to 2 commits. I think this
is due to the above improvement letting the committer run more frequently,
so it commits the rm first.
2012-10-29 09:55:40 -04:00
Joey Hess
4dbdc2b666 Assistant monad, stage 2.5
Converted several threads to run in the monad.

Added a lot of useful combinators for working with the monad.

Now the monad includes the name of the thread.

Some debugging messages are disabled pending converting other threads.
2012-10-29 02:21:04 -04:00
Joey Hess
4e765327ca Assistant monad, stage 1
This adds the Assistant monad, and an AssistantData structure.
So far, none of the assistant's threads run in the monad yet.
2012-10-29 00:15:43 -04:00
Joey Hess
64e1d7b579 send git-annex tag in initial presence
Will be used for finding other git-annex clients for pairing
2012-10-28 17:07:29 -04:00
Joey Hess
4ac2fd0a22 ensure that git-annex branch is pushed after a successful transfer
I now have this topology working:

   assistant ---> {bare repo, special remote} <--- assistant

And, I think, also this one:

        +----------- bare repo --------+
        v                              v
  assistant ---> special remote <--- assistant

While before with assistant <---> assistant connections, both sides got
location info updated after a transfer, in this topology, the bare repo
*might* get its location info updated, but the other assistant has no way to
know that it did. And a special remote doesn't record location info,
so transfers to it won't propigate out location log changes at all.

So, for these to work, after a transfer succeeds, the git-annex branch
needs to be pushed. This is done by recording a synthetic commit has
occurred, which lets the pusher handle pushing out the change (which will
include actually committing any still journalled changes to the git-annex
branch).

Of course, this means rather a lot more syncing action than happened
before. At least the pusher bundles together very close together pushes,
somewhat. Currently it just waits 2 seconds between each push.
2012-10-28 16:05:34 -04:00
Joey Hess
c71836269b (re)start XMPP when it's configured in the webapp 2012-10-27 00:50:14 -04:00
Joey Hess
2dc40ecbd1 reconnect XMPP when NetWatcher notices a change 2012-10-27 00:42:14 -04:00
Joey Hess
9fc8257392 xmpp reconnection
If it managed to run for 5 minutes, reconnect immediately. Otherwise,
wait 5 minutes before reconnecting.
2012-10-27 00:06:17 -04:00
Joey Hess
0c0ef99341 gnutls segfault fixed
Adjust build deps to ensure that only a fixed version of the library will
be used.

Also, removed the bound thread stuff, which I now think was (probably)
a red herring.
2012-10-26 23:07:30 -04:00
Joey Hess
d1e5f2d596 Merge branch 'master' into xmpp
Conflicts:
	Assistant/Threads/NetWatcher.hs
2012-10-26 19:42:24 -04:00
Joey Hess
8e4620a6c7 NetWatcher: When dbus connection is lost, try to reconnect.
MountWatcher can't do this, because it uses the session dbus,
and won't have access to the new DBUS_SESSION_BUS_ADDRESS if a new session
is started.

Bumped dbus library version, FD leak in it is fixed.
2012-10-26 19:38:27 -04:00
Joey Hess
1d04dff4f0 split out xmpp utilities 2012-10-26 14:44:36 -04:00
Joey Hess
82b7e835e4 run xmpp test in bound thread 2012-10-26 14:39:22 -04:00
Joey Hess
07494cbb4b XMPP configuration form
Currently relies on SRV being set, or the JID's hostname being the server
hostname and the port being default. Future work: Allow manual
configuration of user name, hostname, and port.
2012-10-26 14:17:09 -04:00
Joey Hess
a11fb94c65 SRV record construction 2012-10-26 13:03:08 -04:00
Joey Hess
8916ff1a6c hook up SRV lookups for XMPP 2012-10-26 12:55:29 -04:00
Joey Hess
0b1cf3a766 Merge branch 'master' into xmpp
Conflicts:
	Assistant/Threads/MountWatcher.hs
	Assistant/Threads/NetWatcher.hs
2012-10-26 00:10:41 -04:00
Joey Hess
ef7b53e784 improved dbus error handling
Now when the dbus connection is dropped, it'll fall back to polling.

I could make it try to reconnect, but there's a FD leak in the dbus
library, so not yet.
2012-10-26 00:02:03 -04:00
Joey Hess
fe96b28b4d convert the assistant to use a bound thread for XMPP
This *may* solve the segfault I was seeing when the XMPP library called
startTLS. My hypothesis is as follows:

* TLS is documented
  (http://www.gnu.org/software/gnutls/manual/gnutls.html#Thread-safety)
  thread safe, but only when a single thread accesses it.
* forkIO threads are not bound to an OS thread, so it was possible for
  the threaded runtime to run part of the XMPP code on one thread, and
  then switch to another thread later.

So, forkOS, with its bound threads, should be used for the XMPP thread.
Since the crash doesn't happen reliably, I am not yet sure about this fix.

Note that I kept all the other threads in the assistant unbound, because
bound threads have significantly higher overhead.
2012-10-25 15:47:27 -04:00
Joey Hess
8c9c8e3110 only use one push attribute
When pushing to multiple UUIDs, combine them all into a single push
attribute.
2012-10-25 13:31:45 -04:00
Joey Hess
91c0c7b9ef switch from presence toggle hack to git-annex tag in presence extended content
Push notifications are actually working over XMPP now!
2012-10-25 13:04:43 -04:00
Joey Hess
52d380d7f0 add a separate field for the JID 2012-10-24 19:09:02 -04:00
Joey Hess
145202f21e flip availablility
Seems presence notifications are not sent to clients that have marked
themselves unavailable. (Testing with google talk.)

This is the death knell for the presence hack, because it has to stay
available, and even the toggle to unavailable and back could cause it to
miss a notification. Still, flipped it so it basically works, for now.
2012-10-24 17:23:21 -04:00
Joey Hess
422b426460 pull from one of the remotes in a push notification
Still need to do something about transfer queueing, however. This could be
a real can of worms.
2012-10-24 16:21:45 -04:00
Joey Hess
32497feb2a initial implementation of XMPP push notifier (untested)
Lacking error handling, reconnection, credentials configuration,
and doesn't actually do anything when it receives an incoming notification.

Other than that, it might work! :)
2012-10-24 15:42:02 -04:00
Joey Hess
ae8a3ab31e added push notifier thread, currently a no-op
Hooked up everything that needs to notify on pushes. Note that
syncNewRemote does not notify. This is probably ok, and I'd need to thread
more state through to make it do so.

This is only set up to support a single push notification method; I didn't
use a NotificationBroadcaster. Partly because I don't yet know what info
about pushes needs to be communicated, so my data types are only
preliminary.
2012-10-24 13:38:28 -04:00
Joey Hess
bcaf320ed8 push out config changes 2012-10-21 15:50:49 -04:00
Joey Hess
452e6819d0 !! removal 2012-10-21 00:51:42 -04:00
Joey Hess
c7c2015435 add ConfigMonitor thread
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.

Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.

A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.

Also added a uuid.log cache. All other config files already had caches.
2012-10-20 16:43:35 -04:00
Joey Hess
73c28d2ca7 nasty race workaround 2012-10-19 16:59:18 -04:00
Joey Hess
f4c3a45da0 check preferred content when a file is moved or created
This can result in the file being dropped, or being downloaded, or even
being dropped from some other repo.

It's even possible to create a file in a directory where content is not
wanted, which will make the assistant immediately send it elsewhere, and
then drop it.
2012-10-19 14:22:13 -04:00
Joey Hess
9c89924c05 check and drop after uploads 2012-10-18 16:05:43 -04:00
Joey Hess
ee9e0702a2 check and drop unwanted content from remotes after receiving a transfer 2012-10-18 15:37:57 -04:00
Joey Hess
dea125e1b7 split 2012-10-18 15:22:28 -04:00
Joey Hess
f7f34d2072 drop unwanted content in the transfer scan
This was complicated quite a bit by needing to check numcopies. I optimised
that, so it only looks up numcopies once per file, no matter how many
remotes it checks to drop from. Although it did just occur to me that
it might be better to first check if it wants to drop content, and only
then check numcopies..
2012-10-18 15:07:11 -04:00
Joey Hess
dbe8de40ab minor transfer scanner code reworking
Also a small optimisation using a Set
2012-10-18 13:42:17 -04:00
Nicolas Pouillard
6b0a6cd226 import S3 only when enabled 2012-10-18 12:42:57 -04:00
Joey Hess
592d55ef7c fix display of transfers for remotes not in syncRemotes list 2012-10-14 14:51:11 -04:00
Joey Hess
6ea6a2e71c better variable name 2012-10-14 14:50:03 -04:00
Joey Hess
b2d266267f avoid queuing transfers for remotes after syncing to them is paused
This avoids the expensive transfer scan relying on its list of remotes
to scan being accurate throughout, which it will not be when the user
pauses syncing to a remote.

I feel it's ok to queue transfers to *any* known remote, not just the ones
being scanned.

Note that there are still small races where after syncing to a remote is
paused, a transfer can be queued for it. Not just in the expensive transfer
scan, but in the cheap failed transfer scan, and elsewhere.
2012-10-14 14:34:05 -04:00
Joey Hess
549c779a38 use gitRepo 2012-10-12 01:17:45 -04:00
Joey Hess
f7214e4de3 avoid pairlistener crash
I noticed this while offline (so that lack of solar power is good for something).

Apparently it tries to bind multicast to lo, and that fails.
If this happens, catch it, and retry until a real network interface becomes
available.

It may be that this should tie into the NetWatcher, and rebind whenever
an interface comes up. Needs testing..
2012-10-10 16:48:06 -04:00
Joey Hess
b6a3f03f82 add "configure" link to each repository in the webapp's repo list 2012-10-09 14:43:53 -04:00
Joey Hess
8eb1ba4cfe revert bad change 2012-10-09 13:49:27 -04:00
Joey Hess
5ac15149cc assistant: Now honors preferred content settings when deciding what to transfer.
Both when queueing downloads, and uploads, consults the preferred content
settings.

I didn't make it check yet when requeing failed transfers or queuing
deferred downloads; dealing with the preferred content settings (or indeed,
other settings) changing while the assistant is running still needs work.
2012-10-09 12:18:41 -04:00
Joey Hess
47314c0fad fix last zombies in the assistant
Made Git.LsFiles return cleanup actions, and everything waits on
processes now, except of course for Seek.
2012-10-04 19:56:32 -04:00
Joey Hess
9a3471971b avoid crashing committer if it fails to stage changes
Just retry later.
2012-10-02 18:04:06 -04:00
Joey Hess
9aab70de66 always check with ls-files before adding new files
Makes it safe to use git annex unlock with the watcher/assistant.
And also to mix use of the watcher/assistant with regular files stored in git.

Long ago, I had avoided doing this check, except during the startup scan,
because it would be slow to run ls-files repeatedly.

But then I added the lsof check, and to make that fast, got it to detect
batch file adds. So let's move the ls-files check to also occur when it'll
have a batch, and can check them all with one call.

This does slow down adding a single file by just a bit, but really only
a little bit. (The lsof check is probably more expensive.) It also
speeds up the startup scan, especially when there are lots of new files
found by the scan.

Also, fixed the sleep for annex.delayadd to not run while the threadstate
lock is held, so it doesn't unnecessarily freeze everything else.

Also, --force no longer makes it skip the lsof check, which was not
documented, and seems never a good idea.
2012-10-02 17:41:23 -04:00
Joey Hess
74a6e8ff60 avoid some warnings when built w/o dbus 2012-09-29 16:09:07 -04:00
Joey Hess
64514a3db3 close unreproducible bug and remove expensive code added to debug it 2012-09-28 12:56:58 -04:00
Joey Hess
17708dd173 add a configurator for S3 2012-09-26 14:44:07 -04:00
Joey Hess
96b0e3d6e9 quiet calls to transferkey 2012-09-24 13:36:05 -04:00
Joey Hess
364b40e5fc don't queue uploads for failed downloads 2012-09-24 13:16:08 -04:00
Joey Hess
2840105ec4 default annex.delayadd to 1 on OSX 2012-09-21 19:43:15 -04:00
Joey Hess
68ad7de4d0 watch for changes to transfer info files, to update progress bars on upload
This is handled differently for inotify, which can track modifications of
existing files, and kqueue, which cannot (TTBOMK). On the inotify side,
the TransferWatcher just waits for the file to be updated and reads the new
bytesComplete. On the kqueue side, the TransferPoller has to re-read the
file every update (currently 0.5 seconds, might need to increase that).

I did think about working around kqueue's limitations by somehow creating
a new file each time the size changed. But cleaning up all the files that
would result seemed difficult. And really, this is not a lot worse than
the TransferWatcher's behavior for downloads, which stats a file every 0.5
seconds. As long as the OS has decent file caching behavior..
2012-09-20 17:24:40 -04:00
Joey Hess
19e35f7f0d upload progress bar for git remote on same filesystem
cp is used here, but we can just watch the size of the destination file

This commit made from within the ruins of an old mill, overlooking a
beautiful waterfall.
2012-09-20 13:35:53 -04:00
Joey Hess
355ef8f3ea add annex.delayadd configuration 2012-09-19 13:30:25 -04:00
Joey Hess
d50d89eb6f support old versions of git that do not have --allow-empty-message 2012-09-19 12:58:53 -04:00
Joey Hess
c4e8591351 add missing --no-verify to prevent the pre-commit hook's git annex fix 2012-09-19 12:48:32 -04:00
Joey Hess
18bae020ed make other repositories list list all autostarted repos
And add a form to add another, unrelated repository
2012-09-18 17:50:07 -04:00
Joey Hess
ba27483c6a avoid making empty commits
This doesn't avoid it sometimes attempting to commit when there are no
changes. Typically that happens when a change is pushed in from another
repo; the watcher sees the file and tries to stage it, resulting in an
empty commit. Really fixing that would probably use more CPU than
occasionally trying to make an empty commit.

However, this does save a lot of unnecessary work, as those empty commits
had to be synced out, which no longer happens.
2012-09-18 14:43:56 -04:00
Joey Hess
9f05d19108 avoid sending uploads right back to where the download came from
Just an optimisation.
2012-09-18 14:24:51 -04:00