Commit graph

552 commits

Author SHA1 Message Date
Joey Hess
50a8ea4cdc avoid auto-accepting pair requests from friends already paired with
Unless the request is for repo uuid we already know. This way, if A1 pairs
with friend B1, and B1 pairs with device B2, then B1 can request A1 pair
with it and no confirmation is needed. (In future, may want to try to do
that automatically, to make a more robust network.)
2013-04-30 17:38:58 -04:00
Joey Hess
e363cefcb3 assistant: Fix bug that could cause incoming pushes to not get merged into the local tree.
Observed that the pushed refs were received, but not merged into master.
The merger never saw an add event for these refs. Either git is not writing
to a new file and renaming it into place, or the inotify code didn't notice
that. Changed it to also watch for modify events and that seems to have
fixed it!
2013-04-30 16:37:13 -04:00
Joey Hess
e06a4e12b5 more xmpp debugging 2013-04-30 15:56:33 -04:00
Joey Hess
2810807ca5 Internet Archive!
* Add public repository group.
* webapp: Can now set up Internet Archive repositories.

TODO: Enabling IA repositories.
2013-04-25 12:23:36 -04:00
Joey Hess
c6da464051 use a DList for the deferred downloads queue 2013-04-25 01:26:23 -04:00
Joey Hess
82a6db8fe8 committer tweak to wait for Watcher to resume after a max-size commit
Without this, a very large batch add has commits of sizes approx
5000, 2500, 1250, etc down to 10, and then starts over at 5000.
This fixes it so it's 5000+ every time.
2013-04-25 00:48:09 -04:00
Joey Hess
a9081ae473 optimise direct mode startup scan
A recent change made existing symlinks be re-staged. That does not need to
be done during the startup scan though.
2013-04-24 21:20:29 -04:00
Joey Hess
46529c0129 assistant: Sanitize XMPP presence information logged for debugging. 2013-04-24 21:13:10 -04:00
Joey Hess
ebee93a837 get rid of need to run pre-commit hook when assistant commits in direct mode
That hook updates associated file bookkeeping info for direct mode.
But, everything already called addAssociatedFile when adding/changing a
file. It only needed to also call removeAssociatedFile when deleting a file,
or a directory.

This should make bulk adds faster, by some possibly significant amount.
Bulk removals may be a little slower, since it has to use catKeyFile now
on each removed file, but will still be faster than adds.
2013-04-24 18:04:59 -04:00
Joey Hess
b8e45ec9d7 refactoring and minor performance tweak 2013-04-24 17:46:46 -04:00
Joey Hess
7fa2d255da remove last use of TSet 2013-04-24 17:16:04 -04:00
Joey Hess
cd7055631f batch commit every 5 thousand changes, not 10 thousand
There's a tradeoff between making less frequent commits, and
needing to use memory to store all the changes that are coming
in. At 10 thousand, it needs 150 mb of memory. 5 thousand drops
that down to 90 mb or so.

This also turns out to have significant imact on total run time.
I benchmarked 10k changes taking 27 minutes. But two 5k batches
took only 21 minutes.
2013-04-24 16:40:35 -04:00
Joey Hess
bda237f14a convert PendingAddChange back to Change when an add fails
If an add failed, we should lose the KeySource, since it, presumably,
differs due to a change that was made to the file.

(The locked down file is already deleted.)
2013-04-24 16:29:25 -04:00
Joey Hess
a929e6641a show one alert when bulk adding files
Turns out that a lot of the time spent in a bulk add was just updating the
add alert to rotate through each file that was added. Showing one alert
makes for a significant speedup.

Also, when the webapp is open, this makes it take quite a lot less cpu
during bulk adds.

Also, it lets the user know when a bulk add happened, which is sorta
nice..
2013-04-24 13:04:46 -04:00
Joey Hess
ca72b1ac7b assistant: when an add fails, requeue it for later
See analysis in bug report for one way this could happen.
2013-04-23 18:23:04 -04:00
Joey Hess
2c42e70f6c rename module 2013-04-23 11:38:52 -04:00
Joey Hess
8861e270be sync, assistant: Sync with remotes that have annex-ignore set
This is so git remotes on servers without git-annex installed can be used
to keep clients' git repos in sync.

This is a behavior change, but since annex-sync can be set to disable
syncing with a remote, I think it's acceptable.
2013-04-22 14:57:09 -04:00
Joey Hess
780ed8ebf1 fix warning 2013-04-20 18:52:46 -04:00
Joey Hess
090a69f00c assistant: Work around misfeature in git 1.8.2 that makes git commit --alow-empty -m "" run an editor.
See http://git-annex.branchable.com/bugs/assistant_hangs_during_commit/
2013-04-18 16:27:17 -04:00
Joey Hess
824c95376a refactor static route definition -- android webapp build fix
Incidentially should work around the last problem that prevented the webapp
building on Android. (Except for a few places I need to clean up after
hand-fixing the spliced TH code.)
2013-04-17 01:37:08 -04:00
Joey Hess
2c365b8b74 remove debug 2013-04-11 16:36:45 -04:00
Joey Hess
1eb3fff787 addurl: Register transfer so the webapp can see it.
* addurl: Register transfer so the webapp can see it.
* addurl: Automatically retry downloads that fail, as long as some
  additional content was downloaded.
2013-04-11 16:14:17 -04:00
Joey Hess
04a27ad926 assistant: Bug fix to avoid annexing the files that git uses to stand in for symlinks on FAT and other filesystem not supporting symlinks.
also, blog for the day..
2013-04-10 19:57:26 -04:00
Joey Hess
5e2e4347a3 webapp: New --listen= option allows running the webapp on one computer and connecting to it from another.
Does not yet use HTTPS. I'd need to generate a certificate, and I'm not
sure what's the best way to do that.
2013-04-08 15:04:35 -04:00
Joey Hess
602baae12e Bugfix: Direct mode no longer repeatedly checksums duplicated files.
Fixed by storing a list of cached inodes for a key, instead of just one.

Backwards compatability note: An old git-annex version will fail to parse
an inode cache file that has been written by a new version, and has
multiple items. It will succees if just one. So old git-annexes will have
even worse behavior when there are duplicated files, if that is possible.
I don't think it will be a problem. (Famous last words.)

Also, note that it doesn't expire old and unused inode caches for a key.
It would be possible to add this if needed; just look through the
associated files for a key and if there are more cached inodes, throw out
any not corresponding to associated files. Unless a file is being copied
repeatedly and the old copy deleted, this lack of expiry should not be a
problem.
2013-04-06 16:07:25 -04:00
Joey Hess
f1b0a4b404 Use lower case hash directories for storing files on crippled filesystems, same as is already done for bare repositories.
* since this is a crippled filesystem anyway, git-annex doesn't use
  symlinks on it
* so there's no reason to use the mixed case hash directories that we're
  stuck using to avoid breaking everyone's symlinks to the content
* so we can do what is already done for all bare repos, and make non-bare
  repos on crippled filesystems use the all-lower case hash directories
* which are, happily, all 3 letters long, so they cannot conflict with
  mixed case hash directories
* so I was able to 100% fix this and even resuming `git annex add` in the
  test case will recover and it will all just work.
2013-04-04 15:46:33 -04:00
Joey Hess
8b329c0317 refactor alert button creation code 2013-04-04 01:48:26 -04:00
Joey Hess
2c3aeec19b check for unused keys on an unwanted remote, and move them off, before deleting it 2013-04-03 19:03:16 -04:00
Joey Hess
021c564319 clean up urlrenderer handling when the webapp is not built 2013-04-03 17:48:54 -04:00
Joey Hess
9a5f421768 detect when unwanted remote is empty and remove it
Needs fixes to build when the webapp is disabled.
2013-04-03 17:01:40 -04:00
Joey Hess
47950cdf31 more efficient uuid to remote lookup 2013-04-02 16:39:11 -04:00
Joey Hess
6e7842475b convert "./file" from inotify to just "file"
This just prettifies some display.
2013-04-02 16:20:23 -04:00
Joey Hess
38d61f934d Update working tree files fully atomically
This avoids commit churn by the assistant when eg,
replacing a file with a symlink.

But, just as importantly, it prevents the working tree being left with a
deleted file if git-annex, or perhaps the whole system, crashes at the
wrong time.

(It also probably avoids confusing displays in file managers.)
2013-04-02 15:02:00 -04:00
Joey Hess
8c52b20cc7 optimise last commit
Rather than re-adding a direct mode file unnecessarily when it's not
changed, just re-stage the symlink.
2013-04-02 12:58:56 -04:00
Joey Hess
31cbde8190 assistant: Fix bug that could cause direct mode files to be unstaged from git.
My test case for this bug is to have the assistant running and syncing to
a remote, and create a file in the annex. Then at the command line run
git annex drop. The assistant sees that the file is gone, sees it's a wanted
file, and downloads it from the remote.

With a directory special remote and a small file, I was seeing around 1
time in 3, a race where the file got unstaged from git after it got
downloaded.

Looking at what direct mode content managing code does in this case, it
deletes the symlink, and then adds the file content back. It would be
possible, sometimes, to avoid removing the symlink and do this atomically.
And I probably should.. but in some cases, particularly where the file
needs to be run through `cp` (multiple direct mode files with same
content), there's no way to atomically replace the symlink with the
content.

Anyway, the bug turns out to be something that the watcher does right for
indirect mode, but not for direct mode. When it got an add event, it
checked to see if this was a new file, or one we've already added. In the
latter case, no add event was queued. But that means that only the rm event
is queued, and so it unstages the file.

Fixed by queueing an add event even when the file is already in git.

Tested by running hundreds of drops in a loop; file remained staged.
2013-04-02 12:45:31 -04:00
Joey Hess
c57baaaa30 webapp: Added UI to delete repositories. Closes: #689847 2013-03-31 16:38:05 -04:00
Joey Hess
5771cfce02 assistant: Check small files into git directly. 2013-03-29 16:54:59 -04:00
Joey Hess
67e817c6a1 New annex.largefiles setting, which configures which files git annex add and the assistant add to the annex.
I would have sort of liked to put this in .gitattributes, but it seems
it does not support multi-word attribute values. Also, making this a single
config setting makes it easy to only parse the expression once.

A natural next step would be to make the assistant `git add` files that
are not annex.largefiles. OTOH, I don't think `git annex add` should
`git add` such files, because git-annex command line tools are
not in the business of wrapping git command line tools.
2013-03-29 16:17:13 -04:00
Joey Hess
577128e9b8 avoid removing a transfer being made by another process
When another process is running a transfer, if we try to run a dup it'll
fail, and we should not remove the transfer from the transfer display.
2013-03-28 15:16:45 -04:00
Joey Hess
1d0b692198 webapp: Fix a race that sometimes caused alerts or other notifications to be missed if they occurred while a page was loading.
When a page is loaded, the javascript requests an notification url, and
does long polling on the url to be informed of changes. But if a change
occured before the notification url was requested, it would not be notified
of that change, and so the page display would not update.

I fixed this by *always* updating the page display after it gets
the notification url. This is extra work, but the overhead is not noticable
in the other overhead of loading a page.

(A nicer way would be to somehow record the version of a page initially
loaded, and then compare it with the current version when getting the
notification url, and only force an update if it's changed. But getting
the "version" of the different parts of the page that use long polling
is difficult.)
2013-03-27 14:56:20 -04:00
Joey Hess
3ef6b6a200 fix build with xmpp and w/o webapp 2013-03-24 18:55:19 -04:00
Joey Hess
b6d691aff7 maintain pools of running transferkeys processes (untested) 2013-03-19 18:46:29 -04:00
Joey Hess
bb284cd980 move display of transfer scan in progress to transfers section of dashboard
This way it's only visible when transfers are not running, which is about
what a user would expect.
2013-03-19 13:03:41 -04:00
Joey Hess
a30768cf7f new alert while scanning
Like the old one, but does not mention which remotes are scanned.
I think this is less confusing, as it does not imply the remotes
were somehow accessed (which they are not; inaccessible remotes
can be scanned.)
2013-03-18 23:15:48 -04:00
Joey Hess
aadb9069b3 deal with transferkey crashing
If transferkey crashes or even fails to run, the TransferWatcher will not
see the transfer info file be created, so will not remove the transfer
from the list of active transfers. This causes the list to grow
continually, and all active transfers are displayed in the webapp. So, put
in a guard.

I assume that transferkey will not exit 0 while neglecting to clean up.
2013-03-18 22:58:14 -04:00
Joey Hess
29d603b72a ensure 2013-03-18 22:41:16 -04:00
Joey Hess
b543842a7f optimisation for transfers to drives that are not plugged in
Rather than forking a git-annex transferkey only to have it fail,
just immediately record the failed transfer (so when the drive is plugged
in, the scan will retry it).
2013-03-18 20:40:24 -04:00
Joey Hess
cdb21649d0 webapp: Improved alerts displayed when syncing with remotes, and when syncing with a remote fails. 2013-03-18 17:23:47 -04:00
Joey Hess
35a0ae334c assistant: Fix OSX bug that prevented committing changed files to a repository when in indirect mode. 2013-03-17 17:01:43 -04:00
Joey Hess
746ffa773a assistant: Avoid syncing with annex-ignored remotes when reconnecting to the network, or connecting a drive. 2013-03-17 15:59:03 -04:00
Joey Hess
db2fe522ba xmpp: Re-enable XA flag, since disabling it did not turn out to help with the problems Google Talk has with not always sending presence messages to clients. 2013-03-16 16:00:37 -04:00
Joey Hess
55f20ae099 xmpp: send a presence query when there's an important message to send
This may work around google talk's horrible presence handling, in which
clients often don't learn about other clients, at least when using the same
account. This way, every time we start a git push over xmpp, we'll waste
bandwidth asking clients to please try again to identify themselves.
2013-03-16 15:36:47 -04:00
Joey Hess
e3354cf19c xmpp: --debug now enables a sanitized dump of the XMPP protocol
So I can debug these damn google talk presence issues.
2013-03-16 15:29:51 -04:00
Joey Hess
c94c99942b make liftAnnex and liftAssistant polymorphic, like liftIO 2013-03-16 00:19:56 -04:00
Joey Hess
77c82de4ea webapp: Display an alert when there are XMPP remotes, and a cloud transfer repository needs to be configured. 2013-03-15 17:52:41 -04:00
Joey Hess
39e979fb65 webapp: Improved UI for pairing your own devices together using XMPP. 2013-03-15 15:19:07 -04:00
Joey Hess
9cf4701a8f no longer need webapp state storage! excellent 2013-03-15 01:01:25 -04:00
Joey Hess
02facde154 assistant: Be smarter about avoiding unncessary transfers.
Just before starting a transfer, do one last check that it's still
preferred content.

I was just doing this for uploads, as part of the smarter flood filling
bug, but realized it's also possible for a download that was preferred
content to change to not be before the download begins, so check that too.
2013-03-13 13:36:02 -04:00
Joey Hess
60760cb430 tweak 2013-03-13 13:11:49 -04:00
Joey Hess
0ef8d806ac gratuitous rename HomeR -> DashboardR 2013-03-12 22:18:36 -04:00
Joey Hess
8221c2b4ed split repolist out of configuration, into its own tab (temporarily) 2013-03-12 21:51:03 -04:00
Joey Hess
393340dc3b better handling of batch renames
Rather than wait a full second, which may be longer than needed, or too
short to get all the rename events, we start a mode where we wait 1/10th of
a second, and if there are Changes received, wait again. Basically we're
back in batch mode when this happens.
2013-03-11 15:46:09 -04:00
Joey Hess
14fcfced48 detect directory rename and wait up to 1 second to get all the changes 2013-03-11 15:24:13 -04:00
Joey Hess
f340fd324c synthesize RmChange when a directory is deleted
This gets directory renames closer to being fully detected.
There's close to no extra overhead to doing it this way.
2013-03-11 15:14:42 -04:00
Joey Hess
06046a0d2b finish fast direct mode rename handling. wow, it's fast 2013-03-11 14:14:45 -04:00
Joey Hess
87cba71d5a fix changeFile to not be partial
That led to runtime crashes, without even a warning from -Wall. Yipes!
2013-03-11 13:55:36 -04:00
Joey Hess
61c5e8736c detect renames during commit, and .. um, do nothing special because it's lunch time
But I'm well set up to fast-track direct mode adds for renames now.
2013-03-11 12:56:47 -04:00
Joey Hess
74f723bb50 let's put type modules under the parent module, not in a Types directory 2013-03-10 22:24:13 -04:00
Joey Hess
2762ab03b4 assistant: generate better commits for renames 2013-03-10 22:10:26 -04:00
Joey Hess
b2c7ee5551 tweak 2013-03-10 20:20:58 -04:00
Joey Hess
f27c21eb0c avoid ugly alert caused by trying to push to unavailable removable drive 2013-03-10 18:42:28 -04:00
Joey Hess
65a4c7966f moved transfer queueing out of watcher and into committer
This cleaned up the code quite a bit; now the committer just looks at the
Change to see if it's a change that needs to have a transfer queued for it.
If I later want to add dropping keys for files that were removed, or
something like that, this should make it straightforward.

This also fixes a bug. In direct mode, moving a file out of an archive
directory failed to start a transfer to get its content. The problem
was that the file had not been committed to git yet, and so the transfer
code didn't want to touch it, since fileKey failed to get its key.
Only starting transfers after a commit avoids this problem.
2013-03-10 18:16:03 -04:00
Joey Hess
0e508f860a assistant: Sync with all git remotes on startup. 2013-03-08 13:48:27 -04:00
Joey Hess
243e99717d empty buddy list when client is connecting
This is not perfect, because on loss of connection, we do not currently
immediately detect it and stop the client. It has to time out, and then
the buddy list will clear.

The NetWatcher should detect disconnects too..
2013-03-07 03:50:21 -04:00
Joey Hess
f8c2dc82d8 show when not connected to xmpp server 2013-03-06 22:02:47 -04:00
Joey Hess
c16adc25c4 assistant: XMPP git pull and push requests are cached and sent when presence of a new client is detected.
Noticed that, At startup or network reconnect, git push messages were sent,
often before presence info has been gathered, so were not sent to any
buddies.

To fix this, keep track of which buddies have seen such messages,
and when new presence is received from a buddy that has not yet seen it,
resend.

This is done only for push initiation messages, so very little data needs
to be stored.
2013-03-06 21:38:01 -04:00
Joey Hess
060119fdc4 better xmpp debugging 2013-03-06 18:28:34 -04:00
Joey Hess
aaec2cbf03 avoid false alert about syncing with xmpp remote 2013-03-06 17:54:45 -04:00
Joey Hess
cbb6e1fae4 tag xmpp pushes with jid
This fixes the issue mentioned in the last commit.

Turns out just collecting UUID of clients behind a XMPP remote is
insufficient (although I should probably still do it for other reasons),
because a single remote repo might be connected via both XMPP and local
pairing. So a way is needed to know when a push was received from any
client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
Joey Hess
c23ea9e311 assistant: Get back in sync with XMPP remotes after network reconnection, and on startup.
Make manualPull send push requests over XMPP.

When reconnecting with remotes, those that are XMPP remotes cannot
immediately be pulled from and scanned, so instead maintain a set of
(probably) desynced remotes, and put XMPP remotes on it. (This set could be
used in other ways later, if we can detect we're out of sync with other
types of remotes.)

The merger handles detecting when a XMPP push is received from a desynced
remote, and triggers a scan then, if they have in fact diverged.

This has one known bug: A single XMPP remote can have multiple clients
behind it. When this happens, only the UUID of one client is recorded
as the UUID of the XMPP remote. Pushes from the other XMPP clients will not
trigger a scan. If the client whose UUID is expected responds to the push
request, it'll work, but when that client is offline, we're SOL.
2013-03-06 15:09:31 -04:00
Joey Hess
907b0c0d78 better liftAnnex, avoid using runAnnex undefined 2013-03-04 16:36:38 -04:00
Joey Hess
c908672f3d fix another potential race with the watcher and direct mode
Watcher wants to rewrite symlink to fix it. But in direct mode, the symlink
could be replaced at any time with file content that has finished being
transferred by some other process. So, just don't touch it.

FWIW, I audited the rest of the assistant for places where it removes
files, and the rest is ok. I have not audited the rest of git-annex.
2013-03-04 15:09:32 -04:00
Joey Hess
1d388d5579 fixed the race breaking moving files from archive in direct mode
assistant: Fix bug in direct mode that could occur when a symlink is moved
out of an archive directory, and resulted in the file not being set to
direct mode when it was transferred.

The bug was that the direct mode mapping was not up-to-date when the
transferrer finished. So, finding no direct mode place to store the object,
it was put into .git/annex in indirect mode.

To fix this, just make the watcher update the direct mode mapping to
include the new file before it starts the transfer. (Seems we don't need to
update it to remove the old file if the link was moved, because the direct
mode code will notice it's not present and the mapping gets updated for its
removal later.)

The reason this was a race, and was probably not seen often is because
the committer came along and updated the direct mode mapping as part of
adding the moved symlink. But when the file was sufficiently small or
the remote sufficiently fast, this could happen after the transfer
finished.
2013-03-04 14:25:22 -04:00
Joey Hess
08bdea7e52 webapp: New preferences page allows enabling/disabling debug logging at runtime, as well as configuring numcopies and diskreserve. 2013-03-03 17:07:27 -04:00
Joey Hess
724711e4b7 fix 2013-03-03 15:18:24 -04:00
Joey Hess
789ca15012 better prevention of auto repack
Looking through the git sources (documentation is unclear),
it seems commit doesn't ever trigger git-gc, mostly fetching and merging
seems to. I cannot easily override the setting in all those places, so
instead set gc.auto in git config when initializing a repository with
the assistant.

This does mean that the user cannot set gc.auto=0 and completely avoid
repacks, as the assistant does it daily. But, it only does it after there
are 100x the default number of loose objects, so this is probably not going
to be too annoying.
2013-03-03 14:20:07 -04:00
Joey Hess
6dea43831e assistant: Prevent automatic commits from causing git-gc runs, as that can make things quite slow. Instead, git-gc --auto is run once a day. (This can be disabled by the usual gc.auto=0 setting.) 2013-03-03 13:44:35 -04:00
Joey Hess
0c13d3065e git subcommand cleanup
Pass subcommand as a regular param, which allows passing git parameters
like -c before it. This was already done in the pipeing set of functions,
but not the command running set.
2013-03-03 13:39:07 -04:00
Joey Hess
7a8833c81d remove excess log rotation; openLog rotates 2013-03-01 16:56:56 -04:00
Joey Hess
0d4b513ec2 assistant: Fix dropping content when a file is moved to an archive directory.
A transfer is queued, but if the file has already been transferred to the
remote before, the transfer is skipped. In this case, it needs to perform
any actions it would normally take after finishing the transfer, like
dropping the local object.
2013-03-01 16:46:36 -04:00
Joey Hess
4d33423067 assistant: Avoid noise in logs from git commit about typechanged files in direct mode repositories. 2013-03-01 16:21:29 -04:00
Joey Hess
a733271a9c add additional debug info about reasons for drops 2013-03-01 15:58:44 -04:00
Joey Hess
46c9cbeb1e add additional debug info about reasons for transfers 2013-03-01 15:23:59 -04:00
Joey Hess
1865b28094 assistant: Logs are rotated to avoid them using too much disk space.
This cannot completely guard against a runaway log event, and only runs
every hour anyway, but it should avoid most problems with very
long-running, active assistants using up too much space.
2013-03-01 13:30:48 -04:00
Joey Hess
08854afa10 fix inverted logic 2013-02-22 17:01:48 -04:00
Joey Hess
2a4dad8bd4 remove debug prints 2013-02-19 23:18:15 -04:00
Joey Hess
d7c93b8913 fully support core.symlinks=false in all relevant symlink handling code
Refactored annex link code into nice clean new library.

Audited and dealt with calls to createSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:  ( liftIO $ createSymbolicLink linktarget file
  only when core.symlinks=true
Assistant/WebApp/Configurators/Local.hs:                createSymbolicLink link link
  test if symlinks can be made
Command/Fix.hs: liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/FromKey.hs:     liftIO $ createSymbolicLink link file
  command only works in indirect mode
Command/Indirect.hs:                    liftIO $ createSymbolicLink l f
  refuses to run if core.symlinks=false
Init.hs:                createSymbolicLink f f2
  test if symlinks can be made
Remote/Directory.hs:    go [file] = catchBoolIO $ createSymbolicLink file f >> return True
  fast key linking; catches failure to make symlink and falls back to copy
Remote/Git.hs:          liftIO $ catchBoolIO $ createSymbolicLink loc file >> return True
  ditto
Upgrade/V1.hs:                          liftIO $ createSymbolicLink link f
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to readSymbolicLink.
Remaining calls are all safe, because:

Annex/Link.hs:		( liftIO $ catchMaybeIO $ readSymbolicLink file
  only when core.symlinks=true
Assistant/Threads/Watcher.hs:		ifM ((==) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file))
  code that fixes real symlinks when inotify sees them
  It's ok to not fix psdueo-symlinks.
Assistant/Threads/Watcher.hs:		mlink <- liftIO (catchMaybeIO $ readSymbolicLink file)
  ditto
Command/Fix.hs:	stopUnless ((/=) (Just link) <$> liftIO (catchMaybeIO $ readSymbolicLink file)) $ do
  command only works in indirect mode
Upgrade/V1.hs:	getsymlink = takeFileName <$> readSymbolicLink file
  v1 repos could not be on a filesystem w/o symlinks

Audited and dealt with calls to isSymbolicLink.
(Typically used with getSymbolicLinkStatus, but that is just used because
getFileStatus is not as robust; it also works on pseudolinks.)
Remaining calls are all safe, because:

Assistant/Threads/SanityChecker.hs:                             | isSymbolicLink s -> addsymlink file ms
  only handles staging of symlinks that were somehow not staged
  (might need to be updated to support pseudolinks, but this is
  only a belt-and-suspenders check anyway, and I've never seen the code run)
Command/Add.hs:         if isSymbolicLink s || not (isRegularFile s)
  avoids adding symlinks to the annex, so not relevant
Command/Indirect.hs:                            | isSymbolicLink s -> void $ flip whenAnnexed f $
  only allowed on systems that support symlinks
Command/Indirect.hs:            whenM (liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f) $ do
  ditto
Seek.hs:notSymlink f = liftIO $ not . isSymbolicLink <$> getSymbolicLinkStatus f
  used to find unlocked files, only relevant in indirect mode
Utility/FSEvents.hs:                    | Files.isSymbolicLink s = runhook addSymlinkHook $ Just s
Utility/FSEvents.hs:                                            | Files.isSymbolicLink s ->
Utility/INotify.hs:                             | Files.isSymbolicLink s ->
Utility/INotify.hs:                     checkfiletype Files.isSymbolicLink addSymlinkHook f
Utility/Kqueue.hs:              | Files.isSymbolicLink s = callhook addSymlinkHook (Just s) change
  all above are lower-level, not relevant

Audited and dealt with calls to isSymLink.
Remaining calls are all safe, because:

Annex/Direct.hs:			| isSymLink (getmode item) =
  This is looking at git diff-tree objects, not files on disk
Command/Unused.hs:		| isSymLink (LsTree.mode l) = do
  This is looking at git ls-tree, not file on disk
Utility/FileMode.hs:isSymLink :: FileMode -> Bool
Utility/FileMode.hs:isSymLink = checkMode symbolicLinkMode
  low-level

Done!!
2013-02-17 16:43:14 -04:00
Joey Hess
630f4531a7 fix assistant's use of lsof in crippled filesystem mode 2013-02-15 13:08:22 -04:00
Joey Hess
47477b2807 crippled filesystem support, probing and initial support
git annex init probes for crippled filesystems, and sets direct mode, as
well as `annex.crippledfilesystem`.

Avoid manipulating permissions of files on crippled filesystems.
That would likely cause an exception to be thrown.

Very basic support in Command.Add for cripped filesystems; avoids the lock
down entirely since doing it needs both permissions and hard links.
Will make this better soon.
2013-02-14 14:15:26 -04:00
Joey Hess
5737c49804 support Android's crippled lsof 2013-02-11 17:33:45 -04:00
Joey Hess
547d7745fb pre-commit: Update direct mode mappings.
Making the pre-commit hook look at git diff-index to find changed direct
mode files and update the mappings works pretty well.

One case where it does not work is when a file is git annex added, and then
git rmed, and then this is committed. That's a no-op commit, so the hook
probably doesn't even run, and it certianly never notices that the file
was deleted, so the mapping will still have the original filename in it.

For this and other reasons, it's important that the mappings still be
treated as possibly inconsistent.

Also, the assistant now allows the pre-commit hook to run when in direct
mode, so the mappings also get updated there.
2013-02-06 12:44:19 -04:00
Joey Hess
b19c2e6122 assistant: Fix location log when adding new file in direct mode. 2013-02-05 13:41:48 -04:00
Joey Hess
a261412c25 close 2013-01-28 15:39:51 +11:00
Joey Hess
a8bb2749b2 assistant: Ignore .DS_Store on OSX. 2013-01-28 15:13:22 +11:00
Joey Hess
5cd152b8a9 annex.autocommit
New setting, can be used to disable autocommit of changed files by the
assistant, while it still does data syncing and other tasks.

Also wired into webapp UI
2013-01-27 22:43:05 +11:00
Joey Hess
76ddf9b6d3 webapp: Now allows restarting any threads that crash. 2013-01-26 17:09:33 +11:00
Joey Hess
1713ed95f7 use async to track and manage threads 2013-01-26 14:14:32 +11:00
Joey Hess
d7ca6fb856 webapp: Now always logs to .git/annex/daemon.log
It used to not log to daemon.log when a repository was first created, and
when starting the webapp. Now both do. Redirecting stdout and stderr to the
log is tricky when starting the webapp, because the web browser may want to
communicate with the user. (Either a console web browser, or web.browser = echo)
This is handled by restoring the original fds when running the browser.
2013-01-15 13:34:59 -04:00
Joey Hess
f51ad2a00c assistant: Avoid committer crashing if a file is deleted at the wrong instant. 2013-01-14 15:02:13 -04:00
Joey Hess
fb9ab8b2b5 catch failure to start dbus service 2013-01-14 13:22:19 -04:00
Joey Hess
6ec2802228 fix 2013-01-10 16:43:22 -04:00
Joey Hess
f99563ad14 fix fix to names 2013-01-10 16:18:26 -04:00
Joey Hess
7e6fbd3507 seems I got the name wrong 2013-01-10 16:08:40 -04:00
Joey Hess
d22d06a84a assistant: Support new gvfs dbus names used in Gnome 3.6. (untested) 2013-01-10 15:06:08 -04:00
Joey Hess
6f7ae84650 webapp: Use IP address, rather than localhost
since some systems may have configuration problems or other issues that
prevent web browsers from connecting to the right localhost IP for the
webapp.

Tested on both ipv4 and ipv6 localhost. Url for the latter looks like:
http://[::1]:50676
2013-01-09 23:18:00 -04:00
Joey Hess
aedfcde969 guard readSymbolicLink
throws an exception if the file is not a symlink
2013-01-05 16:07:27 -04:00
Joey Hess
1cdf2b923d assistant: Make expensive transfer scan work fully in direct mode.
The expensive scan uses lookupFile, but in direct mode, that doesn't work
for files that are present. So the scan was not finding things that are
present that need to be uploaded. (It did find things not present that
needed to be downloaded.)

Now lookupFile also works in direct mode. Note that it still prefers
symlinks on disk to info committed to git, in direct mode. This is
necessary to make things like Assistant.Threads.Watcher.onAddSymlink
work correctly, when given a new symlink not yet checked into git (or
replacing a file checked into git).
2013-01-05 15:57:53 -04:00
Joey Hess
bad9b6761d restart UI
Browser behavior is not ideal; a new tab is opened on restart.
Browsers won't let me redirect to a file:// so I cannot use the old tab.
2013-01-03 18:50:30 -04:00
Joey Hess
de2e287133 webapp: Add UI to stop assistant.
Would like to also have restart UI, but that's rather harder to do,
seems it'd need to start another copy of the webapp, and redirect the
browser to its new url, but running two assistants in the same repo at
the same time isn't good.
2013-01-03 15:24:21 -04:00
Joey Hess
4008590c68 type based git config handling for remotes
Still a couple of places that use git config ad-hoc, but this is most of it
done.
2013-01-01 13:58:14 -04:00
Joey Hess
7f7c31df1c type based git config handling
Now there's a Config type, that's extracted from the git config at startup.
Note that laziness means that individual config values are only looked up
and parsed on demand, and so we get implicit memoization for all of them.
So this is not only prettier and more type safe, it optimises several
places that didn't have explicit memoization before. As well as getting rid
of the ugly explicit memoization code.

Not yet done for annex.<remote>.* configuration settings.
2012-12-29 23:10:18 -04:00
Joey Hess
8cc27b8afc avoid double commits with inotify when direct mode file is created 2012-12-29 14:58:13 -04:00
Joey Hess
c0f9810f0b OSX assistant: Uses direct mode by default when setting up a new local repository. 2012-12-28 16:42:11 -04:00
Joey Hess
bf3270c5b7 add missing modifyHook for watcher
Needed for FSEvents, which calls that hook for modified files.
inotify seems to call the add hook, so I didn't notice it before.
2012-12-28 16:00:45 -04:00
Joey Hess
eb40227d15 assistant direct mode file add/change bookkeeping
When a file is changed in direct mode, the old content is probably lost
(at least from the local repo), and bookeeping needs to be updated to
reflect this.

Also, synthetic add events are generated at assistant startup, so
make it detect when the file has not really changed, and avoid re-adding
it.

This does add the overhead of querying the runing git cat-file for the
key that's recorded in git for the file, each time a file is added or
modified in direct mode.
2012-12-25 15:48:15 -04:00
Joey Hess
8a8380f1b7 use sync command merge engine in assistant
To handle direct mode merging.
2012-12-25 14:10:07 -04:00
Joey Hess
cc5140d295 assistant adding of modified files in direct mode
Works with inotify, but I think in kqueue we don't get events
existing files that get modified.
2012-12-24 14:42:19 -04:00
Joey Hess
95db595e91 make startup scan for deleted files work in direct mode
git add --update cannot be used, because it'll stage typechanged direct
mode files. Intead, use ls-files to find deleted files, and stage them
ourselves.

It seems that no commit was made before when the scan staged deleted files.
(Probably masked since if files were added, a commit happened then..)
Now that I'm doing the staging, I was also able to fix that bug.
2012-12-24 14:24:13 -04:00
Joey Hess
c6d2bbe402 assistant adding of files in direct mode 2012-12-24 13:37:29 -04:00
Joey Hess
82617b92e9 move thirdparty program installation for standalone bundle into haskell program
This allows it to use Build.SysConfig to always install the programs
configure detected. Amoung other fixes, this ensures the right uuid
generator and checksum programs are installed.

I also cleaned up the handling of lsof's path; configure now checks for
it in PATH, but falls back to looking for it in sbin directories.
2012-12-14 16:07:59 -04:00
Joey Hess
0d50a6105b whitespace fixes 2012-12-13 00:45:27 -04:00
Joey Hess
99a8a5297c --auto fixes
* get/copy --auto: Transfer data even if it would exceed numcopies,
  when preferred content settings want it.
* drop --auto: Fix dropping content when there are no preferred content
  settings.
2012-12-06 13:22:16 -04:00
Joey Hess
5de0215986 remove dead code 2012-12-01 15:21:24 -04:00
Joey Hess
2f50af5273 better function name 2012-12-01 15:00:03 -04:00
Joey Hess
d2df2e52b4 remove hard link when sanity check failed
See http://git-annex.branchable.com/forum/dot_git_slash_annex_slash_tmp/
2012-11-29 16:54:51 -04:00
Joey Hess
3b35cde0e8 assistant: Retrival from glacier now handled. 2012-11-29 15:23:33 -04:00
Joey Hess
18fe34222a allow building webapp w/o webdav 2012-11-25 14:36:24 -04:00
Joey Hess
463cf58140 webapp and assistant glacier support 2012-11-24 16:30:15 -04:00
Joey Hess
c282c8b492 queue uploads when a new or renamed symlink is handled 2012-11-24 15:38:24 -04:00
Joey Hess
7addb89dc1 webapp: support box.com 2012-11-17 15:30:11 -04:00
Joey Hess
d468e37f46 automatic repolist updating 2012-11-13 17:50:54 -04:00
Joey Hess
8b22bd8831 add a yes/no confirm for xmpp pairing requests 2012-11-11 22:29:16 -04:00
Joey Hess
522e228aef auto-accept pair requests from JIDs already paired with 2012-11-11 18:16:11 -04:00
Joey Hess
5e44ab177c don't try to transfer data to/from XMPP remotes
Partition syncRemotes into ones needing git sync (ie, non-special remotes),
and ones needing data sync (ie, non-XMPP remotes).
2012-11-11 16:23:16 -04:00
Joey Hess
217eeede43 allow both one push and one receive-pack to run at the same time
Noticed that when pairing, sometimes both sides start to push, and the other
side sends a PushRequest, and the two deadlock, neither doing anything.
(Timeout eventually breaks this.) So, let both run at the same time.
2012-11-11 15:42:03 -04:00
Joey Hess
41085cdc8c set negative XMPP presence priority
This should help prevent git-annex clients receiving messages that
were intended for normal clients they're sharing the account with.

Changed XMPP protocol use to always send chat messages directed at the
specific client, as the negative priority blocks less directed messages.
2012-11-10 15:33:12 -04:00
Joey Hess
7a20b3f1d5 more refactoring 2012-11-10 14:01:24 -04:00
Joey Hess
0cf4c3ba9c more nice refactoring 2012-11-10 13:00:13 -04:00
Joey Hess
81953c2131 separate data type for push stages
This improves type safety.
2012-11-10 12:18:00 -04:00
Joey Hess
8b8964b523 refactor 2012-11-10 02:35:54 -04:00