Commit graph

1718 commits

Author SHA1 Message Date
Joey Hess
ec28151722
improve data type 2016-01-01 15:56:24 -04:00
Joey Hess
7c5c7bb04a
fix OSX build 2015-12-28 13:28:21 -04:00
Joey Hess
c4152654d2
combine PendingAddChanges for the same file into one
In v6 unlocked mode, this fixes a problem that was making eg,
echo > file cause the assistant to copy the file to the annex object,
instead of hard linking it. That because 2 change events were seen
(one for opening the file and one for closing) and processed together
the file was then locked down twice. Which meant it had mutiple hard links,
and so prevented linkAnnex from hard linking it.

There might be scenarios where multiple events come in, but staggered such
that a file gets locked down repeatedly, and it would still be copied to
the annex object in that case.
2015-12-22 17:52:39 -04:00
Joey Hess
cfaac52b88
populate unlocked files with newly available content when ingesting
This can happen when ingesting a new file in either locked or unlocked
mode, when some unlocked files in the repo use the same key, and the
content was not locally available before.
2015-12-22 16:22:28 -04:00
Joey Hess
4f60234690
finish v6 support for assistant
Seems to basically work now!
2015-12-22 15:23:27 -04:00
Joey Hess
8e9608d7f0
refactoring
no behavior changes
2015-12-22 13:42:58 -04:00
Joey Hess
ca2c977704
wip v6 support for assistant
Files are not yet added to v6 repos in unlocked mode.
2015-12-21 18:41:15 -04:00
Joey Hess
cdd27b8920
reorg 2015-12-15 15:34:28 -04:00
Joey Hess
3311c48631
move InodeSentinal from direct mode code to its own module
Will be used outside of direct mode for v6 unlocked files, and is already
used outside of direct mode when adding files to annex.
2015-12-09 15:52:11 -04:00
Joey Hess
a0fcb8ec93
generalize catchHardwareFault to catchIOErrorType 2015-12-06 16:26:38 -04:00
Joey Hess
e9f43c07f9
webapp: Fix bugs that could result in a relative path such as "." being written to ~/.config/git-annex/autostart
and ignore any such  relative paths in the file

This was a reversion caused by the relative path changes in 5.20150113.
2015-12-02 15:57:30 -04:00
Joey Hess
7c741302cc
assistant: Pass ssh-options through 3 more git pull/push calls that were missed before.
It was used for regular pull, but not for regular push, tagged push, or the
fallback fetching.
2015-11-10 16:52:30 -04:00
Joey Hess
b0e5c09408
fix various build warnings, mostly on Windows
And some when S3 is disabled
2015-10-13 13:24:44 -04:00
Joey Hess
6a72045707
fix local dropping to not require extra locking of copies, but only that the local copy be locked for removal 2015-10-09 15:48:02 -04:00
Joey Hess
c75c79864d
support invalidating existing VerifiedCopys 2015-10-08 17:58:32 -04:00
Joey Hess
90f7c4b6a2
add VerifiedCopy data type
There should be no behavior changes in this commit, it just adds a more
expressive data type and adjusts code that had been passing around a [UUID]
or sometimes a Maybe Remote to instead use [VerifiedCopy].

Although, since some functions were taking two different [UUID] lists,
there's some potential for me to have gotten it horribly wrong.
2015-10-08 16:55:11 -04:00
Joey Hess
4d50958ed7
add lockContentShared
Also, rename lockContent to lockContentExclusive

inAnnexSafe should perhaps be eliminated, and instead use
`lockContentShared inAnnex`. However, I'm waiting on that, as there are
only 2 call sites for inAnnexSafe and it's fiddly.
2015-10-08 14:29:35 -04:00
Joey Hess
933fef6ae0 Merge branch 'winprocfix' 2015-10-04 15:46:25 -04:00
Joey Hess
16d7035ef7 Allow building with S3 disabled again. 2015-10-02 12:12:02 -04:00
Joey Hess
cad3349001 rename fsckKey to verifyKeyContent
No behavior changes.
2015-10-01 13:29:17 -04:00
Joey Hess
f33e760aab Merge branch 's3-classes' 2015-09-22 11:05:53 -04:00
Joey Hess
20205b6073 avoid hard dependency on new version of aws 2015-09-22 11:04:26 -04:00
Joey Hess
3058e11520 webapp: Remove the "disable remote" feature from the UI.
It's confusing, easy to select by accident and get into a situation the
webapp offers no easy recovery from, and pausing syncing works just as well
in most situations.
2015-09-21 13:05:44 -04:00
Joey Hess
26d6566307 S3 storage classes expansion
Added support for storageclass=STANDARD_IA to use Amazon's
new Infrequently Accessed storage.

Also allows using storageclass=NEARLINE to use Google's NearLine storage.

The necessary changes to aws to support this are in
https://github.com/aristidb/aws/pull/176
2015-09-17 17:20:01 -04:00
Joey Hess
9cfb96c53d Special remotes configured with autoenable=true will be automatically enabled when git-annex init is run. 2015-09-14 14:49:48 -04:00
Joey Hess
0390efae8c support gpg.program
When gpg.program is configured, it's used to get the command to run for
gpg. Useful on systems that have only a gpg2 command or want to use it
instead of the gpg command.
2015-09-09 18:06:49 -04:00
Joey Hess
19dbe2a611 webapp: Fix support for entering password when setting up a ssh remote. 2015-09-03 11:03:08 -07:00
Michael Alan Dorman
a6bea7f767 Fix issue with yesod-core > 1.4.13.
yesod-core-1.4.14 changed the Template Haskell behind the expansion of
mkYesodData, such that it now must have the application data type
declared before it is called.  This change simply moves the call to
mkYesodData to be after the data type it wants to access is declared.

https://github.com/yesodweb/yesod/issues/1059 has some discussion of the
issue from the Yesod side.
2015-08-23 12:33:25 -04:00
Joey Hess
43aa881b47 --debug is passed along to git-annex-shell when git-annex is in debug mode. 2015-08-13 15:05:39 -04:00
Joey Hess
b14f76d72e remove unused language extensions, particulary unused TH 2015-08-04 12:02:27 -04:00
Joey Hess
29f3049def webapp: Support enabling known gitlab.com remotes. 2015-07-27 16:03:22 -04:00
Joey Hess
52ecf232fb only push when needsinit 2015-07-27 11:36:39 -04:00
Joey Hess
26d4eaa4e0 use mangled hostname for gitlab repo when using a dedicated git-annex ssh key 2015-07-27 11:03:58 -04:00
Joey Hess
97042d174e push git-annex branch to enable git-annex-shell to auto-init 2015-07-27 10:37:41 -04:00
Joey Hess
343ab2e358 basic gitlab support in webapp
This works, but needs more testing and work on cases like encrypted repos,
enabling existing repositories, etc.

This commit was sponsored by Shaun Westmacott.
2015-07-22 17:50:13 -04:00
Joey Hess
4af3dc72c1 assistant: Fix ANNEX_SHELL_DIR written to ~/.ssh/authorized_keys in local pairing to be the absolute path to the repository, not "."
This was a reversion caused by the relative path changes in 5.20150113.

Other uses of addAuthorizedKeys seem to be ok. If the user enters a
directory like ~/annex, it writes GIT_ANNEX_SHELL_DIRECTORY=annex, and
git-annex-shell assumes that's relative to HOME.
2015-07-02 15:10:50 -04:00
Joey Hess
20edbf850c assistant: Fix local pairing to not include newline in ssh pubkey, which is rejected on the other end for security reasons. 2015-07-02 14:50:20 -04:00
Sebastian Reuße
f66a343113 Support monitoring systemd-networkd connectivity. 2015-06-02 12:51:17 -04:00
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
6ec7c4fef4 webapp: Fix zombie xdg-open process left when opening file browser. Closes: #785498 2015-05-17 17:26:19 -04:00
Joey Hess
03667a162a couple of AMP warnings I missed before 2015-05-10 16:51:03 -04:00
Joey Hess
5c7cdbae46 more {-# OPTIONS_GHC -fno-warn-tabs #-} ... Forcing people who have what is merely a difference of opinion to you to do this is a bit of an asshole move. Just saying. 2015-05-10 16:38:49 -04:00
Joey Hess
9ee2745ca5 typo 2015-05-10 15:56:07 -04:00
Joey Hess
26ac0753c1 more FlexibleContexts 2015-05-10 15:54:58 -04:00
Joey Hess
082b1590ce avoid using now deprecated PortNum constructor 2015-05-10 15:49:47 -04:00
Joey Hess
0375331c61 another FlexibleContexts fix 2015-05-10 15:46:59 -04:00
Joey Hess
9917d2c94c fix build warning with ghc 7.10 2015-05-10 15:44:47 -04:00
Joey Hess
6cf62a9bde support time-1.5.0
This no longer uses old-locale's defaultTimeLocale, but provides one
of its own.

Factored out a Logs.TimeStamp.
2015-05-10 15:21:35 -04:00
Joey Hess
4aba1c74bd remaining dataenc to sandi conversions
I've tested all the dataenc to sandi conversions except Assistant.XMPP,
and all have unchanged behavior, including behavior on large unicode code
points.
2015-05-07 18:07:13 -04:00
Joey Hess
38c458b407 refactor 2015-04-30 14:02:56 -04:00
Joey Hess
995643665a Improve integration with KDE's file manager to work with dolphin version 14.12.3 while still being compatable with 4.14.2. Thanks, silvio. 2015-04-29 15:53:16 -04:00
Joey Hess
8f010bb55b also drop old dbus compat 2015-04-22 16:23:41 -04:00
Joey Hess
eb8ef44133 Dropped support for older versions of yesod and warp than the ones in Debian Jessie.
466 lines of compat cruft deleted!
2015-04-22 16:19:11 -04:00
Joey Hess
5f73bc3a12 Removed dependency on haskell SHA library, instead using cryptohash >= 0.11.0. 2015-04-19 11:05:32 -04:00
Joey Hess
addc82dab7 removed all uses of undefined from code base
It's a code smell, can lead to hard to diagnose error messages.
2015-04-19 00:38:29 -04:00
Joey Hess
2736c59204 fix webapp reversion caused by relative path changes
webapp: When adding another local repository, and combining it with the
current repository, the new repository's remote path was set to "." rather
than the path to the current repository. This was a reversion caused by the
relative path changes in 5.20150113.
2015-04-09 14:59:08 -04:00
Joey Hess
7b32e7acb5 make segmentXargs preserve order 2015-04-02 00:31:36 -04:00
Joey Hess
2b7f3ee3f2 assistant: Committing a whole lot of files at once could overflow command-line length limits and cause the commit to fail. This only happened when using the assistant in an indirect mode repository. 2015-03-26 14:02:35 -04:00
Joey Hess
32b3bed086 add a comment 2015-03-26 13:43:30 -04:00
Joey Hess
450ee53ab6 When re-execing git-annex, use current program location, rather than ~/.config/git-annex/program, when possible.
Most of the time, there will be no discreprancy between programPath and
readProgramFile.

But, the programFile might have been written by an old version of git-annex
that is still installed, while a newer one is currently running. In this
case, we want to run the same one that's currently running.

This is especially important for things like the GIT_SSH=git-annex used for
ssh connection caching.

The only code that still uses readProgramFile directly is the upgrade code,
which needs to know where the standalone git-annex was installed, in order to
upgrade it.
2015-02-28 17:23:13 -04:00
Joey Hess
b9275b65f9 make programPath return FilePath not Maybe FilePath
Looking at the few current callers, it's ok to have programPath throw an
exception, in the unusual case where it cannot find git-annex.
2015-02-28 16:59:52 -04:00
Joey Hess
f5b830e07c sync, assistant: Include repository name in head branch commit message.
Note that while the assistant detects changes made to remote names, I left
the commit message fixed rather than calculating it after every commit. It
doesn't seem worth the CPU to do the latter.
2015-02-11 13:34:05 -04:00
Joey Hess
1316efdc6e assistant: Improve sanity check for control characters when pairing. 2015-02-09 16:34:57 -04:00
Joey Hess
bcf1b59c1b webapp: Fix reversion in opening webapp when starting it manually inside a repository. 2015-02-09 16:34:42 -04:00
Joey Hess
c6b5f2f3c4 assistant: Fix local pairing when ssh pubkey comment contains spaces. 2015-02-04 14:05:27 -04:00
Joey Hess
e8c376e0ad import Data.Default in Common 2015-01-28 16:11:28 -04:00
Joey Hess
70736d2b41 Repository tuning parameters can now be passed when initializing a repository for the first time.
* init: Repository tuning parameters can now be passed when initializing a
  repository for the first time. For details, see
  http://git-annex.branchable.com/tuning/
* merge: Refuse to merge changes from a git-annex branch of a repo
  that has been tuned in incompatable ways.
2015-01-27 17:38:06 -04:00
Joey Hess
587f6a919b addurl: When a Content-Disposition header suggests a filename to use, addurl will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this) 2015-01-22 14:52:52 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
068aaf943b on second thought, InodeCache should use getFileSize
This is necessary for interop between inode caches created on unix and
windows. Which is more important than supporting inodecaches for large keys
with the wrong size, which are broken anyway.

There should be no slowdown from this change, except on Windows.
2015-01-20 19:35:50 -04:00
Joey Hess
4f657aa14e add getFileSize, which can get the real size of a large file on Windows
Avoid using fileSize which maxes out at just 2 gb on Windows.
Instead, use hFileSize, which doesn't have a bounded size.
Fixes support for files > 2 gb on Windows.

Note that the InodeCache code only needs to compare a file size,
so it doesn't matter it the file size wraps. So it has been
left as-is. This was necessary both to avoid invalidating existing inode
caches, and because the code passed FileStatus around and would have become
more expensive if it called getFileSize.

This commit was sponsored by Christian Dietrich.
2015-01-20 17:09:24 -04:00
Joey Hess
11648fd719 fix OSX build 2015-01-09 14:36:02 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
ab039c6fd2 fix warning 2015-01-08 19:39:46 -04:00
Joey Hess
8ad5df7aa3 fix OS build 2015-01-08 19:39:04 -04:00
Joey Hess
fcbb81bbab fix OSX build 2015-01-08 19:37:05 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
5db31e2386 Fix build with process 1.2.1.0. 2015-01-04 13:07:57 -04:00
Joey Hess
c9a3e80d32 fixed all remaining build warnings on Windows 2014-12-29 17:30:20 -04:00
Joey Hess
da1d9d500d fix warning when building w/o dbus 2014-12-29 16:37:41 -04:00
Joey Hess
058956b841 use "the web" page for bittorrent too 2014-12-17 13:59:23 -04:00
Joey Hess
7e422269a6 move dummy uuids to Annex.UUID 2014-12-17 13:57:52 -04:00
Joey Hess
8d1e372a8b remove webUUID special case
Now bittorrent is another special case like web.

If the only remotes are web and bittorrent, it might as well scan those.
2014-12-17 13:52:08 -04:00
Joey Hess
30bf112185 Urls can now be claimed by remotes. This will allow creating, for example, a external special remote that handles magnet: and *.torrent urls. 2014-12-08 19:15:07 -04:00
Jon Ander Peñalba
f95b210a51 Use the new chunk param instead of the old chunksize for new Box.com remotes 2014-12-05 13:55:04 +01:00
Joey Hess
2f83914212 Add undo action to nautilus and konqueror integration. 2014-11-14 14:55:21 -04:00
Joey Hess
5451273be6 remove unused imports 2014-10-28 16:02:42 -04:00
Joey Hess
20a497b181 move remote removal into separate module
This allows using Git.Remote w/o needing to have Git.BuildVersion, which
requires configure. It will simplify github-backup when these libraries are
used there.
2014-10-27 11:28:58 -04:00
Joey Hess
ce9076370e deal with yesod's second gratuitous rename of the same function in a year 2014-10-23 20:26:46 -04:00
Joey Hess
31a1f061c6 typo 2014-10-23 20:25:31 -04:00
Joey Hess
fa1318479e rename isIA to configIA
Already done on s3-aws branch, so reduce divergence.
2014-10-23 15:56:35 -04:00
Joey Hess
1b90838bbd add internet archive item url to info 2014-10-21 15:34:32 -04:00
Joey Hess
1e59df083d Use haskell setenv library to clean up several ugly workarounds for inability to manipulate the environment on windows.
Didn't know that this library existed!

This includes making git-annex not re-exec itself on start on windows, and
making the test suite on Windows run tests without forking.
2014-10-15 20:33:52 -04:00
Joey Hess
9c8abb761f revert reversion
This reverts commit 7bfc4a5442.

Android build is going to have consistent versions again.
2014-10-15 11:49:13 -04:00
Joey Hess
7bfc4a5442 unbreak android build
This reverts commit dd667844b6
and commit e6eff0e951.

Those commits were fine, except the android autobuilder currently has a bit
of a mess of yesod versions and broke. Better to wait on this.
2014-10-14 14:38:38 -04:00
Joey Hess
e6eff0e951 more accurate bounds on yesod boostrap3 ifdefs 2014-10-13 15:33:51 -04:00
Joey Hess
788d86a9f1 fix LANGUAGE pragma 2014-10-09 17:07:35 -04:00
Joey Hess
164f0d73ff Support building with yesod 1.4.
The new yesod needs the ViewPatterns extension.
Also, a TH splice in Assistant/Threads/WebApp.hs failed to work without
OverLoadedStrings.

This commit was sponsored by Brock Spratlen.
2014-10-09 17:01:54 -04:00
Joey Hess
dd667844b6 avoid using Assistant.WebApp.Bootstrap3 when building with current yesod
Only use that when building with ancient yesod, which does not include it.

This also let me remove ifdefs in the file to support building with the new
version of yesod.
2014-10-09 15:19:24 -04:00
Joey Hess
9fd95d9025 indent with tabs not spaces
Found these with:
git grep "^  " $(find -type  f -name \*.hs) |grep -v ':  where'

Unfortunately there is some inline hamlet that cannot use tabs for
indentation.

Also, Assistant/WebApp/Bootstrap3.hs is a copy of a module and so I'm
leaving it as-is.
2014-10-09 15:09:26 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
ebd781984e clean up now unused imports 2014-09-16 13:37:50 -04:00
Joey Hess
09e20c3827 assistant: Detect when repository has been deleted or moved, and automatically shut down the assistant. Closes: #761261 2014-09-15 14:45:37 -04:00
Joey Hess
aebcc395ff use types to enforce that removeAnnex can only be called inside lockContent
This fixed one bug where it needed to be and wasn't (in Assistant.Unused).
And also found one place where lockContent was used unnecessarily (by
drop --from remote).

A few other places like uninit probably don't really need to lockContent,
but it doesn't hurt to do call it anyway.

This commit was sponsored by David Wagner.
2014-08-20 20:13:47 -04:00
Joey Hess
2418bae471 special case edit page for web remote
The crash came from calling Git.repoLocation, but it made sense to fix this
higher up, because there is nothing to edit about the web, it just is what
it is.
2014-08-12 16:35:13 -04:00
Joey Hess
ec11e0b89a fix build warning 2014-08-08 13:55:09 -04:00
Joey Hess
c784ef4586 unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.

Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.

Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.

However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.

Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-07 22:03:29 -04:00
Joey Hess
32e4368377 S3: support chunking
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
2014-08-02 15:51:58 -04:00
Joey Hess
867fd116a7 better exception display 2014-07-26 23:01:44 -04:00
Joey Hess
4f9a29e7f6 use passed program name for kde hooks 2014-07-21 15:30:40 -04:00
Joey Hess
89a72f94e2 webapp: Automatically install Konqueror integration scripts to get and drop files.
Based on the example from the tip, but modified to cd into the repo before
running git-annex, since konqueror does not. Also, at least on my system,
the directory is ~/.kde, not ~/.kde4. (konqueror 4.12.4)

This commit was sponsored by Jürgen Peters.
2014-07-21 15:27:24 -04:00
Joey Hess
67016b5123 Install nautilus hooks even when ~/.local/share/nautilus/ does not yet exist, since it is not automatically created for Gnome 3 users. 2014-07-16 17:19:39 -04:00
Joey Hess
542a8288a8 fix build with path-pieces-0.1.4 2014-07-13 19:24:38 -04:00
Joey Hess
6fe35aab87 Really fix bug that caused the assistant to make many unncessary empty merge commits. 2014-07-09 15:18:00 -04:00
Joey Hess
4a66cd3f91 assistant: Fix bug, introduced in last release, that caused the assistant to make many unncessary empty merge commits. 2014-07-05 17:12:05 -04:00
Joey Hess
326fdffce9
remove unused import 2014-07-04 15:24:35 -04:00
Joey Hess
fc67925fd7
reorg
avoid Git.Command needing Utility.Batch which needs async

For github-backup etc
2014-07-04 12:18:49 -04:00
Joey Hess
d41849bc23
support commit.gpgsign
Support users who have set commit.gpgsign, by disabling gpg signatures for
git-annex branch commits and commits made by the assistant.

The thinking here is that a user sets commit.gpgsign intending the commits
that they manually initiate to be gpg signed. But not commits made in the
background, whether by a deamon or implicitly to the git-annex branch.
gpg signing those would be at best a waste of CPU and at worst would fail,
or flood the user with gpg passphrase prompts, or put their signature on
changes they did not directly do.

See Debian bug #753720.

Also makes all commits done by git-annex go through a few central control
points, to make such changes easier in future.

Also disables commit.gpgsign in the test suite.

This commit was sponsored by Antoine Boegli.
2014-07-04 11:53:51 -04:00
Joey Hess
c4e718d1f5
missing import 2014-07-03 19:49:26 -04:00
Joey Hess
51d6ec6744
force a sane umask when making temp gpg home dir
Someone reported:

gpg: WARNING: unsafe permissions on homedir `/var/folders/m6/zkd11n111m38ff37zbtgq0lr0000gp/T/git-annex-gpg.tmp.0'

Just a warning, but let's fix it anyway. Preumably the user has one of the
many insane and delightful umasks users sometimes use to shoot themselves
in their feet.
2014-07-03 15:43:09 -04:00
Joey Hess
9fb8038166 Windows: Assistant now logs to daemon.log.
Yes, this means that git annex webapp on windows execs git-annex, which
execs itself to set env, and the execs itself again to redirect logs.

This is disgusting. This is Windows(TM).
2014-06-17 19:27:54 -04:00
Joey Hess
8f98d28da4 Windows: Fix opening file browser from webapp when repo is in a directory with spaces. 2014-06-17 14:21:08 -04:00
Joey Hess
7a8f8b5ac9 refactor 2014-06-16 18:59:23 -04:00
Joey Hess
501cc8623a assistant: Fix one-way assistant->assistant sync in direct mode.
When in direct mode, update the master branch after committing to the
annex/direct/master branch. Also, update the synced/master branch.

This fixes a topology A->B where both A and B are in direct mode and
running the assistant, and a change is made to B. Before this fix, A pulled
the changes from B, but since they were only on the annex/direct/master
branch, it did not merge them.

Note that I considered making the assistant merge the
remotes/B/annex/direct/master, but decided to keep it simple and only merge
the sync branches as before.
2014-06-16 11:32:13 -04:00
Joey Hess
e4d7e2ebde fix for Windows file timestamp timezone madness
On Windows, changing the time zone causes the apparent mtime of files to
change. This confuses git-annex, which natually thinks this means the files
have actually been modified (since THAT'S WHAT A MTIME IS FOR, BILL <sheesh>).

Work around this stupidity, by using the inode sentinal file to detect if
the timezone has changed, and calculate a TSDelta, which will be applied
when generating InodeCaches.

This should add no overhead at all on unix. Indeed, I sped up a few
things slightly in the refactoring.

Seems to basically work! But it has a big known problem:
If the timezone changes while the assistant (or a long-running command)
runs, it won't notice, since it only checks the inode cache once, and
so will use the old delta for all new inode caches it generates for new
files it's added. Which will result in them seeming changed the next time
it runs.

This commit was sponsored by Vincent Demeester.
2014-06-12 13:42:21 -04:00
Joey Hess
8ff9938d97 Fix build with wai 0.3.0.
This version of wai changed the type of Middleware, so I cannot seem
to liftIO inside it. So, got rid of a lot of not really needed
complexity to use System.Log.Logger's logging stuff, and just use
the standard wai stdout logger when debug logging is enabled.

Format may change some, and it logs http to stdout instead of stderr
now. Doesn't matter for the webapp since both go to the same log anyway.
2014-06-11 01:29:00 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
71b2a0a955 webapp: Include ssh port in mangled hostname.
This avoids a collision if different ssh ports are used on the same host
for some reason.

Note that it's ok to change the format of the mangled hostname; unmangling
only extracts the hostname from it, and once ssh is configured for a
mangled hostname, that config is not changed.
2014-06-05 12:50:15 -04:00
Fil
86d50149e9 typo ("Connnect") 2014-06-03 00:04:21 +02:00
Joey Hess
9c0f3ae7e1 assistant: Make sanity checker tmp dir cleanup code more robust. 2014-05-30 15:08:49 -04:00
Joey Hess
4f7f61e46e fix UI when enabling existing gcrypt repo
avoid editing repo for same reasons as in
52601eb606

avoid stomping on its description, even though no description exists until
after syncing is complete
2014-05-30 14:50:44 -04:00
Joey Hess
c92d43fb7b avoid stomping on existing description when enabling a special remote
Happened with eg, gcrypt remotes.
2014-05-30 14:30:43 -04:00
Joey Hess
86b9ae15dd also normalize /~/ 2014-05-30 14:23:21 -04:00
Joey Hess
52601eb606 don't show edit form after enabling ssh remote
Just after enabing a ssh remote, we've not synced with it yet, so its
description and group are not known. So, avoid showing edit form so user
doesn't see blank info. Instead, redirect to dashboard.
2014-05-30 14:17:20 -04:00
Joey Hess
9eaabf0382 webapp: avoid overwriting remote configs when enabling it
Avoid stomping on existing group and preferred content settings
when enabling or combining with an already existing remote.

Two level fix. First, use defaultStandardGroup rather than
setStandardGroup, so if there is an existing configuration in the git-annex
branch, it's not overwritten.

To handle pre-existing ssh remotes (including gcrypt), a second level is
needed, because before syncing with the remote, it's configuration won't be
available locally. (And syncing could take a long time.) So, in this case,
keep track of whether the remote is being created or enabled, and only set
configs when creating it.

This commit was sponsored by Anders Lannerback.
2014-05-30 14:03:04 -04:00
Joey Hess
7089e282b5 fetch from removable drive when adding
When adding a repo from a removable drive that already exists, fetch
from it before showing the edit form, so it will have the existing
configuration of that repo.
2014-05-30 12:48:01 -04:00
Joey Hess
de0cba18f6 webapp: Avoid stomping on existing group and preferred content settings when adding a local repository (or removable drive repository) that already exists.
This does mean that if the webapp is asked to add a git repository on
a removable drive that already exists, but is not yet a git-annex
repository, it will avoid putting it in any group. That unlikely edge case
is ok; the next step is the edit repository screen, which will show it's
not in any group and the user can pick one.
2014-05-30 12:32:44 -04:00
Joey Hess
2dd274e4ca webapp: When adding a new local repository, fix bug that caused its group and preferred content to be set in the current repository, even when not combining.
There was a tricky bit here, when it does combine, the edit form is shown,
and so the info needs to be committed to the new repository, but then
pulled into the current one. And caches need to be invalidated for it
to be visible in the edit form.
2014-05-29 20:17:05 -04:00
Joey Hess
b197ec8917 get rid of (completely safe) uses of Char8
Char8 often indicates an encoding bug. It didn't here, but I can avoid it
and not worry about it.
2014-05-27 20:26:10 -04:00
Joey Hess
a41ddabd73 webapp: When setting up a ssh remote, if the user inputs ~/foo, normalize that to foo, since it's in the home directory by default. 2014-05-27 14:33:27 -04:00
Joey Hess
d83c1a26cf
better work around for broken ghc in debian stable and some ubuntus 2014-05-23 13:00:29 -04:00
Joey Hess
2e1179df46 webapp: When setting up a ssh remote, record it using initremote, so that it can be easily enabled elsewhere.
This is the capstone in making the webapp remember ssh remotes
so they can be easily enabled in other clones of the repository.

Currently, the user will need to enter a password to enable the ssh remote,
but everything else is filled in automatically.

This commit was sponsored by Peter Lloyd.
2014-05-22 14:57:42 -04:00
Joey Hess
c11461b860 webapp: Support for enabling known git repositories on ssh servers.
The repository must have been added using initremote.

Turned out to be much much simpler than expected, because I was able to
reuse the existing code for enabling rsync and gcrypt remotes, which
was already sufficiently general that it will also work for ssh remotes.
Total win!

This commit was sponsored by an unknown bitcoin contributor.
2014-05-22 14:10:48 -04:00
Joey Hess
c6d690f350 include path in "Cannot find old distribution bundle; not upgrading" message 2014-05-21 13:27:40 -04:00
Joey Hess
fed509fb3e assistant: When there are multiple remotes giving different ways to access the same repository, honor remote cost settings and use the cheapest available.
Note that TransferInfo does not always contain the Remote, although
any transfer added to the TransferQueue does have a Remote in its
TransferInfo. The transferkeys command still accepts a UUID, which is
useful to handle upgrades, where an old assistant version runs the new
transferkeys.

This commit was sponsored by Kalle Svensson.
2014-05-19 16:19:33 -04:00
Joey Hess
d37675b1fd
work around utterly strange ubuntu ghc Conflicting definitions for `_' problem 2014-05-19 11:00:01 -04:00
Joey Hess
c0e7aeccd7 fix build with older version of yessod
Old yesod's hamlet parser does not understand f@(Data ...) syntax,
work around this.
2014-05-18 11:17:05 -04:00
Joey Hess
1a40a66d72 webapp: Fixed drag and drop to reorder the list of remotes.
Broken by 958312885f, in November!
I missed this because there's no strong type checking across the AJAX call. :(

Need to switch to Fay to avoid such bugs..
2014-05-15 22:01:32 -04:00
Joey Hess
dc72ea4ab5 deal with ssh key expiry
Not a perfect solution, but good enough, few users will wait 10 minutes in
the middle and see it expire, I hope.
2014-05-15 15:16:37 -04:00
Joey Hess
c705df5651 remove windows-specific rsync.net code, no longer necessary thanks to ssh password handling
Since ssh password prompting no longer happens on stdin, the
authorized_keys line can be sent on stdin. Yay!
2014-05-15 12:30:29 -04:00
Joey Hess
f41b585c29 force strict host key checking when host is known
Avoid any possibilty of prompting in ssh setup in webapp.

Prticularly on Windows this was a problem, it seemed to enter an infinite
loop. I think that ssh can sometimes use SSH_ASKPASS for y/n prompting,
when no controlling TTY is available, and since git-annex always answers
back with the host's password, not y/n, it looped.

This commit was sponsored by Simon Michael.
2014-05-14 18:13:53 -04:00
Joey Hess
a66c942645 refactor 2014-05-14 17:31:20 -04:00
Joey Hess
74f937cc55 this will hopefully make the ssh askpass work on windows 2014-05-14 17:13:20 -04:00
Joey Hess
a11176bab3 avoid clobbering existing env
This is necessary on windows, ssh couldn't resolve hostnames without env
for whatever reason.
2014-05-14 16:36:03 -04:00
Joey Hess
740de08461 fix windows build 2014-05-14 15:54:41 -04:00
Joey Hess
85e9e8c0cf webapp: Better ssh password prompting.
When setting up a remote on a ssh server, prompt for a password inside the
webapp, rather than relying on ssh's own password prompting in the terminal
the webapp was started from, or ssh-askpass.

Avoids double prompting for the ssh password (and triple-prompting on
windows for rsync.net), since the entered password is cached for 10 minutes
and this cached password is reused when setting up the repository, after
the initial probe.

When the user has an existing ssh key set up, they can choose to use it,
rather than entering a password. The webapp used to probe for this case
automatically, so this is a little harder, but it's an advanced user thing.

Note that this commit is known to break enabling existing rsync
repositories. It hs not been tested with gcrypt repositories. It's not been
successfully tested yet on Windows.

This commit was sponsored by Ralph Mayer.
2014-05-14 15:38:32 -04:00
Joey Hess
db8590791f Merge branch 'master' into sshpassword 2014-05-14 12:43:34 -04:00
Joey Hess
4e987d05a8 work around an EvilSplicer bug
This should fix the android build. The EvilSplicer cannot fix up ghc's
misformatting of a case expression with a construstor with a lot of
parameters, which spans multiple lines.
2014-05-10 12:27:21 -03:00
Joey Hess
2aed2d8510 Merge branch 'master' into bootstrap3
Conflicts:
	debian/changelog
2014-05-02 15:32:49 -03:00
Sören Brunk
58f92f148b fix more layout issues based on joey's feedback 2014-05-02 16:29:23 +02:00
Joey Hess
44b6d6c5e0 wip 2014-04-30 21:27:17 -04:00
Joey Hess
ac98853f05 add CredPair cache
Note that this does not yet use SecureMem. It would probably make sense for
the Password part of a CredPair to use SecureMem, and making that change
is better than passing in a String and having it converted to SecureMem in
this code.
2014-04-29 18:08:02 -04:00
Joey Hess
90eca40df2 assistant: Clean up stale tmp files on startup.
This commit was sponsored by Melissa Binde.
2014-04-27 22:22:46 -03:00
Sören Brunk
036aa129f7 fix navbar on error page 2014-04-27 23:57:57 +02:00
Sören Brunk
84eaf8b447 backport Yesod.Form.Bootstrap3 to Yesod 1.0.1 2014-04-27 20:18:36 +02:00
Joey Hess
f5b1c3841e avoid build warning from new dbus 2014-04-26 19:24:56 -04:00
Joey Hess
84a88aab9a avoid build warning from new dbus 2014-04-26 18:57:51 -04:00
Sören Brunk
54fe9af0bb create a local copy of Yesod.Form.Bootstrap3 2014-04-25 12:26:33 +02:00
Joey Hess
ca496b6a97 Standalone builds now check gpg signatures before upgrading. 2014-04-23 13:30:30 -04:00
Sören Brunk
ccc8afb59d don't close tags that are closed by hamlet 2014-04-21 19:47:21 +02:00
Sören Brunk
00c1cd0db1 bootstrap3 forms 2014-04-21 19:47:05 +02:00
Sören Brunk
d2b42c30ad bootstrap3 glyphicons 2014-04-21 19:45:22 +02:00
Sören Brunk
23091cc318 adapt templates to bootstrap3 layout and class changes 2014-04-21 19:45:18 +02:00
Sören Brunk
c8dccf9582 update bootstrap files to bootstrap3 2014-04-21 19:05:09 +02:00
Joey Hess
eba3a28a28 webapp: Support using git-annex on a remote server, which was installed from the standalone tarball or OSX app, and so does not have git-annex in PATH (and may also not have git or rsync in PATH).
* webapp: Support using git-annex on a remote server, which was installed
  from the standalone tarball or OSX app, and so does not have
  git-annex in PATH (and may also not have git or rsync in PATH).
* standalone tarball, OSX app: Install a ~/.ssh/git-annex-wrapper, which
  can be used to run git-annex, git, rsync, etc.
2014-04-20 18:39:10 -04:00
Joey Hess
512da29273 send remote-daemon a RELOAD after making a ssh remote
This doesn't work yet, because RELOAD is buggy and does not notice the new
remote.
2014-04-20 15:30:39 -04:00
Joey Hess
1a4c3caa96 avoid showing the connection nudge alert after creating git or gcrypt remote 2014-04-20 15:10:29 -04:00
Joey Hess
9724667a3d webapp: Fix UI for removing XMPP connection. 2014-04-20 12:46:33 -04:00
Joey Hess
e880d0d22c replace (Key, Backend) with Key
Only fsck and reinject and the test suite used the Backend, and they can
look it up as needed from the Key. This simplifies the code and also speeds
it up.

There is a small behavior change here. Before, all commands would warn when
acting on an annexed file with an unknown backend. Now, only fsck and
reinject show that warning.
2014-04-17 18:03:39 -04:00
Joey Hess
915d038bec reinit: New command that can initialize a new reposotory using the configuration of a previously known repository. Useful if a repository got deleted and you want to clone it back the way it was. 2014-04-15 20:13:35 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
2ff9ba9f74
add missing Network.URI Ord instance for Debian stable 2014-04-14 13:25:49 -04:00
Joey Hess
ff192ace2f detect wicd network disconnection events 2014-04-12 18:36:48 -04:00
Joey Hess
bb44df206d NetWatcher: detect when networkmanager has lost network connection
This is a better approach to finding both when NM has lost a network
connection, and when a new network connection is made by NM.

Tested with network-manager 0.9.8.8.

This commit was sponsored by Cedric Staub.
2014-04-12 17:58:19 -04:00
Joey Hess
a33b30d0c4 remotedaemon: When network connection is lost, close all cached ssh connections.
This commit was sponsored by Cedric Staub.
2014-04-12 16:32:59 -04:00
Joey Hess
db38678595 webapp: Rework xmpp nudge to prompt for either xmpp or a ssh remote be set up.
This commit was sponsored by Nathan Howell.
2014-04-09 16:27:24 -04:00
Joey Hess
14349fb752 also show signal icon next to connected xmpp remotes 2014-04-09 15:51:00 -04:00
Joey Hess
33b8cff433 webapp: Show a network signal icon next to ssh remotes that it's currently connected with. 2014-04-09 15:26:41 -04:00
Joey Hess
fb73792f72 make git-remote-daemon ssh transport robust
* Remote system might be available, and connection get lost. Should
  reconnect, but needs to avoid bad behavior (ie, constant reconnect
  attempts.) Use exponential backoff.

* Detect if old system had a too old git-annex-shell, and show the user
  a nice message in the webapp. Required parsing error messages, so perhaps
  this code shoudl be removed once enough time has passed..

* Switch the protocol to using remote URI's, rather than remote names.
  Names change. Also avoids issues with serialization of names containing
  whitespace.

This is nearly ready for merge into master now. I'd still like to make the ssh
transport smarter about reusing ssh connection caching during git pull.

This commit was sponsored by Jim Paris.
2014-04-09 14:14:55 -04:00
Joey Hess
fac7bca05b assistant: Now detects immediately when other repositories push changes to
a ssh remote, and pulls.

XMPP is no longer needed in this configuration!

Requires the remote server have git-annex-shell with notifychanges support.

(untested)

This commit was sponsored by Geog Wechslberger.
2014-04-08 15:23:50 -04:00
Joey Hess
ade5d8ccc5 Fix nautilus script installation to not crash when the nautilus script dir does not exist. Instead, only install scripts when the directory already exists.
nautilus creates this directory each time run, so this avoids putting cruft
in user's home
2014-04-05 16:36:58 -04:00
Joey Hess
39996f8222 assistant: Adjust ssh authorized keys line generated in local pairing or for a remote ssh server to set environment variables in an alternative way that works with the non-POSIX fix shell, as well as POSIX shells. 2014-04-02 15:30:10 -04:00
Joey Hess
065248f3d2 Added required content configuration.
This includes checking when dropping files that any required content
configuration is satisfied. However, it does not yet include an active
check on the required content; the location log is trusted when checking
the required content expression.
2014-03-29 16:03:33 -04:00
Joey Hess
f2ddf9a299 reload preferred content when group preferred content log changes 2014-03-29 14:46:57 -04:00
Joey Hess
fe19e15040 reorg matcher types; no non-type code changes 2014-03-29 14:43:34 -04:00
Joey Hess
1731cb0840
use shebang lib 2014-03-23 08:17:03 -04:00
Johan Kiviniemi
284e6b5e8d installNautilus: “exec” git-annex and protect params with “--” 2014-03-23 07:28:28 +02:00
Joey Hess
f6eb967f1e webapp: Automatically install Nautilus integration scripts to get and drop files.
This commit was sponsored by Gian-Maria Daffre.
2014-03-22 15:51:30 -04:00
Joey Hess
e426fac273 add desktop notifications
Motivation: Hook scripts for nautilus or other file managers
need to provide the user with feedback that a file is being downloaded.

This commit was sponsored by THM Schoemaker.
2014-03-22 14:12:19 -04:00
Joey Hess
8d5a626a0b rename favicon.png to logo_16x16.png
This was never used as a favicon, it's just a png version of the
favicon.ico. It's actually used for the freedesktop menu file.
2014-03-16 15:52:16 -04:00
Joey Hess
66b8b9c094 webapp: Use securemem for constant time auth token comparisons.
Debian stable does not have securemem, but neither does it have warp-tls,
so just disable use of securemem when not building with https support.
2014-03-12 21:41:20 -04:00
Joey Hess
a7b3a2b097 another old warp-tls workaround 2014-03-12 12:32:43 -04:00
Joey Hess
5a37266080 use Monoid instance for Ordering 2014-03-09 14:08:45 -04:00
Joey Hess
54746463bb webapp: Added a "Sync now" item to each repository's menu. 2014-03-06 18:11:44 -04:00
Joey Hess
d32289bab0 assistant: Smarter log file rotation, which takes free disk space into account. 2014-03-05 18:26:53 -04:00
Joey Hess
a3fe8270ca annex.startupscan can be set to false to disable the assistant's startup scan. 2014-03-05 17:44:14 -04:00
Joey Hess
74d3949cd0 webapp: Include no-pty in ssh authorized_keys lines.
git-annex-shell does not need a pty, so this speeds things up.
Also, it may avoid weird misconfigured systems that try to run screen or
tmux on every ssh login from doing so.
2014-03-05 13:53:21 -04:00
Joey Hess
d626a784f3 webapp: detect on first run if git is not installed, and tell the user, rather than crashing non-informatively
Mostly for Windows, which has no sane package management..
2014-03-05 13:43:56 -04:00
Joey Hess
14d1e878ab sync: Automatically resolve merge conflict between and annexed file and a regular git file.
This is a new feature, it was not handled before, since it's a bit of an
edge case. However, it can be handled exactly the same as a file/dir
conflict, just leave the non-annexed item alone.

While implementing this, the core resolveMerge' function got a lot simpler
and clearer. Note especially that where before there was an asymetric call to
stagefromdirectmergedir, now graftin is called symmetrically in both cases.

And, in order to add that `graftin us`, the current branch needed to be
known (if there is no current branch, there cannot be a merge conflict).
This led to some cleanups of how autoMergeFrom behaved when there is no
current branch.

This commit was sponsored by Philippe Gauthier.
2014-03-04 19:35:55 -04:00
Joey Hess
99295f2c1d factor out Annex.AutoMerge from Command.Sync 2014-03-04 16:26:15 -04:00
Joey Hess
6a355686ff annex.listen can be configured, instead of using --listen 2014-03-01 00:31:17 -04:00
Joey Hess
1de3d3b9c9
fix restart/shutdown when using https 2014-02-28 22:20:54 -04:00
Joey Hess
3c3744c9a9 use https when .git/annex/privkey.pem and .git/annex/certificate.pem exist (untested)
I have not managed to generate a key that is accepted by the old version of
warp-tls I have here.
2014-02-28 21:32:18 -04:00
Joey Hess
6469c1aca9 webapp: Don't list the public repository group when editing a git repository; it only makes sense for special remotes. 2014-02-28 20:37:03 -04:00
Joey Hess
1681386b0d webapp: Filter out from Switch Repository list any repositories listed in autostart file that don't have a git directory anymore.
Trying to start in such a repo will, obviously, fail.

Note that assistant --autostart will try to start in such a repo, and fail,
but does start successfully in the other autostart repos.
2014-02-28 19:16:49 -04:00
Joey Hess
a1432bce2f Put non-object tmp files in .git/annex/misctmp, leaving .git/annex/tmp for only partially transferred objects.
This allows eg, putting .git/annex/tmp on a ram disk, if the disk IO
of temp object files is too annoying (and if you don't want to keep
partially transferred objects across reboots).

.git/annex/misctmp must be on the same filesystem as the git work tree,
since files are moved to there in a way that will not work cross-device,
as well as symlinked into there.

I first wanted to put the tmp objects in .git/annex/objects/tmp, but
that would pose transition problems on upgrade when partially transferred
objects existed.

git annex info does not currently show the size of .git/annex/misctemp,
since it should stay small. It would also be ok to make something clean it
out, periodically.
2014-02-26 16:52:56 -04:00
Joey Hess
f8cfcd4e44 couple more warning fixes 2014-02-25 14:53:43 -04:00
Joey Hess
e1cf4e5e4e only warn about no dbus when building on linux 2014-02-25 14:49:27 -04:00
Joey Hess
3f6e4b8c7c fix all remaining -Wall warnings on Windows 2014-02-25 14:48:50 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
c69d6eb035 Make annex.web-options be used in several places that call curl. 2014-02-24 21:29:37 -04:00
Joey Hess
4e0be2792b remove Read instance for Ref
Removed instance, got it all to build using fromRef. (With a few things
that really need to show something using a ref for debugging stubbed out.)

Then added back Read instance, and made Logs.View use it for serialization.
This changes the view log format.
2014-02-19 01:19:57 -04:00
Joey Hess
27c4d5cd1b Revert "work around what is likely a bug in a new version of ssh"
This reverts commit f626d73442.

Bug was fixed in -2 of debian package, and is fixed upstream
(unsure what upstream release specificically)
2014-02-14 22:39:59 -04:00
Joey Hess
f626d73442 work around what is likely a bug in a new version of ssh
This avoids running into it for new setups, but does not transition old
setups. Just a quick fix; I hope to get ssh fixed.
2014-02-14 16:45:39 -04:00
Joey Hess
fd09798e40 windows webapp: fix rsync.net support 2014-02-14 15:47:54 -04:00
Joey Hess
f11f7520b5 windows: Fix process termination code.
The ctrl-c hack used before didn't actually seem to work.

No haskell libraries expose TerminateProcess. I tried just calling it via
FFI, but got segfaults, probably to do with the wacky process handle not
being managed correctly. Moving it all into one C function worked.

This was hell. The EvilLinker hack was just final icing on the cake.
We all know what the cake was made of.
2014-02-13 15:53:10 -04:00
Joey Hess
84083ecdd3 Windows: Crazy hack to make file manager not start hidden underneath web browser window
I can't even..
2014-02-13 13:16:28 -04:00
Joey Hess
36e66997c6 Windows: Fix startAssistant to not wait for ever for a non-daemonizing process
Seemed to fix several problems.
2014-02-13 13:05:07 -04:00
Joey Hess
029a1c431a
remove windows --git-dir unix style path hack
This is no longer necessary, at least with msysgit 1.8.5.2.msysgit.0.
Its root cause may have been fixed by other recent git path fixes.
It was causing the webapp to fail to make repos on other drives.
2014-02-11 16:12:22 -04:00
Joey Hess
7b19c7d25b cleanup thanks to Utility.PID 2014-02-11 15:39:51 -04:00
Joey Hess
30a474b309
fix windows build 2014-02-11 15:04:07 -04:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
4e7c65dca0 Fix build on platforms not supporting the webapp. 2014-02-08 14:25:18 -04:00
Joey Hess
c91bbc9494 fix reversion on repo deletion
A broken symlink would cause the mode setting to fail.
2014-02-03 14:20:39 -04:00
Joey Hess
29bb04aa0d Windows: Fix deletion of repositories by test suite and webapp.
On Windows, a file that is not writable cannot be deleted even if in a
directory with write perms. So git object files were not getting deleted
when removing a git repository.
2014-02-03 11:19:41 -04:00
Joey Hess
a542781f6a remove some monkey faces 2014-02-01 17:14:38 -04:00
Joey Hess
bbef0cddfd improve sync with xmpp and annex-ignore
* sync --content: Honor annex-ignore configuration.
* sync: Don't try to sync with xmpp remotes, which are only currently
  supported when using the assistant.
2014-02-01 10:33:55 -04:00
Joey Hess
cce69eee4d avoid using function named that conflicts with name used in newer version of process library 2014-01-29 13:44:53 -04:00
Joey Hess
74b101d1dd reorg 2014-01-26 16:36:31 -04:00
Joey Hess
78d0fd389e webapp: After upgrading a git repository to git-annex, fix bug that made it temporarily not be synced with. 2014-01-26 13:57:05 -04:00
Joey Hess
f54bb25501 fix delay of daily sanity check (hacked for testing and accidentially committed) 2014-01-23 16:57:49 -04:00
Joey Hess
964a181026 try to drop unused object if it does not need to be transferred anywhere 2014-01-23 16:51:16 -04:00
Joey Hess
3518c586cf fix transfers of key with no associated file
Several places assumed this would not happen, and when the AssociatedFile
was Nothing, did nothing.

As part of this, preferred content checks pass the Key around.

Note that checkMatcher is sometimes now called with Just Key and Just File.
It currently constructs a FileMatcher, ignoring the Key. However, if it
constructed a FileKeyMatcher, which contained both, then it might be
possible to speed up parts of Limit, which currently call the somewhat
expensive lookupFileKey to get the Key.

I have not made this optimisation yet, because I am not sure if the key is
always the same. Will need some significant checking to satisfy myself
that's the case..
2014-01-23 16:44:02 -04:00
Joey Hess
e0bd088f08 add webapp UI to manage unused files 2014-01-23 15:09:43 -04:00
Joey Hess
489972c035 allow annex.expireunused to be set to false, as well as to a duration 2014-01-22 23:10:51 -04:00
Joey Hess
3da0064657 assistant unused file handling
Make sanity checker run git annex unused daily, and queue up transfers
of unused files to any remotes that will have them. The transfer retrying
code works for us here, so eg when a backup disk remote is plugged in,
any transfers to it are done. Once the unused files reach a remote,
they'll be removed locally as unwanted.

If the setup does not cause unused files to go to a remote, they'll pile
up, and the sanity checker detects this using some heuristics that are
pretty good -- 1000 unused files, or 10% of disk used by unused files,
or more disk wasted by unused files than is left free. Once it detects
this, it pops up an alert in the webapp, with a button to take action.

TODO: Webapp UI to configure this, and also the ability to launch an
immediate cleanup of all unused files.

This commit was sponsored by Simon Michael.
2014-01-22 22:53:18 -04:00
Joey Hess
ed7c61914c assistant: Run the periodic git gc in batch mode. 2014-01-22 17:11:41 -04:00
Joey Hess
f7cdc40f7b reorg 2014-01-21 18:08:56 -04:00
Joey Hess
0ef282a116 numcopies cleanup, part 2
This includes several bug fixes.
2014-01-21 17:25:39 -04:00
Joey Hess
b40df4f0d0 reorganize numcopies code (no behavior changes)
Move stuff into Logs.NumCopies. Add a NumCopies newtype.

Better names for various serialization classes that are specific to one
thing or another.
2014-01-21 16:08:59 -04:00
Joey Hess
d66535f065 global numcopies setting
* numcopies: New command, sets global numcopies value that is seen by all
  clones of a repository.
* The annex.numcopies git config setting is deprecated. Once the numcopies
  command is used to set the global number of copies, any annex.numcopies
  git configs will be ignored.
* assistant: Make the prefs page set the global numcopies.

This global numcopies setting is needed to let preferred content
expressions operate on numcopies.

It's also convenient, because typically if you want git-annex to preserve N
copies of files in a repo, you want it to do that no matter which repo it's
running in. Making it global avoids needing to warn the user about gotchas
involving inconsistent annex.numcopies settings.
(See changes to doc/numcopies.mdwn.)

Added a new variety of git-annex branch log file, that holds only 1 value.
Will probably be useful for other stuff later.

This commit was sponsored by Nicolas Pouillard.
2014-01-20 16:47:56 -04:00
Joey Hess
73c420ffcf much better command action handling for sync --content 2014-01-20 13:31:03 -04:00
Joey Hess
b6ba0bd556 sync --content: New option that makes the content of annexed files be transferred.
Similar to the assistant, this honors any configured preferred content
expressions.

I am not entirely happpy with the implementation. It would be nicer if
the seek function returned a list of actions which included the individual
file gets and copies and drops, rather than the current list of calls to
syncContent. This would allow getting rid of the somewhat reundant display
of "sync file [ok|failed]" after the get/put display.

But, do that, withFilesInGit would need to somehow be able to construct
such a mixed action list. And it would be less efficient than the current
implementation, which is able to reuse several values between eg get and
drop.

Note that currently this does not try to satisfy numcopies when
getting/putting files (numcopies are of course checked when dropping
files!) This makes it like the assistant, and unlike get --auto
and copy --auto, which do duplicate files when numcopies is not yet
satisfied. I don't know if this is the right decision; it only seemed to
make sense to have this parallel the assistant as far as possible to start
with, since I know the assistant works.

This commit was sponsored by Øyvind Andersen Holm.
2014-01-19 17:49:54 -04:00
Joey Hess
1d3e60830d remove debug print 2014-01-17 15:09:03 -04:00
Joey Hess
18a3e51d52 assistant: Detect if .git/annex/index is corrupt at startup, and recover. 2014-01-14 17:10:30 -04:00
Joey Hess
207ac67aaa avoid needing a build-dep on hxt for Data.AssocList 2014-01-14 16:42:10 -04:00
Joey Hess
d07f2d7865 Fix a long-standing bug that could cause the wrong index file to be used when committing to the git-annex branch, if GIT_INDEX_FILE is set in the environment. This typically resulted in git-annex branch log files being committed to the master branch and later showing up in the work tree. (These log files can be safely removed.) 2014-01-14 15:36:33 -04:00
Joey Hess
c20f31a1ad add GETAVAILABILITY to external special remote protocol
And some reworking of types, and added an annex-availability git config
setting.
2014-01-13 14:41:10 -04:00
Joey Hess
1c8f5f830b revert use of Data.Map.Strict
memory profile shows this did not contribute to the memory leaks fixed in
b92b54bd42
2014-01-07 14:55:06 -04:00
Joey Hess
5279c4d1df tested transferkeys restarting; fix some bugs 2014-01-06 17:07:08 -04:00
Joey Hess
d356069f3e use strict version of map 2014-01-06 16:04:09 -04:00
Joey Hess
e5b4d447b6 assistant: Start a new git-annex transferkeys process after a network connection change
So that remotes that use a persistent network connection are restarted.

A remote might keep open a long duration network connection, and could
fail to deal well with losing the connection. This is particularly a
concern now that we have external special reotes. An external
special remote that is implemented naively might open the connection only
when PREPARE is sent, and if it loses connection, throw errors on each
request that is made.

(Note that the ssh connection caching should not have this problem; if the
long-duration ssh process loses connection, the named pipe is disconnected
and the next ssh attempt will reconnect. Also, XMPP already deals with
disconnection robustly in its own way.)

There's no way for git-annex to know if a lost network connection actually
affects a given remote, which might have a transfer in process. It does not
make sense to force kill the transferkeys process every time the NetWatcher
detects a change. (Especially because the NetWatcher sometimes polls 1
change per hour.)

In any case, the NetWatcher only detects connection to a network, not
disconnection. So if a transfer is in progress over the network, and the
network goes down, that will need to time out on its own.

An alternate approch that was considered is to use a separate transferkeys
process for each remote, and detect when a request fails, and assume that
means that process is in a failing state and restart it. The problem with
that approach is that if a resource is not available and a remote fails
every time, it degrades to starting a new transferkeys process for every
file transfer, which is too expensive.

Instead, this commit only handles the network reconnection case, and restarts
transferkeys only once the network has reconnected and another transfer needs
to be made. So, a transferkeys process will be reused for 1 hour, or until the
next network connection.

----

The NotificationBroadcaster was rewritten to use TMVars rather than MSampleVars,
to allow checking without blocking if a notification has been received.

----

This commit was sponsored by Tobias Brunner.
2014-01-06 16:03:39 -04:00
Joey Hess
b92b54bd42 assistant: Fixed several minor memory leaks that manifested when adding a large number of files. 2014-01-05 21:30:48 -04:00
Joey Hess
fdcc04096c assistant: Ensure that .ssh/config and .ssh/authorized_keys are not group or world writable when writing to those files, as that can make ssh refuse to use them, if it allows another user to write to them. 2014-01-03 17:44:12 -04:00
Joey Hess
f7727d2df1 Remotes can now be made read-only, by setting remote.<name>.annex-readonly 2014-01-02 13:12:32 -04:00
Joey Hess
f83e5c75dc Assistant: Remove stale MERGE_HEAD files in lockfile cleanup.
See http://git-annex.branchable.com/forum/How_to_resolve_a_interrupted_merge_in_direct_mode__63__/
2014-01-01 22:05:33 -04:00
Joey Hess
079f463d51 mirror: Support --all (and --unused). 2014-01-01 17:39:33 -04:00
Joey Hess
fab7adb161 Windows: Avoid eating stdin when running ssh to add a authorized key, since this is used for password prompting. 2014-01-01 15:49:51 -04:00
Joey Hess
5c834daf59 avoid empty env vars when setting up clean environment 2013-12-31 14:55:55 -04:00
Joey Hess
6c565ec905 external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!

The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)

Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo

Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:

* Need to lock the remote when using it to eg make a request, because
  in theory git-annex could have multiple threads that each try to use
  a remote at the same time. I don't think that git-annex ever does
  that currently, but better safe than sorry.

* Rather than starting up every external special remote program when
  git-annex starts, they are started only on demand, when first used.
  This will avoid slowdown, especially when running fast git-annex query
  commands. Once started, they keep running until git-annex stops, currently,
  which may not be ideal, but it's hard to know a better time to stop them.

* Bit of a chicken and egg problem with caching the cost of the remote,
  because setting annex-cost in the git config needs the remote to already
  be set up. Managed to finesse that.

This commit was sponsored by Lukas Anzinger.
2013-12-26 18:23:13 -04:00
Joey Hess
8e58ff7f71 check that ~/Desktop/annex can be written to; fall back to ~/annex when not 2013-12-24 17:04:07 -04:00
Joey Hess
3a6e0d1215 assistant: Set StrictHostKeyChecking yes when creating ssh remotes, and add it to the configuration for any ssh remotes previously created by the assistant. This avoids repeated prompts by ssh if the host key changes, instead syncing with such a remote will fail. Closes: #732602 2013-12-20 20:58:36 -04:00
Richard Hartmann
974fe009bf Another round of s/amoung/among/ 2013-12-19 12:30:53 -04:00
Joey Hess
adca65b31a remove some unused vars
Warnings for these are normally hidden somehow by TH, but show up when
building with EvilSplicer.
2013-12-18 22:39:15 -04:00
Joey Hess
c99d6a8151 assistant: Fix OSX-specific bug that caused the startup scan to try to follow symlinks to other directories, and add their contents to the annex. 2013-12-18 15:05:29 -04:00
Joey Hess
b7e3fe2ebd flip for clarity 2013-12-16 16:24:57 -04:00
Joey Hess
58c7b0a56d assistant: Always batch changes found in startup scan.
Batch detection is heuristic, so can sometimes fail. I observed one such
failure while starting up in a repository with 87000 files. After the first
several batches of ~5000 files, it fell out of batch mode, and never
re-entered it, and so made many more commits of a few files at a time
than necessary.

So, let's always use batch mode when in the startup scan. This avoids the
heuristic there, at least.

There is clearly also room to improve the heuristic. Possibly 10 files is
too high a bar to be found during a commit, on a system that can commit
quickly.
2013-12-16 16:16:19 -04:00
Joey Hess
0bc4a78db5 squash warning in OSX build 2013-12-15 17:10:24 -04:00
Joey Hess
13f7e51724 add a note about only writable drives being listed 2013-12-12 14:25:50 -04:00
Joey Hess
011b8bc7ec pull in Win32-extras, to be able to get current process id in Windows
Fixed up a number of things that had worked around there not being a way to
get that.

Most notably, transfer info files on windows now include the process id,
since no locking is currently done. This means the file format varies
between windows and unix.
2013-12-11 00:15:10 -04:00
Joey Hess
c01af6285f typo 2013-12-10 23:55:15 -04:00
Joey Hess
ecd42aef8e different PID types for Unix and Windows
Windows has a larger (unsigned) PID space, so cannot use the unix CInt
there.

Note that TransferInfo does not yet ever get the TransferPid populated,
as there is missing locking.
2013-12-10 23:48:42 -04:00
Joey Hess
2fd63f3cfa port transferkeys to windows; make stopping in progress transfers work too (probably)
transferkeys had used special FDs for communication, but that would be
quite annoying to do in Windows.

Instead, use stdin and stdout. But, to avoid commands like rsync stomping
on them and messing up the communications channel, they're duplicated to a
different handle; stdin is replaced with a null handle, and stdout is
replaced with a copy of stderr. This should all work in windows too.

Stopping in progress transfers may work on windows.. if the types unify
anyway. ;) May need some more porting.
2013-12-10 23:19:18 -04:00
Joey Hess
e6c4f550d8 repair: Remove damaged git-annex sync branches. 2013-12-10 16:17:49 -04:00
Joey Hess
ce045a51af Improve repair of git-annex index file.
Fixes a test case I received where a corrupted repo was repaired, but the
git-annex branch was not. The root of the problem was that the
MissingObject returned by the repair code was not necessarily a complete
set of all objects that might have been deleted during the repair.

So, stop trying to return that at all, and instead make the index file
checking code explicitly verify that each object the index uses is present.
2013-12-10 15:40:01 -04:00
Joey Hess
7c5261efb1 import fixes 2013-12-10 01:26:52 -04:00
Joey Hess
063a183a61 windows disk free space checking 2013-12-10 01:18:04 -04:00
Joey Hess
0bb9ace195 fix drive path so fileAccess will work with it 2013-12-09 17:58:49 -04:00
Joey Hess
98bd6e1f89 Revert "add /dashboard route as an alias for /"
This reverts commit a1880a152c.

Didn't help firefox problem.
2013-12-09 17:26:25 -04:00
Joey Hess
6f8693c59a redirect to /dashboard 2013-12-09 17:24:53 -04:00
Joey Hess
a1880a152c add /dashboard route as an alias for / 2013-12-09 17:14:45 -04:00
Joey Hess
d961985679 fix 2013-12-09 17:13:37 -04:00
Joey Hess
86a4b56db3 fix 2013-12-09 17:11:40 -04:00
Joey Hess
a268ac6c1d comment 2013-12-09 16:59:16 -04:00
Joey Hess
4866ec84df try all drive letters 2013-12-09 16:56:52 -04:00
Joey Hess
2066e90421 avoid needing --force on windows despite no lsof
Note that I still need to think this through and make sure handling of open
files is safe. This is just for testing purposes.
2013-12-09 16:56:15 -04:00
Joey Hess
51b43f2b68 better default annex location on windows 2013-12-09 16:39:23 -04:00
Joey Hess
fc3a9a8adb make opening file browser work on windows 2013-12-09 16:39:10 -04:00
Joey Hess
8fb1973c76 move all static site files out of subdirectories
Works around yesod-static path bug on windows.
https://github.com/yesodweb/yesod/issues/626
2013-12-09 14:38:22 -04:00
Joey Hess
a38abecd66 close tmp file handle
May fix permission problem on windows
2013-12-07 11:45:01 -04:00
Joey Hess
621ca07119 squash warning 2013-12-05 14:16:07 -04:00
Joey Hess
7a8e47d178 allow building on windows 2013-12-05 14:10:49 -04:00
Joey Hess
9d323a98e2 avoid trying to use lsof when it's not in path and --forced 2013-12-04 17:39:44 -04:00
Joey Hess
aef1d7e76d stuf on Windows 2013-12-04 15:37:53 -04:00
Joey Hess
8c55003ae4 don't shown non-working file browser opening UI on android 2013-12-03 13:41:23 -04:00
Joey Hess
e425a966ed Deal with box.com changing the url of their webdav endpoint.
Use new url when making new remotes.

Transparently rewrite old url to new for existing remotes.
2013-12-02 16:01:20 -04:00
Joey Hess
0fd6078865 avoid repeatedly searching path to make batch command when running transferkeys 2013-12-01 15:37:51 -04:00
Joey Hess
2711d13d55 assistant: Run transferkeys as batch jobs. 2013-12-01 14:56:37 -04:00
Joey Hess
03932212ec Avoid using git commit in direct mode, since in some situations it will read the full contents of files in the tree.
The assistant's commit code also always avoids git commit, for simplicity.
Indirect mode sync still does a git commit -a to catch unstaged changes.

Note that this means that direct mode sync no longer runs the pre-commit
hook or any other hooks git commit might call. The git annex pre-commit
hook action for direct mode is however explicitly run. (The assistant
already ran git commit with hooks disabled, so no change there.)
2013-12-01 13:59:45 -04:00
Joey Hess
6edac746f0 merge improved fsck types from git-repair and some associated changes 2013-11-30 14:29:11 -04:00
Joey Hess
a7e1dadf50 fix compat with old yesod 2013-11-25 15:14:34 -04:00
Joey Hess
215fe3c028 one more wrong dir 2013-11-25 14:52:04 -04:00
Joey Hess
ee3ff0e584 typo 2013-11-25 14:47:56 -04:00
Joey Hess
85a82957ea fix sanity check on OSX 2013-11-25 14:46:33 -04:00
Joey Hess
3b2468c824 path somehow wrong 2013-11-25 14:37:44 -04:00
Joey Hess
8a9928742f more build fixes (remove phantom shutdown page) 2013-11-25 14:33:14 -04:00
Joey Hess
38a1259813 improve upgrade paths for OSX 2013-11-25 14:29:14 -04:00
Joey Hess
dabae084e8 fix hdiutil eject 2013-11-25 14:01:41 -04:00
Joey Hess
2398f0983d fix hdiutil mount 2013-11-25 13:53:28 -04:00
Joey Hess
6ee56ce685 move sanity check 2013-11-25 13:46:24 -04:00
Joey Hess
dfdffecacf typo 2013-11-25 13:28:52 -04:00
Joey Hess
e21019eac1 only copy git-annex.app directory
The .Trashes directory is not readable.
2013-11-25 13:24:36 -04:00
Joey Hess
88991c4eb2 fix copying of files from mounted dmg 2013-11-25 13:15:27 -04:00
Joey Hess
a9f2785b86 fix hdutil syntax 2013-11-25 13:10:45 -04:00
Joey Hess
5163d5fb39 ensure that Android TH splice is expanded in native side of build 2013-11-25 13:01:53 -04:00
Joey Hess
7dc02b0c91 lifyH getYesod, to work with older version 2013-11-25 12:54:59 -04:00
Joey Hess
dd7310bcfa osx build fix 2013-11-24 16:08:52 -04:00
Joey Hess
5e470957c6 fix osx build 2013-11-24 16:04:12 -04:00
Joey Hess
f913deab78 move programPath out of Config.Files to Annex.Path
This works around horribleness in the Mavericks cpp, which falls over on
the #if when configure is running. Moving it avoids the file being built at
that point.

But it's also a location that makes sense..
2013-11-24 16:03:03 -04:00
Joey Hess
399ef340f3 completely untested OSX upgrade code 2013-11-24 15:53:15 -04:00
Joey Hess
12fd08be81 tested multi-daemon upgrade 2013-11-24 15:20:18 -04:00
Joey Hess
95feec24d1 create directory for upgraded versio early as a locking mechanism 2013-11-24 15:03:50 -04:00
Joey Hess
5ff5d0a854 cleanup on failed upgrade 2013-11-24 14:04:03 -04:00
Joey Hess
542ae4a855 show version in upgrade alert 2013-11-24 13:28:34 -04:00
Joey Hess
6165284e39 add support for fully automatic upgrades
The Upgrader avoids checking for upgrades on startup when it was just
upgraded. This avoids an upgrade loop if something goes wrong. One example
of something going wrong would be if the upgrade info file and the
distribution file get out of sync (or the distribution file is cached in
a proxy), so it thinks it has upgraded to a new version, but has really
not.
2013-11-24 13:20:58 -04:00
Joey Hess
f04786f984 recusive rmdir 2013-11-24 12:56:49 -04:00
Joey Hess
b59998f73d better handling of upgrade directory 2013-11-24 12:49:03 -04:00
Joey Hess
4776e1d7b7 use manifest file instead of blindly removing the whole git-annex.linux directory
FIXME: dirContentsRecursive does not find empty directories
2013-11-24 01:41:13 -04:00
Joey Hess
fead2941cd linux upgrade code debugged and working 2013-11-24 00:26:20 -04:00
Joey Hess
fdc10b9436 completely untested linux upgrade code 2013-11-23 23:45:49 -04:00
Joey Hess
fda641d27b improve android upgrade interface
and refactor
2013-11-23 22:12:36 -04:00
Joey Hess
e563c7e6f4 fsck distribution key 2013-11-23 21:58:39 -04:00
Joey Hess
32acf908bb queue and start download of git-annex from web, using git-annex, when upgrade is started 2013-11-23 17:21:04 -04:00
Joey Hess
6802123f7d use global webapp redirects when shutting down 2013-11-23 16:21:09 -04:00
Joey Hess
b1a89c448a replace horrible old daemon restart code with new method 2013-11-23 15:50:17 -04:00
Joey Hess
183f7355cd global webapp redirects, to finish upgrades
When an automatic upgrade completes, or when the user clicks on the upgrade
button in one webapp, but also has it open in another browser window/tab,
we have a problem: The current web server is going to stop running in
minutes, but there is no way to send a redirect to the web browser to the
new url.

To solve this, used long polling, so the webapp is always listening for
urls it should redirect to. This allows globally redirecting every open
webapp. Works great! Tested with 2 web browsers with 2 tabs each.
May be useful for other purposes later too, dunno.

The overhead is 2 http requests per page load in the webapp. Due to yesod's
speed, this does not seem to noticibly delay it. Only 1 of the requests
could possibly block the page load, the other is async.
2013-11-23 14:47:38 -04:00
Joey Hess
d24f7f94fe better UI flow through upgrade process
Move button to enable automatic upgrades to an alert displayed after
successful upgrade. Unclutters the UI and makes psychological sense.
2013-11-23 13:27:52 -04:00
Joey Hess
6abaf19c41 restart on upgrade is working, including automatic restart
Made alerts be able to have multiple buttons, so the alerts about upgrading
can have a button that enables automatic upgrades.

Implemented automatic upgrading when the program file has changed.

Note that when an automatic upgrade happens, the webapp displays an alert
about it for a few minutes, and then closes. This still needs work.
2013-11-23 00:54:08 -04:00
Joey Hess
56e980215f got assistant upgrade detection to notice when I build a new version with cabal build! 2013-11-22 23:53:24 -04:00
Joey Hess
7cd9f909e4 restart on upgrade now fully working 2013-11-22 23:42:16 -04:00