Commit graph

1918 commits

Author SHA1 Message Date
Joey Hess
1600ed1ab9 prep release 2014-06-13 09:58:51 -04:00
Joey Hess
6ead75a116 changelog update 2014-06-12 15:31:50 -04:00
Joey Hess
b291951180 merge in windows loststamp branch 2014-06-12 15:21:10 -04:00
Joey Hess
4fe2e53f5b finish fixing windows timezone madness
Rather than calculating the TSDelta once, and caching it, this now
reads the inode sential file's InodeCache file once, and then each time a
new InodeCache is generated, looks at the sentinal file to get the current
delta.

This way, if the time zone changes while git-annex is running, it will
adapt.

This adds some inneffiency, but only on Windows, and only 1 stat per new
file added. The worst innefficiency is that `git annex status` and
`git annex sync` will now (on Windows) stat the inode sentinal file once per
file in the repo.

It would be more efficient to use getCurrentTimeZone, rather than needing
to stat the sentinal file. This should be easy to do, once the time
package gets my bugfix patch.

This commit was sponsored by Jürgen Lüters.
2014-06-12 13:54:08 -04:00
Joey Hess
1c18056e15 deal with FAT on Linux timestamp issue
Deal with FAT's low resolution timestamps, which in combination with
Linux's caching of higher res timestamps while a FAT is mounted, caused
direct mode repositories on FAT to seem to have modified files after they
were unmounted and remounted.

This commit was sponsored by Fabrice Rossi.
2014-06-11 14:46:03 -04:00
Joey Hess
8ff9938d97 Fix build with wai 0.3.0.
This version of wai changed the type of Middleware, so I cannot seem
to liftIO inside it. So, got rid of a lot of not really needed
complexity to use System.Log.Logger's logging stuff, and just use
the standard wai stdout logger when debug logging is enabled.

Format may change some, and it logs http to stdout instead of stderr
now. Doesn't matter for the webapp since both go to the same log anyway.
2014-06-11 01:29:00 -04:00
Joey Hess
6eb5e6c135 Windows: Fix opening webapp when repository is in a directory with spaces in the path. 2014-06-10 18:37:33 -04:00
Joey Hess
03f186609f devblog 2014-06-09 20:16:01 -04:00
Joey Hess
d6711800ad avoid bad commits after interrupted direct mode sync (or merge)
It was possible for a interrupted sync or merge in direct mode to
leave the work tree out of sync with the last recorded commit.
This would result in the next commit seeing files missing from the work
tree, and committing their removal.

Now, a direct mode merge happens not only in a throwaway work tree, but using
a temporary index file, and without any commits or index changes
being made until the real work tree has been updated. If the merge is
interrupted, the work tree may have some updated files, but worst case a
commit will redundantly commit changes that come from the merge.

This commit was sponsored by Tony Cantor.
2014-06-09 19:40:28 -04:00
Joey Hess
7671d228af Avoid leaving behind .tmp files when failing in some cases, including importing files to a disk that is full. 2014-06-09 15:24:05 -04:00
Joey Hess
787d9e27ba Ignore setsid failures. 2014-06-09 14:44:18 -04:00
Joey Hess
783975bb8d prep release 2014-06-06 13:06:13 -04:00
Joey Hess
eb86f1338f wip 2014-06-05 15:31:23 -04:00
Joey Hess
1ab3d7c810 Windows: Fix bug introduced in last release that caused files in the git-annex branch to have lines teminated with \r. 2014-06-05 14:57:01 -04:00
Joey Hess
71b2a0a955 webapp: Include ssh port in mangled hostname.
This avoids a collision if different ssh ports are used on the same host
for some reason.

Note that it's ok to change the format of the mangled hostname; unmangling
only extracts the hostname from it, and once ssh is configured for a
mangled hostname, that config is not changed.
2014-06-05 12:50:15 -04:00
Joey Hess
c00b459f3a unused: Avoid checking view branches for unused files.
This avoids a potential slowdown when using lots of views.

I think that it makes sense for unused to ignore (local) view branches,
since these are by definition supposed to be views of an existing branch,
so looking at the branch should be sufficient (and if the view is out of
date and has files that have since been deleted from the branch, the user's
intent is not to preserve those from unused reaping).
2014-06-04 14:03:41 -04:00
Joey Hess
9c0f3ae7e1 assistant: Make sanity checker tmp dir cleanup code more robust. 2014-05-30 15:08:49 -04:00
Joey Hess
c92d43fb7b avoid stomping on existing description when enabling a special remote
Happened with eg, gcrypt remotes.
2014-05-30 14:30:43 -04:00
Joey Hess
9eaabf0382 webapp: avoid overwriting remote configs when enabling it
Avoid stomping on existing group and preferred content settings
when enabling or combining with an already existing remote.

Two level fix. First, use defaultStandardGroup rather than
setStandardGroup, so if there is an existing configuration in the git-annex
branch, it's not overwritten.

To handle pre-existing ssh remotes (including gcrypt), a second level is
needed, because before syncing with the remote, it's configuration won't be
available locally. (And syncing could take a long time.) So, in this case,
keep track of whether the remote is being created or enabled, and only set
configs when creating it.

This commit was sponsored by Anders Lannerback.
2014-05-30 14:03:04 -04:00
Joey Hess
de0cba18f6 webapp: Avoid stomping on existing group and preferred content settings when adding a local repository (or removable drive repository) that already exists.
This does mean that if the webapp is asked to add a git repository on
a removable drive that already exists, but is not yet a git-annex
repository, it will avoid putting it in any group. That unlikely edge case
is ok; the next step is the edit repository screen, which will show it's
not in any group and the user can pick one.
2014-05-30 12:32:44 -04:00
Joey Hess
2dd274e4ca webapp: When adding a new local repository, fix bug that caused its group and preferred content to be set in the current repository, even when not combining.
There was a tricky bit here, when it does combine, the edit form is shown,
and so the info needs to be committed to the new repository, but then
pulled into the current one. And caches need to be invalidated for it
to be visible in the edit form.
2014-05-29 20:17:05 -04:00
Joey Hess
bc6f0a4dfe prep release 2014-05-29 14:42:52 -04:00
Joey Hess
5a0779fd70 remove newlines from static js and css
This works around a bug in the EvilSplicer; see
http://git-annex.branchable.com/bugs/Bootstrap3_icons_missing_on_Android/
2014-05-28 18:21:01 -04:00
Joey Hess
02a4cef66c update for change to exceptions 2014-05-28 17:23:23 -04:00
Joey Hess
a41ddabd73 webapp: When setting up a ssh remote, if the user inputs ~/foo, normalize that to foo, since it's in the home directory by default. 2014-05-27 14:33:27 -04:00
Joey Hess
95ca3bb022 Fix encoding of data written to git-annex branch. Avoid truncating unicode characters to 8 bits.
Allow any encoding to be used, as with filenames (but utf8 is the sane
choice). Affects metadata and repository descriptions, and preferred
content expressions.

The question of what's the right encoding for the git-annex branch is a
vexing one. utf-8 would be a nice choice, but this leaves the possibility
of bad data getting into a git-annex branch somehow, and this resulting in
git-annex crashing with encoding errors, which is a failure mode I want to
avoid.

(Also, preferred content expressions can refer to filenames, and filenames
can have any encoding, so limiting to utf-8 would not be ideal.)

The union merge code already took care to not assume any encoding for a
file. Except it assumes that any \n is a literal newline, and not part of
some encoding of a character that happens to contain a newline. (At least
utf-8 avoids using newline for anything except liternal newlines.)
Adapted the git-annex branch code to use this same approach.

Note that there is a potential interop problem with Windows, since
FileSystemEncoding doesn't work there, and instead things are always
decoded as utf-8. If someone uses non-utf8 encoding for data on the
git-annex branch, this can lead to an encoding error on windows. However,
this commit doesn't actually make that any worse, because the union merge
code would similarly fail with an encoding error on windows in that
situation.

This commit was sponsored by Kyle Meyer.
2014-05-27 14:16:33 -04:00
Joey Hess
44a3e76115 android: Run busybox install with -s, since some versions of Android prohibit making hard links. 2014-05-23 16:18:55 -04:00
Joey Hess
2e1179df46 webapp: When setting up a ssh remote, record it using initremote, so that it can be easily enabled elsewhere.
This is the capstone in making the webapp remember ssh remotes
so they can be easily enabled in other clones of the repository.

Currently, the user will need to enter a password to enable the ssh remote,
but everything else is filled in automatically.

This commit was sponsored by Peter Lloyd.
2014-05-22 14:57:42 -04:00
Joey Hess
c11461b860 webapp: Support for enabling known git repositories on ssh servers.
The repository must have been added using initremote.

Turned out to be much much simpler than expected, because I was able to
reuse the existing code for enabling rsync and gcrypt remotes, which
was already sufficiently general that it will also work for ssh remotes.
Total win!

This commit was sponsored by an unknown bitcoin contributor.
2014-05-22 14:10:48 -04:00
Joey Hess
c07343e4f7 initremote/enableremote: Basic support for using with regular git remotes
initremote stores the location of an already existing git remote, and
enableremote setups up a remote using its stored location.
2014-05-22 13:42:17 -04:00
Joey Hess
9adcf8fff5 webapp: More robust startup when annex directory is not a git repo or not intialized by git-annex. 2014-05-21 15:41:54 -04:00
Joey Hess
fed509fb3e assistant: When there are multiple remotes giving different ways to access the same repository, honor remote cost settings and use the cheapest available.
Note that TransferInfo does not always contain the Remote, although
any transfer added to the TransferQueue does have a Remote in its
TransferInfo. The transferkeys command still accepts a UUID, which is
useful to handle upgrades, where an old assistant version runs the new
transferkeys.

This commit was sponsored by Kalle Svensson.
2014-05-19 16:19:33 -04:00
Joey Hess
1600b2728e prep release 2014-05-17 13:31:42 -04:00
Joey Hess
5903d20d5e Support symlinking git-annex and git-annex-shell from the Linux standalone bundle into PATH.
Unless busybox doesn't support readlink -f, then it just uses readlink and
symlinking won't work. Also, OSX has no readlink -f so not done there.

Thanks, jlebar.
2014-05-16 16:22:56 -04:00
Joey Hess
dd6b32e9a2 changelog 2014-05-16 15:47:07 -04:00
Joey Hess
00986d19f4 group: When no groups are specified to set, lists the current groups of a repository. 2014-05-16 14:43:40 -04:00
Joey Hess
1a40a66d72 webapp: Fixed drag and drop to reorder the list of remotes.
Broken by 958312885f, in November!
I missed this because there's no strong type checking across the AJAX call. :(

Need to switch to Fay to avoid such bugs..
2014-05-15 22:01:32 -04:00
Joey Hess
b8a1bc4daf update for ddar special remote 2014-05-15 16:37:46 -04:00
Joey Hess
6641f947b2 add git-remote-gcrypt to build depends
This ensures it's pulled in when building the standalone tarball
2014-05-15 15:40:36 -04:00
Joey Hess
da202eb008 remove ssh-askpass recommends; close bugs obsoleted by sshpassword branch 2014-05-15 15:25:36 -04:00
Joey Hess
9384f0b27e changelog 2014-05-15 14:47:04 -04:00
Joey Hess
8cf3534194 gcrypt needs a new version, which I am releaseing 2014-05-15 14:36:39 -04:00
Joey Hess
85e9e8c0cf webapp: Better ssh password prompting.
When setting up a remote on a ssh server, prompt for a password inside the
webapp, rather than relying on ssh's own password prompting in the terminal
the webapp was started from, or ssh-askpass.

Avoids double prompting for the ssh password (and triple-prompting on
windows for rsync.net), since the entered password is cached for 10 minutes
and this cached password is reused when setting up the repository, after
the initial probe.

When the user has an existing ssh key set up, they can choose to use it,
rather than entering a password. The webapp used to probe for this case
automatically, so this is a little harder, but it's an advanced user thing.

Note that this commit is known to break enabling existing rsync
repositories. It hs not been tested with gcrypt repositories. It's not been
successfully tested yet on Windows.

This commit was sponsored by Ralph Mayer.
2014-05-14 15:38:32 -04:00
Joey Hess
2427832bed relicense general utility library code to BSD
Omitted a couple of files what have had significant contributions from
others.
2014-05-10 11:01:27 -03:00
Joey Hess
fe71815f10 add copyright for Sören Brunk for his excellent work 2014-05-02 15:36:41 -03:00
Joey Hess
2aed2d8510 Merge branch 'master' into bootstrap3
Conflicts:
	debian/changelog
2014-05-02 15:32:49 -03:00
Joey Hess
45e794e381 changelog 2014-05-02 15:32:26 -03:00
Joey Hess
2920554787 updated license for bootstrap 3 2014-04-29 15:03:18 -04:00
Joey Hess
90eca40df2 assistant: Clean up stale tmp files on startup.
This commit was sponsored by Melissa Binde.
2014-04-27 22:22:46 -03:00
Joey Hess
f80f0e12fe Simplified repository description line format. The remote name, if any, is always in square brackets after the description. 2014-04-26 21:28:49 -03:00
Joey Hess
ca496b6a97 Standalone builds now check gpg signatures before upgrading. 2014-04-23 13:30:30 -04:00
Joey Hess
6fbf4ab776 prep release 2014-04-20 19:46:01 -04:00
Joey Hess
eba3a28a28 webapp: Support using git-annex on a remote server, which was installed from the standalone tarball or OSX app, and so does not have git-annex in PATH (and may also not have git or rsync in PATH).
* webapp: Support using git-annex on a remote server, which was installed
  from the standalone tarball or OSX app, and so does not have
  git-annex in PATH (and may also not have git or rsync in PATH).
* standalone tarball, OSX app: Install a ~/.ssh/git-annex-wrapper, which
  can be used to run git-annex, git, rsync, etc.
2014-04-20 18:39:10 -04:00
Joey Hess
a8bd7a607d When init detects that git is not configured to commit, and sets user.email to work around the problem, also make it set user.name.
I was able to reproduce git failing to commit despite user.email being set,
in a test account on my laptop. The account had no GECOS information.
2014-04-20 14:17:57 -04:00
Joey Hess
9724667a3d webapp: Fix UI for removing XMPP connection. 2014-04-20 12:46:33 -04:00
Joey Hess
ecc3dc8433 findref: New command, like find but shows files in a specified git ref. 2014-04-17 18:41:24 -04:00
Joey Hess
41c7aaa65c typo 2014-04-17 17:11:18 -04:00
Joey Hess
4ce64ff998 webapp: Start even if the current directory is listed in ~/.config/git-annex/autostart but no longer has a git repository in it. 2014-04-17 16:47:49 -04:00
Joey Hess
f00cb21037 Bring back rsync -p, but only when git-annex is running on a non-crippled file system. This is a better approach to fix #700282 while not unncessarily losing file permissions on non-crippled systems. 2014-04-17 14:31:42 -04:00
Joey Hess
584ab89ea2 drop --from: When local repository is untrusted, its copy of a file does not count. 2014-04-17 13:31:39 -04:00
Joey Hess
915d038bec reinit: New command that can initialize a new reposotory using the configuration of a previously known repository. Useful if a repository got deleted and you want to clone it back the way it was. 2014-04-15 20:13:35 -04:00
Joey Hess
162565d928 info: Allow use in a repository where annex.uuid is not set. 2014-04-15 19:42:49 -04:00
Joey Hess
03ce5cd8d2
found a way to make uninit always fast
To do so, I slightly changed the behavior of unannex. Now in fast mode, it
only makes a hard link when the annexed file's link count is 1. This avoids
unannexing 2 files with the same content in fast mode from hard linking
them together. (One will end up hard linked to the annex, which the docs
warn about.)

With that change, uninit can simply always run unannex in fast mode. Since
.git/annex/objects is being blown away anyway, there's no worry in this
case about a hard link pointing into it causing an annexed object to be
modified.
2014-04-15 14:23:08 -04:00
Joey Hess
b70cd37a6f it's landed! merge remotecontrol into master 2014-04-14 14:22:44 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
ab09a50f3a Avoid depending on shakespeare except for when building the webapp. 2014-04-13 13:12:14 -04:00
Joey Hess
a33b30d0c4 remotedaemon: When network connection is lost, close all cached ssh connections.
This commit was sponsored by Cedric Staub.
2014-04-12 16:32:59 -04:00
Joey Hess
15917ec1a8 sync, assistant, remotedaemon: Use ssh connection caching for git pushes and pulls.
For sync, saves 1 ssh connection per remote. For remotedaemon, the same
ssh connection that is already open to run git-annex-shell notifychanges
is reused to pull from the remote.

Only potential problem is that this also enables connection caching
when the assistant syncs with a ssh remote. Including the sync it does
when a network connection has just come up. In that case, cached ssh
connections are likely to be stale, and so using them would hang.
Until I'm sure such problems have been dealt with, this commit needs to
stay on the remotecontrol branch, and not be merged to master.

This commit was sponsored by Alexandre Dupas.
2014-04-12 15:59:34 -04:00
Joey Hess
96ce2812e0 Merge branch 'master' into remotecontrol
Conflicts:
	debian/changelog
2014-04-12 14:20:41 -04:00
Joey Hess
d955cfe78a Improve handling on monthly/yearly scheduling.
Code was still buggy, it turns out (though the recursion checker caught
it). In the case of (Schedule (Monthly Nothing) AnyTime), where the last
run was on yyyy-12-31, it looped forever.

Also, the handling of (Schedule (Yearly Nothing) AnyTime) was wacky where
the last run was yyyy-12-31. It would suggest a window starting on the 3rd
for the next run (because 31 mod 28 is 3).

I think that originally I was wanted to avoid running on 01-01 if it had
just run on 12-31. But the code didn't accomplish this, and it's not
necessary anyway. This is supposed to calculate the next window meeting the
schedule, and for (Schedule (Monthly Nothing), the window starts at 01-01
and runs through 01-31. If that causes two back-to-back runs, well the next
one will not be until 02-01 at the earliest.

Also, back-to-back runs can be avoided, if desired, by using Divisible 2.
2014-04-11 21:42:43 -04:00
Joey Hess
e26dead493 guard against any remaining infinite recursion bugs; throw error if no candidate cn be found in next hundred years
Note that the exception thrown is not visible in the webapp currently
because it crashes one of Cronner's 2 worker threads, which is never
checked.
2014-04-11 18:08:46 -04:00
Joey Hess
9e16171c7f Last release didn't quite fix the high cpu issue in all cases, this should.
This is supposed to look for a day past the last day it ran, not a month
past.

Seems to work, at least in anarcat's test case.
2014-04-11 17:14:19 -04:00
Joey Hess
91fa570aad more prep 2014-04-11 15:10:19 -04:00
Joey Hess
bdb81af880
prep release 2014-04-11 15:01:10 -04:00
Joey Hess
c678798a5c assistant: Fix high CPU usage triggered when a monthly fsck is scheduled, and the last time the job ran was a day of the month > 12. This caused a runaway loop. Thanks to Anarcat for his assistance, and to Maximiliano Curia for identifying the cause of this bug. 2014-04-11 14:34:09 -04:00
Joey Hess
a6df4698a3 Fix rsync progress parsing in locales that use comma in number display. Closes: #744148 2014-04-10 15:36:51 -04:00
Joey Hess
407d60ca5d Remove wget from OSX dmg, due to issues with cert paths that broke git-annex automatic upgrading. Instead, curl is used, unless the OSX system has wget installed, which will then be used. 2014-04-10 12:54:58 -04:00
Joey Hess
db38678595 webapp: Rework xmpp nudge to prompt for either xmpp or a ssh remote be set up.
This commit was sponsored by Nathan Howell.
2014-04-09 16:27:24 -04:00
Joey Hess
14349fb752 also show signal icon next to connected xmpp remotes 2014-04-09 15:51:00 -04:00
Joey Hess
33b8cff433 webapp: Show a network signal icon next to ssh remotes that it's currently connected with. 2014-04-09 15:26:41 -04:00
Joey Hess
fac7bca05b assistant: Now detects immediately when other repositories push changes to
a ssh remote, and pulls.

XMPP is no longer needed in this configuration!

Requires the remote server have git-annex-shell with notifychanges support.

(untested)

This commit was sponsored by Geog Wechslberger.
2014-04-08 15:23:50 -04:00
Joey Hess
836f2cc816 importfeed: Filename template can now contain an itempubdate variable. Needs feed 0.3.9.2. 2014-04-07 16:55:04 -04:00
Joey Hess
aff7691d8d prepping for a linux-only release 2014-04-05 16:57:05 -04:00
Joey Hess
ade5d8ccc5 Fix nautilus script installation to not crash when the nautilus script dir does not exist. Instead, only install scripts when the directory already exists.
nautilus creates this directory each time run, so this avoids putting cruft
in user's home
2014-04-05 16:36:58 -04:00
Joey Hess
66f7354dae
credit 2014-04-05 16:28:35 -04:00
Joey Hess
0fbbec261d git-annex-shell: Added notifychanges command.
This will be used by the remote-daemon to quickly tell when changes have
been pushed from some other repository into a ssh remote.

Adjusted the remote-daemon protocol to communicate changed shas, rather
than git branch refs. This way, it can easily check if a sha is new.

This commit was sponsored by Carlos Trijueque Albarran.
2014-04-05 16:10:39 -04:00
Joey Hess
aad6efe890 prep release 2014-04-02 16:45:04 -04:00
Joey Hess
c09fd6443e
fix fish 2014-04-02 15:37:43 -04:00
Joey Hess
39996f8222 assistant: Adjust ssh authorized keys line generated in local pairing or for a remote ssh server to set environment variables in an alternative way that works with the non-POSIX fix shell, as well as POSIX shells. 2014-04-02 15:30:10 -04:00
Joey Hess
065248f3d2 Added required content configuration.
This includes checking when dropping files that any required content
configuration is satisfied. However, it does not yet include an active
check on the required content; the location log is trusted when checking
the required content expression.
2014-03-29 16:03:33 -04:00
Joey Hess
3b8d5f03bb Fix glacier repo creation bug
Version 5.20140227 broke creation of glacier repositories, not including
the datacenter and vault in their configuration. This bug is fixed, but
glacier repositories set up with the broken version of git-annex need to
have the datacenter and vault set in order to be usable. This can be done
using git annex enableremote to add the missing settings. For details, see
http://git-annex.branchable.com/bugs/problems_with_glacier/
2014-03-27 14:30:36 -04:00
Joey Hess
2f538dd65c add --include-dotfiles: New option, perhaps useful for backups. 2014-03-26 14:52:07 -04:00
Joey Hess
72978a8de0 git-annex-shell: Make configlist automatically initialize a remote git repository, as long as a git-annex branch has been pushed to it, to simplify setup of remote git repositories, including via gitolite. 2014-03-26 14:22:21 -04:00
Joey Hess
0df4848a9e forget --drop-dead: Avoid removing the dead remote from the trust.log, so that if git remotes for it still exist anywhere, git annex info will still know it's dead and not show it. 2014-03-26 13:28:26 -04:00
Joey Hess
27d27b4da2 tahoe: Pass -d parameter before subcommand; putting it after the subcommand no longer works with tahoe-lafs version 1.10. (Thanks, Alberto Berti) 2014-03-26 12:59:54 -04:00
Joey Hess
f6eb967f1e webapp: Automatically install Nautilus integration scripts to get and drop files.
This commit was sponsored by Gian-Maria Daffre.
2014-03-22 15:51:30 -04:00
Joey Hess
fb8a32cc7f notifications on drop 2014-03-22 15:01:48 -04:00
Joey Hess
e426fac273 add desktop notifications
Motivation: Hook scripts for nautilus or other file managers
need to provide the user with feedback that a file is being downloaded.

This commit was sponsored by THM Schoemaker.
2014-03-22 14:12:19 -04:00
Joey Hess
7dc6804154 unannex, uninit: Avoid committing after every file is unannexed, for massive speedup.
pre-commit hook lock added, so unannex can prevent the hook from running
in a confusing state.

This commit was sponsored by Fredrik Hammar
2014-03-21 14:41:05 -04:00
Joey Hess
2230828779 prep release 2014-03-20 13:24:24 -04:00
Joey Hess
ed30b81e2c Improve behavior when unable to parse a preferred content expression (thanks, ion).
Fall back to "present" as the preferred conent expression, which will
not result in any content movement.
2014-03-20 00:10:12 -04:00
Joey Hess
f64c2d6138 toplevel lastchanged field 2014-03-19 19:10:55 -04:00
Joey Hess
1052eeface Windows: Fix some filename encoding bugs.
http://git-annex.branchable.com/bugs/Unicode_file_names_ignored_on_Windows/

Not a complete fix yet.
2014-03-19 15:57:56 -04:00
Joey Hess
caa97d1271 Each for each metadata field, there's now an automatically maintained "$field-lastchanged" that gives the timestamp of the last change to that field.
Note that this is a nearly entirely free feature. The data was already
stored in the metadata log in an easily accessible way, and already was
parsed to a time when parsing the log. The generation of the metadata
fields may even be done lazily, although probably not entirely (the map
has to be evaulated to when queried).
2014-03-18 18:55:43 -04:00
Joey Hess
40b599eff2 rsync special remote: Fix slashes when used on Windows. 2014-03-18 13:02:10 -04:00
Joey Hess
fb9ddfa231 map: Fix crash when one of the remotes of a repo is a local directory that does not exist, or is not a git repo. 2014-03-17 15:44:42 -04:00
Joey Hess
b4d12855d6
close 2014-03-17 15:26:18 -04:00
Joey Hess
8bcd67b9d8 metadata: Add --get (from bremner) 2014-03-15 17:29:40 -04:00
Joey Hess
3a9a1b9f63 doc updates for groupwanted 2014-03-15 16:39:13 -04:00
Joey Hess
417aea25be vicfg: Allows editing preferred content expressions for groups.
This is stored in the git-annex branch, but not yet actually hooked up and
used.
2014-03-15 16:17:01 -04:00
Joey Hess
b7eb1d834a Avoid encoding errors when using the unused log file. 2014-03-15 11:57:27 -04:00
Joey Hess
3551d40b05 "standard" can now be used as a first-class keyword in preferred content expressions.
For example "standard or (include=otherdir/*)" or even "not standard"

Note that the implementation avoids any potential for loops (if a
standard preferred content expression itself mentioned standard).

This commit was sponsored by Jochen Bartl.
2014-03-14 15:04:33 -04:00
Joey Hess
b7b9afa9ad Better workaround for problem umasks when eg, setting up ssh keys. 2014-03-14 13:37:58 -04:00
Joey Hess
ba02cd8a80 Fix ssh connection caching stop method to work with openssh 6.5p1, which broke the old method.
Old ssh did not check the hostname passed to -O stop, so I had used "any".
But now ssh does check it! I think this happened as part of the client-side
hostname canonicalization changes in 6.5p1, but have not verified that
introduced the problem.

The symptom was that it would try to dns lookup "any", which often caused a
bit of a delay at shutdown. And the old ssh connection kept running, so
it would do it over and over again.

Fixed by using localhost, which hopefully reliably resolves to some address
that ssh will accept.. Also nukeFile the socket after ssh has been asked to
shutdown, just in case.
2014-03-13 19:35:06 -04:00
Joey Hess
005e8744c1 Commands that allow specifying which repository to act on using the repository's description will now fail when multiple repositories match, rather than picking a repository at random.
So will --in=

Note that since limitIn is not used by preferred content expressions,
it's ok for it to throw an exception.
2014-03-13 15:36:58 -04:00
Joey Hess
4cc34973d9 copy --fast --to remote: Avoid printing anything for files that are already believed to be present on the remote. 2014-03-13 14:51:22 -04:00
Joey Hess
66b8b9c094 webapp: Use securemem for constant time auth token comparisons.
Debian stable does not have securemem, but neither does it have warp-tls,
so just disable use of securemem when not building with https support.
2014-03-12 21:41:20 -04:00
Joey Hess
0e0d396b27 Improve memory usage when git fsck finds a great many broken objects.
From 1.7 gb to 900 mb on 300 thousand unique reported shas.

When shas are not unique, this streams much better than before, so won't
buffer the full list before putting them into the Set and throwing away
dups. And when fsck output includes ignorable lines, especially
dangling object lines, they won't be buffered in memory at all.
2014-03-10 15:14:09 -04:00
Joey Hess
a10b5b19ad enable haskell-dns library, rather than needing "host" command
Done on Debian to keep the autobuilds and Debian packages the same,
but can be reverted for backports, since haskell-dns is not available
there.
2014-03-07 13:33:24 -04:00
Joey Hess
1f99a6778f Fix direct mode getKeysPresent false positive & also sped up direct mode unused and unannex
unused: In direct mode, files that are deleted from the work tree are no longer incorrectly detected as unused.

Direct mode `git annex info` slows down a bit due to more stringent
checking, but not by a lot.
2014-03-07 12:43:56 -04:00
Joey Hess
54746463bb webapp: Added a "Sync now" item to each repository's menu. 2014-03-06 18:11:44 -04:00
Joey Hess
4d06037fdd Fix zombie leak and general inneficiency when copying files to a local git repo.
Benchmarking this with 1000 small files being copied, the time reduced from
15.98s to 14.64s -- an 8% improvement in the non-data-transfer overhead of
git-annex copy.
2014-03-06 17:13:27 -04:00
Joey Hess
93a526ba82 prep release 2014-03-06 12:29:11 -04:00
Joey Hess
d32289bab0 assistant: Smarter log file rotation, which takes free disk space into account. 2014-03-05 18:26:53 -04:00
Joey Hess
a3fe8270ca annex.startupscan can be set to false to disable the assistant's startup scan. 2014-03-05 17:44:14 -04:00
Joey Hess
74d3949cd0 webapp: Include no-pty in ssh authorized_keys lines.
git-annex-shell does not need a pty, so this speeds things up.
Also, it may avoid weird misconfigured systems that try to run screen or
tmux on every ssh login from doing so.
2014-03-05 13:53:21 -04:00
Joey Hess
aa377ed567 webdav: When built with a new enough haskell DAV (0.6), disable the http response timeout, which was only 5 seconds. 2014-03-05 13:51:54 -04:00
Joey Hess
1f98d6fb00 glacier: Pass --region to glacier checkpresent.
I suppose this is not necessary when it has a local cache, so I didn't
notice it was missing.
2014-03-04 23:22:24 -04:00
Joey Hess
14d1e878ab sync: Automatically resolve merge conflict between and annexed file and a regular git file.
This is a new feature, it was not handled before, since it's a bit of an
edge case. However, it can be handled exactly the same as a file/dir
conflict, just leave the non-annexed item alone.

While implementing this, the core resolveMerge' function got a lot simpler
and clearer. Note especially that where before there was an asymetric call to
stagefromdirectmergedir, now graftin is called symmetrically in both cases.

And, in order to add that `graftin us`, the current branch needed to be
known (if there is no current branch, there cannot be a merge conflict).
This led to some cleanups of how autoMergeFrom behaved when there is no
current branch.

This commit was sponsored by Philippe Gauthier.
2014-03-04 19:35:55 -04:00
Joey Hess
4a847cdc08 finish fixing direct mode merge bug involving unstaged local files
Added test cases for both ways this can happen, with a conflict involving a
file, or a directory.

Cleaned up resolveMerge to not touch the work tree in direct mode, which
turned out to be the only way to handle things.. And makes it much nicer.

Still need to run test suite on windows.
2014-03-04 02:03:15 -04:00
Joey Hess
1192d98721 sync: Fix bug in direct mode that caused a file not checked into git to be deleted when merging with a remote that added a file by the same name. (Thanks, jkt) 2014-03-03 14:57:16 -04:00
Joey Hess
d0fce426c4 pre-commit-annex hook script to automatically extract metadata from lots of types of files
Using the extract(1) program to do the heavy lifting.

Decided to make git-annex run pre-commit-annex when committing. Since
git-annex pre-commit also runs it, it'll be run when git commit is run too,
via the pre-commit hook. This basically gives back the pre-commit hook
that git-annex took away. The implementation avoids repeatedly looking
for the hook script when the assistant is running and committing
repeatedly; only checks if the hook is available once.

To make the script simpler, made git-annex metadata -s field?=value
only set a field when it's not already got a value.

This commit was sponsored by bak.
2014-03-02 20:11:58 -04:00
Joey Hess
0ac8912f9c view: Refuse to enter a view when no branch is currently checked out. 2014-03-02 16:00:56 -04:00
Joey Hess
7d9486a709 vadd: Allow listing multiple desired values for a field. 2014-03-02 15:36:45 -04:00
Joey Hess
c2e8c21ca6 view, vfilter: Add support for filtering tags and values out of a view, using !tag and field!=value.
Note that negated globs are not supported. Would have complicated the code
to add them, without changing the data type serialization in a
non-backwards-compatable way.

This commit was sponsored by Denver Gingerich.
2014-03-02 14:53:19 -04:00
Joey Hess
6a355686ff annex.listen can be configured, instead of using --listen 2014-03-01 00:31:17 -04:00
Joey Hess
d41367cc08 remove buggy --listen=host:port support 2014-02-28 22:53:26 -04:00
Joey Hess
cdc5b4a385 depend on warp-tls.. at least for now 2014-02-28 21:37:33 -04:00
Joey Hess
6469c1aca9 webapp: Don't list the public repository group when editing a git repository; it only makes sense for special remotes. 2014-02-28 20:37:03 -04:00
Joey Hess
c8528864e1 assistant --autostart: Refuse to start in a bare git repository. 2014-02-28 19:47:05 -04:00
Joey Hess
f70d0479d5 webapp: Refuse to start in a bare git repository. 2014-02-28 19:44:15 -04:00
Joey Hess
1681386b0d webapp: Filter out from Switch Repository list any repositories listed in autostart file that don't have a git directory anymore.
Trying to start in such a repo will, obviously, fail.

Note that assistant --autostart will try to start in such a repo, and fail,
but does start successfully in the other autostart repos.
2014-02-28 19:16:49 -04:00
Joey Hess
7ac37a7854 Probe for quvi version at run time.
Overhead: git annex addurl runs quvi --version once.
And more bloat to Annex state..
2014-02-28 14:54:02 -04:00
Joey Hess
0609d754cc prep release 2014-02-27 11:36:38 -04:00
Joey Hess
a1432bce2f Put non-object tmp files in .git/annex/misctmp, leaving .git/annex/tmp for only partially transferred objects.
This allows eg, putting .git/annex/tmp on a ram disk, if the disk IO
of temp object files is too annoying (and if you don't want to keep
partially transferred objects across reboots).

.git/annex/misctmp must be on the same filesystem as the git work tree,
since files are moved to there in a way that will not work cross-device,
as well as symlinked into there.

I first wanted to put the tmp objects in .git/annex/objects/tmp, but
that would pose transition problems on upgrade when partially transferred
objects existed.

git annex info does not currently show the size of .git/annex/misctemp,
since it should stay small. It would also be ok to make something clean it
out, periodically.
2014-02-26 16:52:56 -04:00
Joey Hess
a19ad12018 amusing caseo 2014-02-26 15:45:14 -04:00
Joey Hess
3b35e3f88a Disable test suite on sparc, which is missing optparse-applicative. 2014-02-26 13:44:28 -04:00
Joey Hess
692fbc6d7e Add build dep on regex-compat to fix build on mipsel, which lacks regex-tdfa. 2014-02-26 13:41:33 -04:00
Joey Hess
06e9080f01 metadata: FIeld names are now case insensative. 2014-02-25 18:45:09 -04:00
Joey Hess
3bf5183136 reorg 2014-02-25 17:25:53 -04:00
Joey Hess
2b66aaa763 Windows webdav: Fix DOS path separator bug.
Use posix </> etc for urls.
2014-02-25 15:26:33 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
c69d6eb035 Make annex.web-options be used in several places that call curl. 2014-02-24 21:29:37 -04:00
Joey Hess
46cc39f1a4 repair: Optimise unpacking of pack files, and avoid repeated error messages about corrupt pack files. 2014-02-24 19:36:58 -04:00
Joey Hess
d5a2b498f6 webdav: When built with DAV 0.6.0, use the new DAV monad to avoid locking files, which is not needed by git-annex's use of webdav, and does not work on Box.com. 2014-02-24 18:21:51 -04:00
Joey Hess
45e7040142 webapp: Fix creation of box.com, S3, and Glacier repositories, broken in 5.20140221. 2014-02-24 15:29:17 -04:00
Joey Hess
8d5158fa31 Preserve metadata when staging a new version of an annexed file.
Performance impact: When adding a large tree of new files, this needs
to do some git cat-file queries to check if any of the files already
existed and might need a metadata copy. I tried a benchmark in a copy
of my sound repository (so there was already a significant git tree
to check against.

Adding 10000 small files, with a cold cache:
  before: 1m48.539s
  after:  1m52.791s

So, impact is 0.0004 seconds per file added. Which seems acceptable, so did
not add some kind of configuration to enable/disable this.

This commit was sponsored by Lisa Feilen.
2014-02-24 14:41:33 -04:00
Joey Hess
fb4e1ebfbe metadata: Support --json 2014-02-23 13:58:16 -04:00
Joey Hess
b437787eee metadata: Field names limited to alphanumerics and a few whitelisted punctuation characters to avoid issues with views, etc. 2014-02-23 13:34:59 -04:00
Joey Hess
7498c5dd96 annex.genmetadata can be set to make git-annex automatically set metadata (year and month) when adding files 2014-02-23 00:08:29 -04:00
Joey Hess
079b35a1a8 views: add automatically constructed file location metadata
When constructing views, metadata is available about the location of the
file in the view's reference branch. Allows incorporating parts of the
directory hierarchy in a view.

For example `git annex view tag=* podcasts/=*` makes a view in the form
tag/showname.

Performance impact: I benchmarked git annex view tag=* in the conference
proceedings repo to take 6.459s before this change, and 6.544s after.

FWIW, I considered making the syntax for this be podcasts/*, which might
be easier for the user to learn. However, I think it's not as good:

* The user has to then juggle two different syntaxes, and podcasts/* will
  be expanded by the shell so they also need to quote it, while podcasts/=*
  is unlikely to be expanded by the shell.
* It would allow for things like podcasts/*/* and *.mp3 which do not
  map well into views.

This commit was sponsored by Aurélien Pinceaux.
2014-02-22 16:27:53 -04:00
Joey Hess
24f8136504 --metadata field=value can now use globs to match, and matches case insensatively, the same as git annex view field=value does.
Also refactored glob code into its own module.
2014-02-21 18:34:34 -04:00
Joey Hess
ded4ab5704 Fix handling of rsync remote urls containing a username, including rsync.net.
This breakage seems to have been caused way back in a1eded86,
but I am pretty sure rsync.net support has not been entirely
broken since last April. AFAICS, the generated .ssh/config
has not changed since then -- it has never included a Username setting
line. So, I am puzzled at when this reversion was introduced.

Note that the breakage only affected checkpresent and remove. Upload and
download use the ssh connection caching, which includes a -l username.
2014-02-21 13:20:57 -04:00
Joey Hess
1f6f034e90 prep release 2014-02-21 11:24:18 -04:00
Joey Hess
9edc3a735d fsck: Refuse to do anything if more than one of --incremental, --more, and --incremental-schedule are given, since it's not clear which option should win. 2014-02-20 15:56:45 -04:00
Joey Hess
7d288d83c9 glacier: Do not try to run glacier value create when an existing glacier remote is enabled. 2014-02-20 15:56:26 -04:00
Joey Hess
21610294da trust, untrust, semitrust, dead: Warn when the trust level is overridden in .git/config. 2014-02-20 15:17:39 -04:00
Joey Hess
134fdefb8c fsck: When run with --all or --unused, while .gitattributes annex.numcopies cannot be honored since it's operating on keys instead of files, make it honor the global numcopies setting, and the annex.numcopies git config setting. 2014-02-20 14:45:17 -04:00
Joey Hess
db48b8a4a3 unused: Fix to actually detect unused keys when in direct mode. 2014-02-20 13:53:49 -04:00
Joey Hess
dd7b99c860 add tip about metadata driven views (and more flexible view filtering)
While writing this documentation, I realized that there needed to be a way
to stay in a view like tag=* while adding a filter like tag=work that
applies to the same field.

So, there are really two ways a view can be refined. It can have a new
"field=explicitvalue" filter added to it, which does not change the
"shape" of the view, but narrows the files it shows.
Or, it can have a new view added, which adds another level of
subdirectories.

So, added a vfilter command, which takes explicit values to add to the
filter, and rejects changes that would change the shape of the view.

And, made vadd only accept changes that change the shape of the view.

And, changed the View data type slightly; now components that can match
multiple metadata values can be visible, or not visible.

This commit was sponsored by Stelian Iancu.
2014-02-19 16:29:56 -04:00
Joey Hess
39ebfa1a2e pre-commit: Update metadata when committing changes to annexed files within a view.
So the user can now switch to a view and then move files around within it
to manage metadata. For example, moving a file into a new directory
when in the tags=* view adds a tag to it.

Implementation is fairly efficient. One diff-index, which is no more
expensive than the first stage of a git commit, followed by possibly
some cat-file --batch traffic to find the key (when deleting a file).
Very similar to what's done in direct mode when committing. And like
direct mode when updating the WC after a merge, it has to buffer the
diff-tree values in order to make 2 passes over them.

When not in a view, pre-commit now does one extra git symbolic-ref,
which is tiny overhead.

This commit was sponsored by Andrew Eskridge.
2014-02-19 14:17:58 -04:00
Joey Hess
67a5f02a0b add vcycle command 2014-02-18 20:16:28 -04:00
Joey Hess
f603692a72 add vadd command 2014-02-18 20:02:09 -04:00
Joey Hess
67fd06af76 add git annex view command
(And a vpop command, which is still a bit buggy.)

Still need to do vadd and vrm, though this also adds their documentation.

Currently not very happy with the view log data serialization. I had to
lose the TDFA regexps temporarily, so I can have Read/Show instances of
View. I expect the view log format will change in some incompatable way
later, probably adding last known refs for the parent branch to View
or something like that.

Anyway, it basically works, although it's a bit slow looking up the
metadata. The actual git branch construction is about as fast as it can be
using the current git plumbing.

This commit was sponsored by Peter Hogg.
2014-02-18 18:22:20 -04:00
Joey Hess
fd09798e40 windows webapp: fix rsync.net support 2014-02-14 15:47:54 -04:00
Joey Hess
d007d1ac0c windows: hack to ensure HOME is always set 2014-02-14 14:03:24 -04:00
Joey Hess
2075cdeb59
limiting files based on metadata
Note that there is currently no caching, so
	--metadata foo=bar --metadata tag=blah
will currently read the log 2x per file.
2014-02-13 02:24:30 -04:00
Joey Hess
cf44fc1acb Windows webapp: Can create repos on removable drives. 2014-02-11 16:15:40 -04:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
e885080d06 Add progress display for transfers to/from external special remotes. 2014-02-10 21:33:22 -04:00
Joey Hess
b3a002f21d prep release 2014-02-10 12:59:18 -04:00
Joey Hess
c95d0cf7a8 Windows: Fix handling of absolute unix-style git repository paths.
Note that on Windows a remote with a path like /home/foo/bar
is interpreted by git as being some screwy relative path (relative to what
exactly seems ill-defined -- it seemed relative to C:\Program Files\Git\ in
my tests!) So no attempt has been made to handle such a path sanely, just not
to crash when encountering it.

Note that "C:\\foo" </> "/home/foo/bar" yields /home/foo/bar even though
that is not absolute! I don't know what to make of all this,
except that I will be very happy when this crock of **** vanishes from
the face of the earth.
2014-02-08 15:39:04 -04:00
Joey Hess
dfcd455860 version DAV dep 2014-02-08 15:04:48 -04:00
Joey Hess
4e7c65dca0 Fix build on platforms not supporting the webapp. 2014-02-08 14:25:18 -04:00
Joey Hess
250df4685c avoid broken version of DAV 2014-02-07 17:42:52 -04:00
Joey Hess
3cbaa68be8 Android: Avoid crashing when unable to set file mode for ssh config file due to Android filesystem horribleness. 2014-02-07 14:57:23 -04:00
Joey Hess
3a3fda5bc0 Fix initremote with encryption=pubkey to work with S3, glacier, webdav, and external special remotes. 2014-02-06 18:25:31 -04:00
Joey Hess
897d877472 work around absNormPath not working on Windows
When making git-annex links, we want unix-style paths in the link targets.
2014-02-06 17:17:35 -04:00
Joey Hess
28cabd9909 Revert "work around absNormPath not working on Windows"
This reverts commit 0fc3ad82c5.

That caused 2 other test cases to fail. Windows path slash issues are
horrible.
2014-02-06 16:59:39 -04:00
Joey Hess
0fc3ad82c5 work around absNormPath not working on Windows
Seems I punted on this while porting before. This hack relies on DOS not
using / in filenames, it's effectively an alternate path separatr in at
least current versions of windows..
2014-02-06 15:06:45 -04:00
Joey Hess
a44e01c29c --in can now refer to files that were located in a repository at some past date. For example, --in="here@{yesterday}" 2014-02-06 12:43:56 -04:00
Joey Hess
751f496c11 add test case & fix conflict resolution bug on Windows & FAT
Fix bug in automatic merge conflict resolution code when used
on a filesystem not supporting symlinks, which resulted in it losing
track of the symlink bit of annexed files.

This was the underlying bug that was causing another test to fail,
which got worked around in 1c997fd08c.
I've chosen to keep 2 separate test cases since the old test case only
detected the problem accidentially.

Test suite passes on FAT & in windows, as well as on proper unix systems.

This commit was sponsored by Ellis Whitehead.
2014-02-04 17:24:12 -04:00
Joey Hess
1c997fd08c fix test suite to pass on windows
On windows, the sync of the second cloned repo to origin failed, because
synced/master was a non-fast-forward. This may be a bug of its own, but
it's not the issue that this test was intended to test, so disconnect
the repos from origin before syncing.
2014-02-03 12:12:50 -04:00
Joey Hess
29bb04aa0d Windows: Fix deletion of repositories by test suite and webapp.
On Windows, a file that is not writable cannot be deleted even if in a
directory with write perms. So git object files were not getting deleted
when removing a git repository.
2014-02-03 11:19:41 -04:00
Joey Hess
f657e4070b reorg 2014-02-03 10:45:28 -04:00
Joey Hess
fded408b44 sync --content: Drop files from remotes that don't have them after getting them.
Need to include the uuid of the local repo in the list of belived locations
of a key after getting it, in order for the drop from remote to include it
in the numcopies calculation.
2014-02-02 22:48:45 -04:00
Joey Hess
4676706756 sync --content: Reuse smart copy code from copy command, including handling and repairing out of date location tracking info. Closes: #737480 2014-02-02 19:57:22 -04:00
Joey Hess
1858c1f44a Document in man page that sshcaching uses ssh ControlMaster. Closes: #737476 2014-02-02 19:27:47 -04:00
Joey Hess
089c0109a2 Added ways to configure rsync options to be used only when uploading or downloading from a remote. Useful to eg limit upload bandwidth. 2014-02-02 16:06:34 -04:00
Joey Hess
15ac2ea4de sync --content: Re-pull from remotes after downloading content, since that can take a while and other changes may be pushed in the meantime. 2014-02-01 10:49:50 -04:00
Joey Hess
bbef0cddfd improve sync with xmpp and annex-ignore
* sync --content: Honor annex-ignore configuration.
* sync: Don't try to sync with xmpp remotes, which are only currently
  supported when using the assistant.
2014-02-01 10:33:55 -04:00
Joey Hess
1669e80e85 Windows: Avoid using unix-compat's rename, which refuses to rename directories.
Opened a bug about this: https://github.com/jystic/unix-compat/issues/10
2014-01-29 15:19:03 -04:00
Joey Hess
721cc0cd22 rework annexed object locking in direct mode & support Windows
Seems that locking of annexed objects when they're being dropped was broken
in direct mode:

* When taking the lock before dropping, it created the .git/annex/objects
  file, as an empty file. It seems that the dropping code deleted that,
  but that is not right, and for all I know could in some situation cause
  a corrupted object to leak out.
* When the lock was checked, it actually tried to open each direct mode
  file, and checked if it was locked. Not the same lock used above, and
  could also fail if some consumer of the file locked it.

Fixed this, and added windows support by switching direct mode to lock a
.lck file.
2014-01-28 16:43:11 -04:00
Joey Hess
891c85cd88 use locking on Windows
This is all the easy cases, where there was already a separate lock file.
2014-01-28 14:42:03 -04:00
Joey Hess
998d76ae04 Revert "temporarily revert tasty-rerun support for this release"
This reverts commit 8fcf041159.
2014-01-27 14:22:44 -04:00
Joey Hess
8fcf041159 temporarily revert tasty-rerun support for this release
revert c340d9506c

tasty-rerun is stuck in NEW in debian
2014-01-27 13:58:43 -04:00
Joey Hess
e0224f01c1 prep release 2014-01-27 13:44:12 -04:00
Joey Hess
ec7443eb06 All commands that support --all also support a --key option, which limits them to acting on a single key. 2014-01-26 14:59:47 -04:00
Joey Hess
85398d068e whereis: Support --all. 2014-01-26 14:26:32 -04:00
Joey Hess
78d0fd389e webapp: After upgrading a git repository to git-annex, fix bug that made it temporarily not be synced with. 2014-01-26 13:57:05 -04:00
Joey Hess
5fc2d760ea Optimise non-bare http remotes; no longer does a 404 to the wrong url every time before trying the right url. Needs annex-bare to be set to false, which is done when initially probing the uuid of a http remote. 2014-01-26 13:03:25 -04:00
Joey Hess
b93e485ef1 added annex.secure-erase-command config option. 2014-01-24 12:58:52 -04:00
Joey Hess
c340d9506c tasty-rerun! make rerest runs much much faster than running whole test suite 2014-01-24 12:21:30 -04:00
Joey Hess
3da0064657 assistant unused file handling
Make sanity checker run git annex unused daily, and queue up transfers
of unused files to any remotes that will have them. The transfer retrying
code works for us here, so eg when a backup disk remote is plugged in,
any transfers to it are done. Once the unused files reach a remote,
they'll be removed locally as unwanted.

If the setup does not cause unused files to go to a remote, they'll pile
up, and the sanity checker detects this using some heuristics that are
pretty good -- 1000 unused files, or 10% of disk used by unused files,
or more disk wasted by unused files than is left free. Once it detects
this, it pops up an alert in the webapp, with a button to take action.

TODO: Webapp UI to configure this, and also the ability to launch an
immediate cleanup of all unused files.

This commit was sponsored by Simon Michael.
2014-01-22 22:53:18 -04:00
Joey Hess
ed7c61914c assistant: Run the periodic git gc in batch mode. 2014-01-22 17:11:41 -04:00
Joey Hess
4b55afe9e9 add "unused" preferred content expression
With a really nice optimisation that keeps it from having any overhead
in normal operation!

This commit was sponsored by Ulises Vitulli.
2014-01-22 16:35:32 -04:00
Joey Hess
f2713a3bb9 benchmarked numcopies .gitattributes in preferred content
Checking .gitattributes adds a full minute to a git annex find looking for
files that don't have enough copies. 2:25 increasts to 3:27. I feel this is
too much of a slowdown to justify making it the default. So, exposed two
versions of the preferred content expression, a slow one and a fast but
approximate one.

I'm using the approximate one in the default preferred content expressions
to avoid slowing down the assistant.
2014-01-21 18:49:25 -04:00
Joey Hess
4a4e2f9a6c use tasty resources 2014-01-21 14:04:50 -04:00
Joey Hess
78ead70ea4 repair: Check git version at run time. 2014-01-21 13:22:48 -04:00
Joey Hess
d1bf61464f expose tasty test suite's option parser 2014-01-21 00:08:43 -04:00
Joey Hess
3159da2693 Add and use numcopiesneeded preferred content expression.
* Add numcopiesneeded preferred content expression.
* Client, transfer, incremental backup, and archive repositories
  now want to get content that does not yet have enough copies.

This means the asssistant will make copies of files that don't yet
meet the configured numcopies, even to places that would not normally want
the file.

For example, if numcopies is 4, and there are 2 client repos and
2 transfer repos, and 2 removable backup drives, the file will be sent
to both transfer repos in order to make 4 copies. Once a removable drive
get a copy of the file, it will be dropped from one transfer repo or the
other (but not both).

Another example, numcopies is 3 and there is a client that has a backup
removable drive and two small archive repos. Normally once one of the small
archives has a file, it will not be put into the other one. But, to satisfy
numcopies, the assistant will duplicate it into the other small archive
too, if the backup repo is not available to receive the file.

I notice that these examples are fairly unlikely setups .. the old behavior
was not too bad, but it's nice to finally have it really correct.

.. Almost. I have skipped checking the annex.numcopies .gitattributes
out of fear it will be too slow.

This commit was sponsored by Florian Schlegel.
2014-01-20 17:35:29 -04:00
Joey Hess
d66535f065 global numcopies setting
* numcopies: New command, sets global numcopies value that is seen by all
  clones of a repository.
* The annex.numcopies git config setting is deprecated. Once the numcopies
  command is used to set the global number of copies, any annex.numcopies
  git configs will be ignored.
* assistant: Make the prefs page set the global numcopies.

This global numcopies setting is needed to let preferred content
expressions operate on numcopies.

It's also convenient, because typically if you want git-annex to preserve N
copies of files in a repo, you want it to do that no matter which repo it's
running in. Making it global avoids needing to warn the user about gotchas
involving inconsistent annex.numcopies settings.
(See changes to doc/numcopies.mdwn.)

Added a new variety of git-annex branch log file, that holds only 1 value.
Will probably be useful for other stuff later.

This commit was sponsored by Nicolas Pouillard.
2014-01-20 16:47:56 -04:00
Joey Hess
b6ba0bd556 sync --content: New option that makes the content of annexed files be transferred.
Similar to the assistant, this honors any configured preferred content
expressions.

I am not entirely happpy with the implementation. It would be nicer if
the seek function returned a list of actions which included the individual
file gets and copies and drops, rather than the current list of calls to
syncContent. This would allow getting rid of the somewhat reundant display
of "sync file [ok|failed]" after the get/put display.

But, do that, withFilesInGit would need to somehow be able to construct
such a mixed action list. And it would be less efficient than the current
implementation, which is able to reuse several values between eg get and
drop.

Note that currently this does not try to satisfy numcopies when
getting/putting files (numcopies are of course checked when dropping
files!) This makes it like the assistant, and unlike get --auto
and copy --auto, which do duplicate files when numcopies is not yet
satisfied. I don't know if this is the right decision; it only seemed to
make sense to have this parallel the assistant as far as possible to start
with, since I know the assistant works.

This commit was sponsored by Øyvind Andersen Holm.
2014-01-19 17:49:54 -04:00
Joey Hess
85185b8f50 Allow --all to be mixed with matching options like --copies and --in (but not --include and --exclude). 2014-01-18 14:58:56 -04:00
Joey Hess
2b1a7895fc list: Fix specifying of files to list. 2014-01-18 13:25:06 -04:00
Joey Hess
456c9eedb0 status: Support --json. 2014-01-18 12:05:10 -04:00
Joey Hess
1d90e72e91 Remove --json option from commands not supporting it. 2014-01-18 11:54:43 -04:00
Joey Hess
3cbb366ebc Really fix FTBFS on mipsel and sparc due to test suite not being available on those architectures. 2014-01-17 14:48:56 -04:00
Joey Hess
31b80e55f8 prep release 2014-01-16 11:38:16 -04:00
Joey Hess
4e19e87921 repair: Fix bug in packed refs file exploding code that caused a .gitrefs directory to be created instead of .git/refs 2014-01-15 16:34:18 -04:00
Joey Hess
daec7ffafe Fix FTBFS on mipsel and sparc due to test suite not being available on those architectures. 2014-01-15 12:49:56 -04:00
Joey Hess
18a3e51d52 assistant: Detect if .git/annex/index is corrupt at startup, and recover. 2014-01-14 17:10:30 -04:00
Joey Hess
d07f2d7865 Fix a long-standing bug that could cause the wrong index file to be used when committing to the git-annex branch, if GIT_INDEX_FILE is set in the environment. This typically resulted in git-annex branch log files being committed to the master branch and later showing up in the work tree. (These log files can be safely removed.) 2014-01-14 15:36:33 -04:00
Joey Hess
5e6e89f423 repair: Support old git versions from before git fsck --no-dangling was implemented. 2014-01-13 18:10:45 -04:00
Joey Hess
7ee87c7bd6 map: Fix display of v5 direct mode repos.
The direct mode guard confused map. So need to run through Annex.new to get
the actual path to the repo.
2014-01-13 15:36:02 -04:00
Joey Hess
5a5adc44f5 Refuse to build with git older than 1.7.1.1, which is needed for git checkout -B 2014-01-13 15:17:48 -04:00
Joey Hess
0cac4402ac Android: Avoid passing --clobber to busybox wget. 2014-01-13 14:52:49 -04:00
Joey Hess
c20f31a1ad add GETAVAILABILITY to external special remote protocol
And some reworking of types, and added an annex-availability git config
setting.
2014-01-13 14:41:10 -04:00
Joey Hess
57edce8ad9 external special remote protocol: Added GETGITDIR. 2014-01-13 14:00:09 -04:00
Joey Hess
93161d0dea copyright year 2014-01-08 16:29:15 -04:00
Joey Hess
85272d8a98 Added tahoe special remote.
Known problems:

1. Tries to tahoe start when daemon is already running.

2. If multiple tahoe remotes are set up on the same computer,
   they will have the same node.url configured by default,
   and this confuses tahoe commands.

This commit was sponsored by LeastAuthority.com
2014-01-08 16:14:41 -04:00
Joey Hess
5e23dfabd6 add DEBUG 2014-01-07 13:23:58 -04:00
Joey Hess
0a98f1be15 prep release 2014-01-07 12:25:12 -04:00
Joey Hess
1c02fc4987 no lsof on kfreebsd 2014-01-07 12:12:36 -04:00
Joey Hess
e5b4d447b6 assistant: Start a new git-annex transferkeys process after a network connection change
So that remotes that use a persistent network connection are restarted.

A remote might keep open a long duration network connection, and could
fail to deal well with losing the connection. This is particularly a
concern now that we have external special reotes. An external
special remote that is implemented naively might open the connection only
when PREPARE is sent, and if it loses connection, throw errors on each
request that is made.

(Note that the ssh connection caching should not have this problem; if the
long-duration ssh process loses connection, the named pipe is disconnected
and the next ssh attempt will reconnect. Also, XMPP already deals with
disconnection robustly in its own way.)

There's no way for git-annex to know if a lost network connection actually
affects a given remote, which might have a transfer in process. It does not
make sense to force kill the transferkeys process every time the NetWatcher
detects a change. (Especially because the NetWatcher sometimes polls 1
change per hour.)

In any case, the NetWatcher only detects connection to a network, not
disconnection. So if a transfer is in progress over the network, and the
network goes down, that will need to time out on its own.

An alternate approch that was considered is to use a separate transferkeys
process for each remote, and detect when a request fails, and assume that
means that process is in a failing state and restart it. The problem with
that approach is that if a resource is not available and a remote fails
every time, it degrades to starting a new transferkeys process for every
file transfer, which is too expensive.

Instead, this commit only handles the network reconnection case, and restarts
transferkeys only once the network has reconnected and another transfer needs
to be made. So, a transferkeys process will be reused for 1 hour, or until the
next network connection.

----

The NotificationBroadcaster was rewritten to use TMVars rather than MSampleVars,
to allow checking without blocking if a notification has been received.

----

This commit was sponsored by Tobias Brunner.
2014-01-06 16:03:39 -04:00
Joey Hess
b92b54bd42 assistant: Fixed several minor memory leaks that manifested when adding a large number of files. 2014-01-05 21:30:48 -04:00
Joey Hess
0cc1bd7e53 add: Fix rollback when disk is completely full.
Noticed that it was possible for add to move a file to .git/annex/objects
and not make the link if the disk was full. This happened because the
location log update failed, and so addLink never got a chance to run.
Running addLink first fixes it; on error it will unwind by moving the file
back to where it was originally.
2014-01-05 14:09:57 -04:00
Joey Hess
f9e7b6cf61 addurl, importfeed: Honor annex.diskreserve as long as the size of the url can be checked.
This adds a http HEAD before the download is done. That was already the
case when the assistant was running, and it seems worth it to avoid filling
up the whole disk, like happened to my server today.
2014-01-04 15:08:06 -04:00
Joey Hess
fdcc04096c assistant: Ensure that .ssh/config and .ssh/authorized_keys are not group or world writable when writing to those files, as that can make ssh refuse to use them, if it allows another user to write to them. 2014-01-03 17:44:12 -04:00
Joey Hess
3e68c1c2fd add remote state logs
This allows a remote to store a piece of arbitrary state associated with a
key. This is needed to support Tahoe, where the file-cap is calculated from
the data stored in it, and used to retrieve a key later. Glacier also would
be much improved by using this.

GETSTATE and SETSTATE are added to the external special remote protocol.

Note that the state is left as-is even when a key is removed from a remote.
It's up to the remote to decide when it wants to clear the state.

The remote state log, $KEY.log.rmt, is a UUID-based log. However,
rather than using the old UUID-based log format, I created a new variant
of that format. The new varient is more space efficient (since it lacks the
"timestamp=" hack, and easier to parse (and the parser doesn't mess with
whitespace in the value), and avoids compatability cruft in the old one.

This seemed worth cleaning up for these new files, since there could be a
lot of them, while before UUID-based logs were only used for a few log
files at the top of the git-annex branch. The transition code has also
been updated to handle these new UUID-based logs.

This commit was sponsored by Daniel Hofer.
2014-01-03 16:35:57 -04:00