Commit graph

103 commits

Author SHA1 Message Date
guilhem
a1eded8641 Allow rsync to use other remote shells.
Introduced a new per-remote option 'annex-rsync-transport' to specify
the remote shell that it to be used with rsync. In case the value is
'ssh', connections are cached unless 'sshcaching' is unset.
2013-04-13 19:26:24 -04:00
Joey Hess
9e11699c76 connect existing meters to the transfer log for downloads
Most remotes have meters in their implementations of retrieveKeyFile
already. Simply hooking these up to the transfer log makes that information
available. Easy peasy.

This is particularly valuable information for encrypted remotes, which
otherwise bypass the assistant's polling of temp files, and so don't have
good progress bars yet.

Still some work to do here (see progressbars.mdwn changes), but this
is entirely an improvement from the lack of progress bars for encrypted
downloads.
2013-04-11 17:32:31 -04:00
Joey Hess
d440b6047b Added annex.web-download-command setting. 2013-04-08 23:34:05 -04:00
Joey Hess
84de4a14ad better archive directory handling
Adjust preferred content expressions so that content in archive directories
is preferred until it has reached an archive or smallarchive repository.
2013-04-06 18:29:52 -04:00
Joey Hess
f1b0a4b404 Use lower case hash directories for storing files on crippled filesystems, same as is already done for bare repositories.
* since this is a crippled filesystem anyway, git-annex doesn't use
  symlinks on it
* so there's no reason to use the mixed case hash directories that we're
  stuck using to avoid breaking everyone's symlinks to the content
* so we can do what is already done for all bare repos, and make non-bare
  repos on crippled filesystems use the all-lower case hash directories
* which are, happily, all 3 letters long, so they cannot conflict with
  mixed case hash directories
* so I was able to 100% fix this and even resuming `git annex add` in the
  test case will recover and it will all just work.
2013-04-04 15:46:33 -04:00
Joey Hess
8a5b397ac4 hlint 2013-04-03 03:52:41 -04:00
Joey Hess
a73de165f4 fix preferred content check for 1 semitrusted or better copy
Let's make semitrusted+:1 mean that, since it cannot be expressed easily
with the current syntax (could use (semitrusted:1 or trusted:1), but that
does not scale to higher values than 2 copy, and also fails if I add more
trust levels).

Thanks to Andy for spotting this bug by just reading my blog.
2013-04-02 23:40:13 -04:00
Joey Hess
d888fb6a4b move content from remote when user asks to delete it 2013-03-31 19:00:43 -04:00
Joey Hess
940e167ecc Adjust built-in preferred content expressions to make most types of repositories want content that is only located on untrusted or dead repositories. 2013-03-31 17:10:25 -04:00
guilhem
55f0f858ee Allow other MAC algorithms in the Remote Config. 2013-03-29 18:04:52 -04:00
Joey Hess
67e817c6a1 New annex.largefiles setting, which configures which files git annex add and the assistant add to the annex.
I would have sort of liked to put this in .gitattributes, but it seems
it does not support multi-word attribute values. Also, making this a single
config setting makes it easy to only parse the expression once.

A natural next step would be to make the assistant `git add` files that
are not annex.largefiles. OTOH, I don't think `git annex add` should
`git add` such files, because git-annex command line tools are
not in the business of wrapping git command line tools.
2013-03-29 16:17:13 -04:00
Joey Hess
cf07a2c412 webapp: Progess bar fixes for many types of special remotes.
There was confusion in different parts of the progress bar code about
whether an update contained the total number of bytes transferred, or the
number of bytes transferred since the last update. One way this bug
showed up was progress bars that seemed to stick at zero for a long time.
In order to fix it comprehensively, I add a new BytesProcessed data type,
that is explicitly a total quantity of bytes, not a delta.

Note that this doesn't necessarily fix every problem with progress bars.
Particularly, buffering can now cause progress bars to seem to run ahead
of transfers, reaching 100% when data is still being uploaded.
2013-03-28 17:04:37 -04:00
Joey Hess
0d72519ae0 make usage less terrifying
Need to make `git annex help command` show the options for that command.
2013-03-25 12:41:57 -04:00
Joey Hess
3babb935d3 Group subcommands into sections in usage. Closes: #703797 2013-03-25 10:23:05 -04:00
Joey Hess
cfd3b16fe1 add section metadata to all commands
Not yet used .. mindless train work.
2013-03-24 18:28:21 -04:00
Joey Hess
17c9ff576d Fix several bugs caused by a bad Ord instance for Remote.
A long time ago I made Remote be an instance of the Ord typeclass, with an
implementation that compared the costs of Remotes. That seemed like a good
idea at the time, as it saved typing.. But at the time I was still making
custom Read and Show instances too. I've since learned that this is *not* a
good idea, and neither is making custom Ord instances, without deep thought
about the possible sets of values in a type. Haskell typeclasses are not a
toy.

This Ord instance came around and bit me when I put Remotes into a Set,
because now remotes with the same cost appeared to be in the Set even if
they were not. Also affected putting Remotes into a Map.

Rarely does a bug go this deep. I've fixed it comprehensively, first
removing the Ord instance entirely, and fixing the places that wanted to
order remotes by cost to do it explicitly. Then adding back an Ord instance
that is much more sane. Also by checking the rest of the Ord instances in
the code base (which were all ok).

While doing that, I found lots of places that kept remotes in Maps and
Sets. All of it was probably subtly broken in one way or another before
this fix, but it would be hard to say exactly how the bugs would
manifest.
2013-03-16 17:43:42 -04:00
Joey Hess
e856bd7c6a forgot to add to sum type 2013-03-16 12:09:22 -04:00
Joey Hess
79f10b75c1 Add incrementalbackup repository group. 2013-03-16 11:17:53 -04:00
Joey Hess
449520a573 add globallyAvailable to remotes 2013-03-15 19:16:13 -04:00
Joey Hess
19c0a0d5b1 split cost out into its own module
Added a function to insert a new cost into a list, which could be used to
asjust costs after a drag and drop.
2013-03-13 16:30:34 -04:00
guilhem
d2bc0e9f3e GnuPG options for symmetric encryption. 2013-03-11 09:48:38 -04:00
Joey Hess
24316f6562 improve imports 2013-02-27 21:48:46 -04:00
Joey Hess
a2f17146fa move Arbitrary instances out of Test and into modules that define the types
This is possible now that we build-depend on QuickCheck.
2013-02-27 21:42:07 -04:00
Joey Hess
5ea4b91fb4 start to support core.symlinks=false
Utility functions to handle no symlink mode, and converted Annex.Content to
use them; still many other places to convert.
2013-02-15 16:03:11 -04:00
Joey Hess
7ce30b534f add: Improved detection of files that are modified while being added.
In indirect mode, now checks the inode cache to detect changes to a file.
Note that a file can still be changed if a process has it open for write,
after landing in the annex.

In direct mode, some checking of the inode cache was done before, but
from a much later point, so fewer modifications could be detected. Now it's
as good as indirect mode.

On crippled filesystems, no lock down is done before starting to add a
file, so checking the inode cache is the only protection we have.
2013-02-14 16:54:36 -04:00
Joey Hess
47477b2807 crippled filesystem support, probing and initial support
git annex init probes for crippled filesystems, and sets direct mode, as
well as `annex.crippledfilesystem`.

Avoid manipulating permissions of files on crippled filesystems.
That would likely cause an exception to be thrown.

Very basic support in Command.Add for cripped filesystems; avoids the lock
down entirely since doing it needs both permissions and hard links.
Will make this better soon.
2013-02-14 14:15:26 -04:00
Joey Hess
0e3f931f37 add another setting to GitConfig 2013-01-28 00:33:19 +11:00
Joey Hess
5cd152b8a9 annex.autocommit
New setting, can be used to disable autocommit of changed files by the
assistant, while it still does data syncing and other tasks.

Also wired into webapp UI
2013-01-27 22:43:05 +11:00
Joey Hess
41ecdfe7b2 adjust archive exclusion 2013-01-22 07:10:24 +11:00
Joey Hess
6a067e5c28 manual and source repository groups 2013-01-21 04:18:46 -04:00
Joey Hess
4008590c68 type based git config handling for remotes
Still a couple of places that use git config ad-hoc, but this is most of it
done.
2013-01-01 13:58:14 -04:00
Joey Hess
7f7c31df1c type based git config handling
Now there's a Config type, that's extracted from the git config at startup.
Note that laziness means that individual config values are only looked up
and parsed on demand, and so we get implicit memoization for all of them.
So this is not only prettier and more type safe, it optimises several
places that didn't have explicit memoization before. As well as getting rid
of the ugly explicit memoization code.

Not yet done for annex.<remote>.* configuration settings.
2012-12-29 23:10:18 -04:00
Joey Hess
e71f85645e handle sha*sum's leading \ in checksum with certian unsual filenames
* Bugfix: Remove leading \ from checksums output by sha*sum commands,
  when the filename contains \ or a newline. Closes: #696384
* fsck: Still accept checksums with a leading \ as valid, now that
  above bug is fixed.
* migrate: Remove leading \ in checksums
2012-12-20 17:07:10 -04:00
Joey Hess
df1540787f test 2012-12-19 23:44:08 -04:00
Joey Hess
a6bd0076d4 remove unused fields 2012-12-19 23:41:54 -04:00
Joey Hess
0d50a6105b whitespace fixes 2012-12-13 00:45:27 -04:00
Joey Hess
727223a229 fix preferred content definition for the backup group
Broken by commit 99a8a5297c, which made empty
preferred content strings match nothing, rather than everything.
2012-12-10 15:44:58 -04:00
Joey Hess
020a25abe1 avoid unnecessary Maybe 2012-11-30 00:55:59 -04:00
Joey Hess
5f3661238d Display a warning when a non-existing file or directory is specified. 2012-11-25 17:54:08 -04:00
Joey Hess
463cf58140 webapp and assistant glacier support 2012-11-24 16:30:15 -04:00
Joey Hess
e2b7fc1ebd refactor 2012-11-18 21:50:16 -04:00
Joey Hess
e250f6f11f factor out Creds 2012-11-14 19:32:27 -04:00
Joey Hess
2172cc586e where indenting 2012-11-11 00:51:07 -04:00
Joey Hess
c7c2015435 add ConfigMonitor thread
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.

Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.

A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.

Also added a uuid.log cache. All other config files already had caches.
2012-10-20 16:43:35 -04:00
Joey Hess
2d9682f3c6 logic error 2012-10-14 16:21:18 -04:00
Joey Hess
27ea68ac6e handle transfer repository bootstrapping
When there's just 1 client repo, and a transfer repo is created, its
preferred content will now make it prefer all content in the client,
even though there's no other client yet to transfer it to. Presumably,
another client will be created eventually. It might even already exist,
and the transfer repo will be used to connect up with it.
2012-10-14 16:08:21 -04:00
Joey Hess
4e2e08b45a ui for selecting a repository group 2012-10-10 16:23:41 -04:00
Joey Hess
39be7eea40 add standard group selector to repo edit form 2012-10-10 16:04:28 -04:00
Joey Hess
e375b931c0 add --ingroup limit 2012-10-08 15:18:58 -04:00
Joey Hess
7a7f63182c vicfg: New command, allows editing (or simply viewing) most of the repository configuration settings stored in the git-annex branch.
Incomplete; I need to finish parsing and saving. This will also be used
for editing transfer control expresssions.

Removed the group display from the status output, I didn't really
like that format, and vicfg can be used to see as well as edit rempository
group membership.
2012-10-03 17:04:52 -04:00