Commit graph

166 commits

Author SHA1 Message Date
Joey Hess
b77903af48
New annex.synccontent config setting
.. which can be set to true to make git annex sync default to --content.

This may become the default at some point in the future.

As well as being configuable by git config, it can be configured by
git-annex config to control the default behavior in all clones of a
repository.

Had to add a separate --no-content switch to we can tell if it's been
explicitly set, and should override annex.synccontent. If --content was the
default, this complication would not be necessary.

This commit was sponsored by Jake Vosloo on Patreon.
2017-02-03 14:31:17 -04:00
Joey Hess
ed56dba868
annex.autocommit can be configured via git-annex config
... to control the default behavior in all clones of a repository.

This includes a new Configurable data type, so the GitConfig type indicates
which values can be configured this way.

The implementation should be quite efficient; the config log is only read
once, and only when a Configurable value has not already been set by
git-config.

Indeed, it would be nice in the future to extend this, so that git-config
is itself only read on demand. Some commands may not need to look at the
git configuration at all.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 13:58:53 -04:00
Joey Hess
ed60f60e9b
unused: Improved memory use significantly when there are a lot of differences between branches.
Argh, didn't need an accumulator here!

I think I use accumulators a lot more than I need to when recusively
processing lists..

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-01-31 19:42:00 -04:00
Joey Hess
062286135c
unused: When large files are checked right into git, avoid buffering their contents in memory.
This makes it a little bit slower since it has to check file size,
but worth it to fix a potential memory use problem.

This commit was sponsored by Fernando Jimenez on Patreon.
2017-01-31 19:09:37 -04:00
Joey Hess
9eb10caa27
Some optimisations to string splitting code.
Turns out that Data.List.Utils.split is slow and makes a lot of
allocations. Here's a much simpler single character splitter that behaves
the same (even in wacky corner cases) while running in half the time and
75% the allocations.

As well as being an optimisation, this helps move toward eliminating use of
missingh.

(Data.List.Split.splitOn is nearly as slow as Data.List.Utils.split and
allocates even more.)

I have not benchmarked the effect on git-annex, but would not be surprised
to see some parsing of eg, large streams from git commands run twice as
fast, and possibly in less memory.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2017-01-31 19:06:22 -04:00
Joey Hess
3300911b14
lts-7.18 finally!
esqueleto finally got fixed, thanks to @bitemyapp

Since XMPP was removed, the previous build failures related to it should
no longer be a problem either.

Meanwhile, lts-5.18 fails to build anymore on Debian due to linker
hardening breaking the version of ghc stack uses with that version.

This commit was sponsored by Francois Marier on Patreon.
2017-01-31 12:27:08 -04:00
Joey Hess
339464e847
config: New command for storing configuration in the git-annex branch.
Any config names can be set using this; git-annex commands will only look
at specific ones that make sense and are worth the overhead of querying the
branch.

This might also be useful for storing whatever other config-type stuff the
user might want to shove into the git-annex branch.

This commit was sponsored by Jochen Bartl on Patreon.
2017-01-30 16:46:38 -04:00
Joey Hess
26d23e38f1
vicfg: Include the numcopies configuation.
Docs say vicfg can configure everything from git-annex branch,
so it ought to configure numcopies.

Note that commenting out existing numcopies does not unset it.

This commit was sponsored by Thom May on Patreon.
2017-01-30 15:27:25 -04:00
Joey Hess
280442ca2c
Remove -j short option for --json-progress; that option was already taken for --json.
This commit was sponsored by Trenton Cronholm.
2017-01-30 12:46:42 -04:00
Joey Hess
f275caf732
Increase default cost for p2p remotes from 200 to 1000. This makes git-annex prefer transferring data from special remotes when possible. 2017-01-06 15:23:30 -04:00
Joey Hess
8740cd9716
releasing package git-annex version 6.20170101 2016-12-31 23:59:56 -04:00
Joey Hess
10e4d93212
Support all common locations of the torrc file. 2016-12-28 15:12:31 -04:00
Joey Hess
b68d2a4b68
webapp: full wormhole pairing UI (untested)
This commit was sponsored by Riku Voipio.
2016-12-27 16:41:35 -04:00
Joey Hess
8484c0c197
Always use filesystem encoding for all file and handle reads and writes.
This is a big scary change. I have convinced myself it should be safe. I
hope!
2016-12-24 14:46:31 -04:00
Joey Hess
e08691b393
enable-tor: When run as a regular user, test a connection back to the hidden service over tor.
This way we know that after enable-tor, the tor hidden service is fully
published and working, and so there should be no problems with it at
pairing time.

It has to start up its own temporary listener on the hidden service. It
would be nice to have it start the remotedaemon running, so that extra
step is not needed afterwards. But, there may already be a remotedaemon
running, in communication with the assistant and we don't want to start
another one. I thought about trying to HUP any running remotedaemon, but
Windows does not make it easy to do that. In any case, having the user
start the remotedaemon themselves lets them know it needs to be running
to serve the hidden service.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2016-12-24 12:50:23 -04:00
Joey Hess
22252e8e4c
Revert "close"
This reverts commit 3aaabc906b.

Commit contained incomplete work.
2016-12-24 12:07:15 -04:00
Joey Hess
3aaabc906b
close 2016-12-22 13:59:21 -04:00
Joey Hess
f7ca2b92fb
enable-tor: No longer needs to be run as root.
When run by not root, su's to root automatically.

This commit was sponsored by Brock Spratlen on Patreon.
2016-12-20 17:40:36 -04:00
Joey Hess
944a6503b9
relocate tor socket out of /etc
weasel explained that apparmor limits on what files tor can read do not
apply to sockets (because they're not files). And apparently the
problems I was seeing with hidden services not being accessible had to
do with onion address propigation and not the location of the socket
file.

remotedaemon looks up the HiddenServicePort in torrc, so if it was
previously configured with the socket in /etc, that will still work.

This commit was sponsored by Denis Dzyubenko on Patreon.
2016-12-20 16:24:46 -04:00
Joey Hess
8f3b2c206c
Debian: Suggest tor and magic-wormhole.
Suggests, not recommends, because tor is not for everyone.
2016-12-20 15:26:14 -04:00
Joey Hess
e312ec3750
Fix build with directory-1.3.
See https://github.com/haskell/directory/issues/66
2016-12-20 15:23:59 -04:00
Joey Hess
a171e576b2
rekey --force: Incorrectly marked the new key's content as being present in the local repo even when it was not. 2016-12-19 18:18:57 -04:00
Joey Hess
95c8b37544
Linux standalone: Improve generation of locale definition files, supporting locales such as, en_GB.UTF-8. 2016-12-19 17:03:52 -04:00
Joey Hess
ccde0932a5
p2p --pair with magic wormhole (untested)
It builds. I have not tried to run it yet. :)

This commit was sponsored by Jake Vosloo on Patreon.
2016-12-18 16:51:41 -04:00
Joey Hess
38f9337e16
Revert "p2p --link now defaults to setting up a bi-directional link"
This reverts commit 3037feb1bf.

On second thought, this was an overcomplication of what should be the
lowest-level primitive. Let's build bi-directional links at the pairing
level with eg magic wormhole.
2016-12-16 18:26:07 -04:00
Joey Hess
bd811d3853
p2p: Added --one-way option.
This commit was sponsored by Fernando Jimenez on Patreon.
2016-12-16 16:43:37 -04:00
Joey Hess
3037feb1bf
p2p --link now defaults to setting up a bi-directional link
Both the local and remote git repositories get remotes added
pointing at one-another.

Makes pairing twice as easy!

Security: The new LINK command in the protocol can be sent repeatedly,
but only by a peer who has authenticated with us. So, it's entirely safe to
add a link back to that peer, or to some other peer it knows about.
Anything we receive over such a link, the peer could send us over the
current connection.

There is some risk of being flooded with LINKs, and adding too many
remotes. To guard against that, there's a hard cap on the number of remotes
that can be set up this way. This will only be a problem if setting up
large p2p networks that have exceptional interconnectedness.

A new, dedicated authtoken is created when sending LINK.

This also allows, in theory, using a p2p network like tor, to learn about
links on other networks, like telehash.

This commit was sponsored by Bruno BEAUFILS on Patreon.
2016-12-16 16:38:06 -04:00
Joey Hess
e67a310da1
p2p: --link no longer takes a remote name, instead the --name option can be used. 2016-12-16 15:37:50 -04:00
Joey Hess
469bfa7ff3
Make all --batch input, as well as fromkey and registerurl stdin be processed without requiring it to be in the current encoding. 2016-12-13 15:35:04 -04:00
Joey Hess
48d9624a2d
Revert ServerAliveInterval
Revert ServerAliveInterval change in 6.20161111, which caused problems
with too many old versions of ssh and unusual ssh configurations.

It should have not been needed anyway since ssh is supposted to
have TCPKeepAlive enabled by default.
2016-12-13 12:12:38 -04:00
Joey Hess
59fead6da3
Pass annex.web-options to wget and curl after other options, so that eg --no-show-progress can be set by the user to disable the default --show-progress. 2016-12-13 11:56:23 -04:00
Joey Hess
d9490685fd
metadata --batch: Fix bug when conflicting metadata changes were made in the same batch run.
1 microsecond delay is ugly.. but, maintaining an queue of a list of timestamps
and taking a new one from the queue each time around, or maintaining a timestamp
counter, would probably be slower.
2016-12-13 11:07:49 -04:00
Joey Hess
a52c011581
Debian: Build webapp on armel. 2016-12-11 21:30:07 -04:00
Joey Hess
bb66e098b1
linux standalone builds should have "unable to decommit memory" bug fixed 2016-12-11 15:37:52 -04:00
Joey Hess
73a79147b1
releasing package git-annex version 6.20161210 2016-12-10 12:23:18 -04:00
Joey Hess
749623df86
fixed 2016-12-10 10:47:16 -04:00
Joey Hess
15be5c04a6
git-annex-shell, remotedaemon, git remote: Fix some memory DOS attacks.
The attacker could just send a very lot of data, with no \n and it would
all be buffered in memory until the kernel killed git-annex or perhaps OOM
killed some other more valuable process.

This is a low impact security hole, only affecting communication between
local git-annex and git-annex-shell on the remote system. (With either
able to be the attacker). Only those with the right ssh key can do it. And,
there are probably lots of ways to construct git repositories that make git
use a lot of memory in various ways, which would have similar impact as
this attack.

The fix in P2P/IO.hs would have been higher impact, if it had made it to a
released version, since it would have allowed DOSing the tor hidden
service without needing to authenticate.

(The LockContent and NotifyChanges instances may not be really
exploitable; since the line is read and ignored, it probably gets read
lazily and does not end up staying buffered in memory.)
2016-12-09 13:34:32 -04:00
Joey Hess
2fb6fd7434
Merge branch 'master' into tor 2016-12-07 14:32:25 -04:00
Joey Hess
f61508aed4
add: Stage modified non-large files when running in indirect mode.
(This was already done in v6 mode and direct mode.)
2016-12-05 14:10:21 -04:00
Joey Hess
82d01f5619
rekey: Added --batch mode.
Would have liked to make the Parser parse the file and key pairs, but it
seems that optparse-applicative is unable to handle eg:

	many ((,) <$> argument <*> argument)

This commit was sponsored by Thomas Hochstein on Patreon.
2016-12-05 12:55:50 -04:00
Joey Hess
e65c31e56b
changelog 2016-12-05 12:16:35 -04:00
Joey Hess
93852dd7e8
rmurl: --batch
* rmurl: Multiple pairs of files and urls can be provided on the
  command line.
* rmurl: Added --batch mode.

This commit was sponsored by Trenton Cronholm on Patreon.
2016-12-05 12:10:07 -04:00
Joey Hess
bfc8305814
implement p2p command 2016-11-30 14:35:24 -04:00
Joey Hess
24593aaa32
Merge branch 'master' into tor 2016-11-30 14:16:36 -04:00
Joey Hess
8354612131
prefer xdot over dot
* map: Run xdot if it's available in PATH. On OSX, the dot command
  does not support graphical display, while xdot does.
* Debian: xdot is a better interactive viewer than dot, so Suggest
  xdot, rather than graphviz.
2016-11-30 12:50:49 -04:00
Joey Hess
398345cb26
Merge branch 'master' into tor 2016-11-29 15:45:29 -04:00
Joey Hess
ae9f99f342
Relicense 5 source files that are not part of the webapp from AGPL to GPL.
Building w/o the webapp is not supposed to pull in any AGPLed files.

I appear to have written all the code in these files;
the only commit by anyone else is 64e844e1fe
and is a spelling fix that is not copyrightable.
2016-11-21 23:46:59 -04:00
Joey Hess
070fb9e624
Added git-remote-tor-annex, which allows git pull and push to the tor hidden service.
Almost working, but there's a bug in the relaying.

Also, made tor hidden service setup pick a random port, to make it harder
to port scan.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2016-11-21 17:27:38 -04:00
Joey Hess
6e6d1a8c15
addurl: Fix bug in checking annex.largefiles expressions using largerthan, mimetype, and smallerthan; the first two always failed to match, and the latter always matched. 2016-11-21 11:30:53 -04:00
Joey Hess
74691ddf0e
remotedaemon: serve tor hidden service 2016-11-20 15:48:12 -04:00