Commit graph

922 commits

Author SHA1 Message Date
Joey Hess
9dc7a10448
Drop support for building with aws older than 0.14.
debian stable has 0.14 so lose the complexity for old versions
2019-04-19 14:27:59 -04:00
Joey Hess
c0c38e986d
added renameremote command 2019-04-15 13:49:03 -04:00
Joey Hess
2a99d7ffc0
improve error message 2019-03-22 12:23:59 -04:00
Joey Hess
519cadd1de
refactor RemoteTrackingBranch
Not specific to Import; export will use it too.
2019-03-01 14:47:56 -04:00
Joey Hess
8fdea8f444
WIP
Added graftTree but it's buggy.

Should use graftTree in Annex.Branch.graftTreeish; it will be faster
than the current implementation there.

Started Annex.Import, but untested and it doesn't yet handle tree
grafting.
2019-02-21 17:32:59 -04:00
Joey Hess
fd304dce60
split out Types.Import and some changes to the types in it 2019-02-21 13:39:09 -04:00
Joey Hess
5a294f0dd7
add Logs.ContentIdentifier 2019-02-20 17:22:56 -04:00
Joey Hess
a818bc5e73
add Database.ContentIdentifier
Does not yet have a way to update with new information from the
git-annex branch, which will be needed when multiple repos are importing
from the same remote.
2019-02-20 16:59:10 -04:00
Joey Hess
ccc0684d21
no remotes support import yet 2019-02-20 16:59:04 -04:00
Joey Hess
e8bfc3640b
storing ContentIdentifier in the git-annex branch 2019-02-20 15:40:07 -04:00
Joey Hess
f47ee98337
releasing package git-annex version 7.20190219 2019-02-19 12:19:53 -04:00
Joey Hess
b9a1db8068
update copyright metadata for recent changes
List all current AGPL files in COPYRIGHT; several that were GPL have had
new AGPL code added.

In git-annex.cabal, use AGPL-3 as the overall license, since it's not
possible to build git-annex without AGPL code anymore.
2019-02-08 15:19:11 -04:00
Joey Hess
3fa6be1fef
Added NetworkBSD build flag to deal with Network.BSD moving to a new package.
Like with the network-uri split, cabal will automatically turn off the flag
when building with an old network.

I have not tested building with the new network-3.0.0.0 yet; several
other dependencies including aws are still pinned on network-2.*
2019-02-08 13:36:39 -04:00
Joey Hess
4cf7deb57e
releasing package git-annex version 7.20190129 2019-01-29 15:21:44 -04:00
Joey Hess
467c3b393d
refactor magic 2019-01-23 12:40:59 -04:00
Joey Hess
f918e8798f
releasing package git-annex version 7.20190122 2019-01-22 12:28:14 -04:00
Joey Hess
9a4406e5e7
webapp: remove configurators for obsolste cloud storage services
* webapp: Remove configurator for box.com repository, since their
  webdav support is going away at the end of this January.
* webapp: Remove configurator for gitlab, which stopped supporting git-annex
  some time ago.

This commit was sponsored by Brock Spratlen on Patreon.
2019-01-22 11:48:35 -04:00
Joey Hess
d5f2463702
misctmp cleanup
* Switch to using .git/annex/othertmp for tmp files other than partial
  downloads, and make stale files left in that directory when git-annex
  is interrupted be cleaned up promptly by subsequent git-annex processes.
* The .git/annex/misctmp directory is no longer used and git-annex will
  delete anything lingering in there after it's 1 week old.

Also, in Annex.Ingest, made the filename it uses in the tmp dir be
prefixed with "ingest-" to avoid potentially using a filename used by
some other code.
2019-01-17 16:02:22 -04:00
Joey Hess
5d98cba923
use ByteStrings when reading annex symlinks and pointers
Now there's a ByteString used all the way from disk to Key.

The main complication in this conversion was the use of fromInternalGitPath
in several places to munge things on Windows. The things that used that
were changed to parse the ByteString using either path separator.

Also some code that had read from files to a String lazily was changed
to read a minimal strict ByteString.
2019-01-14 15:37:08 -04:00
Joey Hess
5ba14b5095
build cleanrly when benchmark flag is not enabled 2019-01-05 08:09:28 -04:00
Joey Hess
11d6e2e260
new improved benchmark command that can benchmark anything git-annex does 2019-01-04 13:46:36 -04:00
Joey Hess
b781fbcccf
update copyright year 2019-01-01 15:37:40 -04:00
Joey Hess
426bdbf113
releasing package git-annex version 7.20181211 2018-12-11 16:33:30 -04:00
Joey Hess
1d16605f93
releasing package git-annex version 7.20181205 2018-12-05 16:19:11 -04:00
Joey Hess
f81eaaf411
releasing package git-annex version 7.20181121 2018-11-21 14:24:04 -04:00
Joey Hess
f62114e5ad
Merge branch 'remove-esqueleto' 2018-11-20 11:50:04 -04:00
Joey Hess
de26cfbf89
Revert "depend on newer concurrent-output to fix crash"
This reverts commit 4511457382.

Not needed, 872af2b2f1 avoids the
underlying bug in concurrent-output.
2018-11-19 18:41:34 -04:00
Joey Hess
4511457382
depend on newer concurrent-output to fix crash
At least temporarily.. It may cause too much breakage to build
environments to keep this change yet.
2018-11-19 17:47:33 -04:00
Joey Hess
9127fe4821
add DebugLocks build flag
Using the method described in
https://www.fpcomplete.com/blog/2018/05/pinpointing-deadlocks-in-haskell
but my own code to implement it, and with callstacks added.

This work is supported by the NIH-funded NICEMAN (ReproNim TR&D3) project.
2018-11-19 15:02:43 -04:00
Sean Parsons
42bdc9fa2f Removed Esqueleto as a dependency. 2018-11-06 22:18:55 +00:00
Joey Hess
3016e94dbb
releasing package git-annex version 7.20181105 2018-11-05 13:33:36 -04:00
Joey Hess
2ca408dc33
Increase minimum QuickCheck version. 2018-10-31 15:53:22 -04:00
Joey Hess
8f9278787f
releasing package git-annex version 7.20181031 2018-10-31 15:46:57 -04:00
Joey Hess
3963c5fcf5
better approach to enabling WAL mode
The old approach opened the database an extra time to enable WAL mode,
but more recent persistent-sqlite has a better API to enable it.
2018-10-30 13:47:38 -04:00
Joey Hess
c75807ba3a
remove note about ghc bug
ghc 8.6 fixed that, apparently
2018-10-30 00:51:14 -04:00
Joey Hess
2e9f128dea
moved module and relicensed 2018-10-29 23:13:36 -04:00
Joey Hess
5d97898a7c
touch files with high-resolution timestamp
Needs unix 2.7.2, but that was included in ghc 8.0.1 (and much older)
so not really a new dep.
2018-10-29 22:25:21 -04:00
Joey Hess
94b7968f1f
forgot to remove this when dropping support for old ghc 2018-10-29 22:01:06 -04:00
Joey Hess
234842a347
v7
Install new git hooks in this version.

This does beg the question of what to do if git later gets eg a
post-smudge hook, that could run git-annex smudge --update. I think the
thing to do in that case would be to make git-annex smudge --update
install the new hooks. That way, as the user uses git-annex, the hook
would be created pretty quickly and without needing any extra syscalls
except for when git-annex smudge --update is called.

I considered doing something like that for installation of the
post-checkout and post-merge hooks, which would have avoided the need
for v7. But the only place it was cheap to do it would be in git-annex smudge
which could cheaply notice that smudge.log didn't exist yet and so know
the hooks needed to be installed. But since smudge used to populate pointer
files, it would be quite surprising if a single git checkout/merge failed
to update the work tree, and so that idea didn't work out.

The other reason for v7 is psychological -- users don't need to worry
about whether they might be running an old version of git-annex that
doesn't support their v7 repository very well. And bug reports about
"v6" have gotten a bit of a bad association in my head since they often
hit one of the known limitations and didn't realize it was experimental.

newtyped RepoVersion Int to avoid needing 2 comparisons in
versionSupportsUnlockedPointers etc. Also it's just nicer.

This commit was sponsored by John Pellman on Patreon.
2018-10-25 18:24:23 -04:00
Joey Hess
917a2c6095
defer updating unlocked files until after smudge filter
The smuge filter no longer provides git with annexed file content, to
avoid a git memory leak, and because that did not honor annex.thin.

git annex smudge --update has to be run after a checkout to update
unlocked files in the working tree with annexed file contents.

No hooks yet to run it.

This commit was sponsored by Nick Piper on Patreon.
2018-10-25 15:08:20 -04:00
Joey Hess
8be5a7269a
refactor getCurrentBranch
Both Command.Sync and Annex.Ingest had their own versions of this.

The one in Annex.Ingest used Git.Branch.currentUnsafe, but does not seem
to need it. That is only checking to see if it's in an adjusted unlocked
branch, and when in an adjusted branch, the branch does in fact exist,
so the added check that Git.Branch.current does is fine.

This commit was sponsored by Denis Dzyubenko on Patreon.
2018-10-19 17:29:18 -04:00
Joey Hess
38d691a10f
removed the old Android app
Running git-annex linux builds in termux seems to work well enough that the
only reason to keep the Android app would be to support Android 4-5, which
the old Android app supported, and which I don't know if the termux method
works on (although I see no reason why it would not).
According to [1], Android 4-5 remains on around 29% of devices, down from
51% one year ago.

[1] https://www.statista.com/statistics/271774/share-of-android-platforms-on-mobile-devices-with-android-os/

This is a rather large commit, but mostly very straightfoward removal of
android ifdefs and patches and associated cruft.

Also, removed support for building with very old ghc < 8.0.1, and with
yesod < 1.4.3, and without concurrent-output, which were only being used
by the cross build.

Some documentation specific to the Android app (screenshots etc) needs
to be updated still.

This commit was sponsored by Brett Eisenberg on Patreon.
2018-10-13 01:41:11 -04:00
Joey Hess
426f0f3f4b
releasing package git-annex version 6.20181011 2018-10-11 13:50:53 -04:00
Joey Hess
012d67c3eb
releasing package git-annex version 6.20180926 2018-09-26 13:16:54 -04:00
Joey Hess
547d01fd0e
releasing package git-annex version 6.20180913 2018-09-13 15:50:50 -04:00
Joey Hess
0a7c5a9982
dropdead per-remote metadata
Had to refactor pure code into separate modules so it is accessible
inside Annex.Branch.Transitions.

This commit was sponsored by Peter on Patreon.
2018-09-05 13:52:46 -04:00
Joey Hess
4ddce51a52
move Utility.OSX
stack complains when a module whose import is hidden behind a
ifdef os is not listed when building for other os's.
2018-09-05 11:55:08 -04:00
Joey Hess
69907e397f
revert a few problem areas of git-annex.cabal patch 2018-09-05 11:47:00 -04:00
fftmp
5f8f83b884
optional build assistant and other small cabal refinings 2018-09-05 11:42:30 -04:00
Joey Hess
55f8d90dee
remove Utlity.SRV, no longer used 2018-09-05 11:15:33 -04:00
Joey Hess
c565340adc
stop using external hash programs, since cryptonite is faster
In 2013, I wrote "Cryptohash benchmarks 90 to 101% faster than external
hashers". Re-benchmarking today, I found cryptonite's sha256 consistently
outperformed coreutils by 10% for large files. Tested 10 mb, 100 mb, 1 gb
files with both sha256 and sha512. And for smaller files, the external
process startup time swamps the hash time.

Perhaps cryptonite has improved. Or it could just do better on my
current CPU Intel(R) Pentium(R) CPU 4410Y @ 1.50GHz). Anyway, even if cryptonite
is slower in some situations, seems likely it would only be marginally slower;
it's got the same class of highly optimised C code under the hood as coreutils.
The main difference between the two sha256 implementations seems to be
how much of the inner loop they unroll..

This commit was sponsored by Henrik Riomar on Patreon.
2018-08-28 18:10:58 -04:00
Joey Hess
401a79675b
run git status before enabling clean filter
Avoids annex.largefiles inconsitency and also avoids a lot of
unneccessary calls to the clean filter when a large repo's clone
is being initialized.

This commit was supported by the NSF-funded DataLad project.
2018-08-28 10:36:22 -04:00
Joey Hess
e094cf3377
split out modules from Annex.Content 2018-08-22 14:45:53 -04:00
Joey Hess
42c5d7c64f
moving to filterdriver branch
Not used in master, so remove until/unless filterdriver branch is
merged.
2018-08-14 13:37:17 -04:00
Joey Hess
fd42df78a0
git long-running process handshake implementation
This commit was supported by the NSF-funded DataLad project.
2018-08-10 16:03:04 -04:00
Joey Hess
1272ccdf1e
git pkt-line format
Git uses pkt-line in the pack and http protocols, and for the long-running
filter processes protocol as well.

This should be a quite efficient parser and builder since it uses
attoparsec and bytestring-builder.

This adds a dependency on attoparsec, but it's a free dependency because
eg aeson depends on attoparsec and git-annex depends on aeson.

This commit was supported by the NSF-funded DataLad project.
2018-08-10 14:12:33 -04:00
Joey Hess
2503cd63d0
prep release 2018-08-06 20:30:38 -04:00
Joey Hess
22ff136230
prep for release tomorrow 2018-07-18 14:45:44 -04:00
Joey Hess
841788cad4
add transformers to setup-depends
per Peter Simons:

Setup.hs depends on 'transformers' for versions of base
prior to 4.9.x so that Control.Monad.IO.Class can be included. Just
adding the library to 'setup-depends' fixes the build with older
compilers like GHC 7.10.3.
2018-07-01 13:08:40 -04:00
Joey Hess
14557a3ff6
git-annex.cabal: Fix network version.
Needed for hostAddressToTuple.

Which means the build flag for the network-uri split is no longer needed.
2018-07-01 13:07:24 -04:00
Joey Hess
57dc30a029
finalize release 2018-06-22 10:37:01 -04:00
Joey Hess
991265e724
version deps
need at least http-client-0.4.31 to build now, and connection-0.2.6
2018-06-19 19:55:22 -04:00
Joey Hess
014a3fef34
added isPrivateAddress and isLoopbackAddress
For use in a security boundary enforcement.

Based on https://en.wikipedia.org/wiki/Reserved_IP_addresses

Including supporting IPv4 addresses embedded in IPv6 addresses. Because
while RFC6052 3.1 says "Address translators MUST NOT translate packets
in which an address is composed of the Well-Known Prefix and a non-
global IPv4 address; they MUST drop these packets", I don't want to
trust that implementations get that right when enforcing a security
boundary.

This commit was sponsored by John Pellman on Patreon.
2018-06-17 13:28:25 -04:00
Joey Hess
40e8358284
add Utility.HttpManagerRestricted
This is a clean way to add IP address restrictions to http-client, and
any library using it.
See https://github.com/snoyberg/http-client/issues/354#issuecomment-397830259

Some code from http-client and http-client-tls was copied in and
modified. Credited its author accordingly, and used the same MIT license.

The restrictions don't apply to http proxies. If using http proxies is a
problem, http-client already has a way to disable them.
SOCKS support is not included. As far as I can tell, http-client-tls
does not support SOCKS by default, and so git-annex never has.

The additional dependencies are free; git-annex already transitively
depended on them via http-conduit.

This commit was sponsored by Eric Drechsel on Patreon.
2018-06-16 18:44:13 -04:00
Joey Hess
33834140e6
releasing package git-annex version 6.20180529 2018-05-29 13:06:56 -04:00
Joey Hess
0632c49c22
releasing package git-annex version 6.20180509 2018-05-09 16:20:43 -04:00
Joey Hess
f98251c97c
releasing package git-annex version 6.20180427 2018-04-27 12:37:01 -04:00
Joey Hess
9807e5bead
fix webapp opening in termux
Open real url not html shim since android and file:// urls is a nasty
kettle of fish.

This commit was sponsored by John Pellman on Patreon.
2018-04-25 14:38:42 -04:00
Joey Hess
89e1a05a8f
Fix mangling of --json output of utf-8 characters when not running in a utf-8 locale
As long as all code imports Utility.Aeson rather than Data.Aeson,
and no Strings that may contain utf-8 characters are used for eg, object
keys via T.pack, this is guaranteed to fix the problem everywhere that
git-annex generates json.

It's kind of annoying to need to wrap ToJSON with a ToJSON', especially
since every data type that has a ToJSON instance has to be ported over.
However, that only took 50 lines of code, which is worth it to ensure full
coverage. I initially tried an alternative approach of a newtype FileEncoded,
which had to be used everywhere a String was fed into aeson, and chasing
down all the sites would have been far too hard. Did consider creating an
intentionally overlapping instance ToJSON String, and letting ghc fail
to build anything that passed in a String, but am not sure that wouldn't
pollute some library that git-annex depends on that happens to use ToJSON
String internally.

This commit was supported by the NSF-funded DataLad project.
2018-04-16 16:21:21 -04:00
Joey Hess
ba8a3156ea
releasing package git-annex version 6.20180409 2018-04-09 13:24:45 -04:00
Joey Hess
0f6775f1ff
refactor sinkResponseFile and add downloadC
Remote.S3 and Remote.Helper.Http both had similar code to sink a
http-conduit Response to a file; refactor out sinkResponseFile.

downloadC downloads an url to a file using http-conduit, and supports
resuming. Falls back to curl to handle urls that http-conduit does not
support. This is not used yet, but the goal is to replace download with
it.

git-annex.cabal: conduit-extra was not actually used for a long time,
remove the dep. conduit moves into the main dependency list, but since
http-conduit was already in there, and it depends on conduit, that's not
really adding a new build dep.

This commit was supported by the NSF-funded DataLad project.
2018-04-06 16:07:08 -04:00
Joey Hess
2927618d35
Added adb special remote which allows exporting files to Android devices.
git annex testremote passes.

exportree not implemented yet, although the documentation talks about it,
since it will be the main way this remote will be used.

The adb push/pull progress is displayed for now; it would be better
to consume it and use it to update the git-annex progress bar.

This commit was sponsored by andrea rota.
2018-03-27 14:54:41 -04:00
Joey Hess
bc1d56bdfd
cabal sdist refuses to build with -O1
Sigh, disabled the full fix. What we have now will work once the ghc bug
is fixed, but not currently.
2018-03-16 12:56:28 -04:00
Joey Hess
14100f4f74
releasing package git-annex version 6.20180316 2018-03-16 12:40:27 -04:00
Joey Hess
521d4ede1e
fix build with cryptonite-0.20
Some blake hash varieties were not yet available in that version.
Rather than tracking exact details of what cryptonite supported when,
disable blake unless using a current cryptonite.
2018-03-15 11:16:00 -04:00
Joey Hess
050ada746f
Added backends for the BLAKE2 family of hashes.
There are a lot of different variants and sizes, I suppose we might as well
export all the common ones.

Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support.
Even android has 0.16 or newer.

On Debian, Blake2bp_512 is buggy, so I have omitted it for now.
http://bugs.debian.org/892855

This commit was sponsored by andrea rota.
2018-03-13 16:23:42 -04:00
Joey Hess
16af259209
refactor p2p remote action code
Make a Remote.Helper.P2P using code that was in Remote.P2P, converted to
use generic protocol runner actions.

This will allow it to be reused in Remote.Git.

This commit was sponsored by mo on Patreon.
2018-03-08 16:11:00 -04:00
Joey Hess
6ddfa9807b
implemented git-annex-shell p2pstdio
Not yet used by git-annex, but this will allow faster transfers etc than
using individual ssh connections and rsync.

Not called git-annex-shell p2p, because git-annex p2p does something
else and I don't want two subcommands with the same name between the two
for sanity reasons.

This commit was sponsored by Øyvind Andersen Holm.
2018-03-07 15:38:01 -04:00
Joey Hess
6063b3df3f
Dial back optimisation when building on arm
Prevent ghc and llc from running out of memory when optimising some
files.

Sean Whitton reported that doing this only in Test.hs was insufficient,
the build still OOMed by the time it got to Test.hs. He had earlier found
the build worked when these options are applied globally.

See https://ghc.haskell.org/trac/ghc/ticket/14821 for why it needs -O1;
once that's fixed it may suffice to use "GHC-Options: -O2 -optlo-O2",
although it may also be that the -O1 prevents ghc from using/leaking
as much memory.

os(arm) should match armel, armhf, armeb, and arm.
It probably also matches arm64, somewhat unfortunately since arm64
systems probably tend to have more memory. See list of arches in
https://hackage.haskell.org/package/Cabal-1.22.2.0/docs/src/Distribution-System.html

This commit was sponsored by Henrik Riomar on Patreon.
2018-03-04 19:48:07 -04:00
Joey Hess
8927ef7613
releasing package git-annex version 6.20180227 2018-02-27 12:05:35 -04:00
Joey Hess
39b59c341f
send stderr to json when --json-error-messages enabled 2018-02-19 15:28:38 -04:00
Joey Hess
8ccfbd14d0
Split Test.hs and avoid optimising it much, to need less memory to compile.
The ghc options were found by Sean Whitton; the debian arm autobuilders
need those to build w/o OOM, and it seems to involve llvm using too much
memory to optimize Test.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2018-02-18 11:48:48 -04:00
Joey Hess
bbdfdaa833
git-annex.cabal: Once more try to not build the assistant on the hurd, hopefully hackage finally recognises that OS.
Last try was in 2015 https://github.com/haskell/hackage-server/issues/269
2018-02-18 11:09:16 -04:00
Joey Hess
04d2ca36dd
fix build w/o webapp, s3 2018-01-24 20:40:50 -04:00
Joey Hess
84ea1eb762
releasing package git-annex version 6.20180112 2018-01-12 15:47:32 -04:00
Joey Hess
24df95f0f6
Fix several places where files in .git/annex/ were written with modes that did not take the core.sharedRepository config into account.
git grep writeFile finds some more that might also be problems, but
for now I've concentrated on .git/annex/ log files. There are certianly
cases where writeFile is not a problem too.

This commit was sponsored by mo on Patreon.
2018-01-02 17:25:25 -04:00
Joey Hess
75366d3c34
split BuildInfo and BuildFlags
The problem with combining these is that Build.Standalone etc need only
the BuildInfo, and since not built with cabal, the BuildFlags ifdefs
were causing bogus warnings.
2018-01-02 13:47:51 -04:00
Joey Hess
25703e1413
finally really add back custom-setup stanza
Fourth or fifth try at this and finally found a way to make it work.

Absurd amount of busy-work forced on me by change in cabal's behavior.
Split up Utility modules that need posix stuff out of ones used by
Setup. Various other hacks around inability for Setup to use anything
that ifdefs a use of unix.

Probably lost a full day of my life to this.
This is how build systems make their users hate them. Just saying.
2017-12-31 16:36:39 -04:00
Joey Hess
1f5bf73af0
Revert "git-annex.cabal: Add back custom-setup stanza, so cabal new-build works."
This reverts commit 51228c2306.

No, still doesn't work when built with cabal. It did with stack; stack
must somehow make the unix package implicitly available.

With cabal, System.Posix.Process and System.Posix.Env are both missing.
2017-12-31 14:09:41 -04:00
Joey Hess
51228c2306
git-annex.cabal: Add back custom-setup stanza, so cabal new-build works.
Seems I had all the work in past commits to make this build, at least on
linux. I'm actually surprised it does, without a unix dep, Utility.Env
still builds ok somehow despite using System.Posix.Env.

This commit was sponsored by Fernando Jimenez on Patreon.
2017-12-31 13:54:41 -04:00
Joey Hess
67338fd7ac
Added inprogress command for accessing files as they are being downloaded.
Chose to make this only handle files actively being downloaded, not temp
files for downloads that were interrupted or files that have been fully
downloaded.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2017-12-28 11:46:39 -04:00
Joey Hess
79857d7e9f
Removed the testsuite build flag
Test suite is always included.

Building with this flag disabled has actually been broken for some time,
since Command.TestRemote uses tasty. Fewer build flags are better, so good
time to drop it.

This commit was sponsored by Thomas Hochstein on Patreon.
2017-12-20 12:25:03 -04:00
Joey Hess
adeaa84ac3
add git-annex-export.mdwn 2017-12-14 13:32:24 -04:00
Joey Hess
61b515d71d
releasing package git-annex version 6.20171214 2017-12-14 11:54:36 -04:00
Joey Hess
f00770d736
Removed no longer needed dependency on yesod-default.
This commit was sponsored by Nick Daly on Patreon.
2017-12-05 13:18:25 -04:00
Joey Hess
24f27ec39d
convert importfeed to youtube-dl
Fully working, including --fast/--relaxed.

Note that, while git-annex addurl --relaxed is not going to check
youtube-dl, I kept git annex importfeed --relaxed checking it.
Thinking is that, let's not break people's importfeed cron jobs, and
importfeed does not typically have to check a large number of new items,
so it's ok if it's a little bit slower when used with youtube playlist
feeds.

importfeed's behavior is also improved (?) when a feed has links in it
to non-media files. Before, those were skipped. Now, the content of the
link is downloaded. This had to be done, because trying to use
youtube-dl is slow, and if those were skipped, it would have to check
every time importfeed was run. While this behavior change may not be
desirable for some feeds, that intersperse links to web pages with
enclosures, it will be desirable for other feeds, that have
non-enclosure directy links to media files.

Remove old quvi modules.

This commit was sponsored by Øyvind Andersen Holm.
2017-11-29 17:30:02 -04:00
Joey Hess
99bebdface
youtube-dl working
Including resuming and cleanup of incomplete downloads.

Still todo: --fast, --relaxed, importfeed, disk reserve checking,
quvi code cleanup.

This commit was sponsored by Anthony DeRobertis on Patreon.
2017-11-29 16:40:32 -04:00
Joey Hess
57b4c5bdff
add Utility.HtmlDetect
This will be used in youtube-dl integration, to tell when a html page has
been downloaded by addurl, in which case it is worth running youtube-dl
to see if it can extract media from it.

tagsoup is an almost free dependency, because yesod depends on it.
So, this only really adds a dep when git-annex is built without the
webapp.

I'd like this to as closely as possible match how browsers decide if a
page is html or not. Unfortunately, that is fairly heuristic, in order
to support malformed html. And, we don't want to falsely detect
something as html just because it has something that looks like a html
tag embedded somewhere in it. Probably any major video hosting site is
going to be serving html documents that at least start with a <html>
tag, so requiring that or a DOCTYPE should be good enough.

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-11-28 13:03:11 -04:00
Joey Hess
28565919af
releasing package git-annex version 6.20171124 2017-11-24 10:49:57 -04:00
Joey Hess
187b3e7780
enable LambdaCase and convert around 10% of places that could use it
Needs ghc 7.6.1, so minimum base version increased slightly. All builds
are well above this version of ghc, and debian oldstable is as well.

Code that could use lambdacase can be found by running:
git grep -B 1 'case ' | less
and searching in less for "<-"

This commit was sponsored by andrea rota.
2017-11-15 16:59:32 -04:00
Joey Hess
1b6cbb63e9
still can't express custom-setup deps
They need unix on non-windows, for Utility.Env, which Build.Configure uses,
but cabal can't express that in a custom-setup stanza.

To avoid this problem, Utility.Env would need to be moved into
unix-compat..
2017-11-14 14:59:51 -04:00
Joey Hess
c598cef0b6
typo 2017-11-14 14:34:09 -04:00
Joey Hess
87293e5be9
add utf8-string to custom-setup deps 2017-11-14 14:32:01 -04:00
Joey Hess
8d68112be5
split out setEnv to avoid adding dep
Windows needs the setenv package in custom-setup, but I don't want to
pull it in on unix, which would probably break some builds and need more
work. Instead, split out setEnv to a separate module.

Quite likely, unix-compat will get a portable environment layer, and
then both modules can be removed from here.

This commit was sponsored by Øyvind Andersen Holm.
2017-11-14 14:28:49 -04:00
Joey Hess
ea12fb38f4
bring back custom-setup stanza
Now that windows is using unix-compat 0.5, don't need a dep on Win32
or unix in custom-setup, so the stanza can be parsed by cabal's limited
parser again.

Note that unix-compat 0.5 is needed on windows, but the cabal file
hasn't bumped it yet, since that would necessarily affect non-windows
builds. Instead, the stack-windows.yaml makes sure the right version is
pulled in.

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-11-14 14:14:44 -04:00
Joey Hess
8dd84b87f9
use unix-compat 0.5 on windows
Re-applying 3ec579f5e1
2017-11-14 14:00:24 -04:00
Joey Hess
1bd956bed4
Revert "Revert "remove dep on Win32-extras""
This reverts commit d18bc52caf.
2017-11-13 12:55:23 -04:00
Joey Hess
77341f135b
use win32 2.6.1.0 (second try)
Using patched http-client.
2017-11-13 12:36:07 -04:00
Joey Hess
5f55082d10
Revert "use unix-compat 0.5 on windows"
This reverts commit 3ec579f5e1.

Too early for this; needs newer Win32 version. Le sigh.
2017-11-09 15:14:00 -04:00
Joey Hess
d18bc52caf
Revert "remove dep on Win32-extras"
This reverts commit 8b5480c66a.

Yeah, too early for that too
2017-11-09 15:09:14 -04:00
Joey Hess
8d6ff88218
Revert "use win32 2.6.1.0"
This reverts commit a0c38d7c96.

Blocked by https://github.com/snoyberg/http-client/issues/309
this commit can be reverted once that's fixed.
2017-11-09 15:01:59 -04:00
Joey Hess
928567f5f0
bump process to version supporting newer Win32 2017-11-09 14:28:54 -04:00
Joey Hess
8b5480c66a
remove dep on Win32-extras
Win32 now has its own getCurrentProcessId.
2017-11-09 14:03:06 -04:00
Joey Hess
25854c73db
still need stack-windows.yaml to specify newer versions of Win32 and unix-compat 2017-11-09 13:54:05 -04:00
Joey Hess
a0c38d7c96
use win32 2.6.1.0
That has my patches merged into it, so stack-windows.yaml is not needed
any longer.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2017-11-09 13:39:07 -04:00
Joey Hess
3ec579f5e1
use unix-compat 0.5 on windows
That version has my patches for the problems that Utility.PosixFiles
was working around, so am able to get rid of that module now.

This will later allow bringing back the custom-setup stanza in the cabal
file. It will need to depend on unix-compat 0.5 on all OS's, which I'm
not ready to do yet.

This commit was sponsored by Nick Daly on Patreon.
2017-11-09 12:47:05 -04:00
Joey Hess
01d940dc10
releasing package git-annex version 6.20171109 2017-11-09 12:23:34 -04:00
Joey Hess
20b04a465f
remove custom-setup again until cabal issue is fixed 2017-10-27 14:35:03 -04:00
Joey Hess
d6573dab1d
this is the custom-setup I want
Unfortunately, cabal fails to parse this.
https://github.com/haskell/cabal/issues/4852
2017-10-27 14:20:58 -04:00
Joey Hess
15f680d33b
try putting back custom-setup
will this build on doze?
2017-10-27 14:03:50 -04:00
Joey Hess
43d011a524
releasing package git-annex version 6.20171026 2017-10-26 14:02:40 -04:00
Joey Hess
833b3f06cd
build for windows with forked win32 package that has terminateProcessId
Get ugly reversion out of CHANGELOG.

Also, relocated the windows stack.yaml to top, and updated windows build
instructions.

This commit was sponsored by Henrik Riomar on Patreon.
2017-10-25 14:45:23 -04:00
Joey Hess
901807cf75
Revert "try to avoid TerminateProcess link error on windows"
This reverts commit 839ec7e26c.

Neither way is working.. The other way failed:

.stack-work\dist\5f9bc736\build\git-annex\git-annex-tmp\Assistant.o:fake:(.text+0x6bb3): undefined reference to `terminatepid'

Seems that winprocess.c is not getting linked in.
2017-10-24 13:05:24 -04:00
Joey Hess
bcd394657f
indentation 2017-10-24 12:54:33 -04:00
Joey Hess
75dc8230e1
add includes to cabal 2017-10-24 12:52:21 -04:00
Joey Hess
839ec7e26c
try to avoid TerminateProcess link error on windows
Building with stack, it failed:

`_TerminateProcess' referenced in section `.text' of .stack-work\dist\5f9bc736\build\git-annex\git-annex-tmp\Utility\WinProcess.o: defined in discarded section `.text' of C:/Users/jenkins/AppData/Local/Programs/stack/i386-windows/ghc-8.0.2/mingw/bin/../lib/gcc/i686-w64-mingw32/5.2.0/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(dacgs01154.o)

This is a reversion of 86e638567a,
to try the other way to implement it, which will hopefully avoid the problem.
2017-10-24 12:33:44 -04:00
Joey Hess
0e766a41c6
try lts-9.10 with old win32
toolchain issue is preventing nightly from building anything on windows
2017-10-22 14:20:54 -04:00
Joey Hess
2412744e0b
temporarily remove custom-setup stanza
This needs to include unix except on windows, but when I tried
an if (! os(windows)) cabal crashed:

'parseField' called on a non-field. This is possibly a bug in Cabal.

Cabal is able to configure w/o the custom-setup stanza, so omit it.
2017-10-22 14:08:55 -04:00
Joey Hess
11928ae432
add IfElse version 2017-10-22 14:08:45 -04:00
Joey Hess
bb20b1ed35
releasing package git-annex version 6.20171018 2017-10-18 15:41:08 -04:00
Joey Hess
20d5d12079
Revert "new Win32 fails to build; fall back to old one"
This reverts commit ffa81d2d7f.

Old win32-extras fails to install. I suspect toolchain breakage because
a lot of stuff is failing on the windows autobuilder.
2017-10-16 15:38:35 -04:00
Joey Hess
ffa81d2d7f
new Win32 fails to build; fall back to old one 2017-10-16 15:26:29 -04:00
Joey Hess
e7a9ed330f
releasing package git-annex version 6.20171003 2017-10-03 13:18:47 -04:00
Joey Hess
0a34ad1a54
fix build without S3 2017-09-29 20:56:14 -04:00
Joey Hess
76c9b580b3
releasing package git-annex version 6.20170925 2017-09-25 11:34:55 -04:00
Joey Hess
d71c65ca0a
add exporter thread to assistant
This is similar to the pusher thread, but a separate thread because git
pushes can be done in parallel with exports, and updating a big export
should not prevent other git pushes going out in the meantime.

The exportThread only runs at most every 30 seconds, since updating an
export is more expensive than pushing. This may need to be tuned.

Added a separate channel for export commits; the committer records a
commit in that channel.

Also, reconnectRemotes records a dummy commit, to make the exporter
thread wake up and make sure all exports are up-to-date. So,
connecting a drive with a directory special remote export will
immediately update it, and getting online will automatically
update S3 and WebDAV exports.

The transfer queue is not involved in exports. Instead, failed
exports are retried much like failed pushes.

This commit was sponsored by Ewen McNeill.
2017-09-20 15:29:13 -04:00
Joey Hess
b03d77c211
add ExportTree table to export db
New table needed to look up what filenames are used in the currently
exported tree, for reasons explained in export.mdwn.

Also, added smart constructors for ExportLocation and ExportDirectory to
make sure they contain filepaths with the right direction slashes.

And some code refactoring.

This commit was sponsored by Francois Marier on Patreon.
2017-09-18 13:59:59 -04:00
Joey Hess
e1f5c90c92
split out Types.Export 2017-09-15 16:46:03 -04:00
Joey Hess
969ac892a8
add Command.Export 2017-09-13 14:21:08 -04:00
Joey Hess
7eb9889bfd
track exported files in a sqlite database
Went with a separate db per export remote, rather than a single export
database. Mostly because there will probably not be a lot of separate
export remotes, and it might be convenient to be able to delete a given
remote's export database.

This commit was supported by the NSF-funded DataLad project.
2017-09-04 13:53:08 -04:00
Joey Hess
a4328b49d2
refactor ExportActions
This will allow disabling exports for remotes that are not configured to
allow them. Also, exportSupported will be useful for the external
special remote to probe.

This commit was supported by the NSF-funded DataLad project
2017-09-01 13:05:09 -04:00
Joey Hess
978885247e
implement export.log and resolve export conflicts
Incremental export updates work now too.

This commit was sponsored by Anthony DeRobertis on Patreon.
2017-08-31 15:47:23 -04:00
Joey Hess
252994e095
releasing package git-annex version 6.20170818 2017-08-18 11:19:14 -04:00
Joey Hess
55495c5a98
git-annex.cabal: Deal with breaking changes in Cabal 2.0
https://github.com/haskell/cabal/issues/4655

This means that when a module is conditionally imported via ifdef
depending on the OS or build flags, the cabal file has to mirror the
same logic there to only list the module then.

Since there are lots of OS's and lots of combinations of build flags
here, it's rather difficult to know if the cabal file has been completelty
correctly updated to match the source code.

So I am very unhappy with needing to update things in two places. I've
only tested this on linux with most build flags enables; this will
probably need significant time and testing to catch every cabal file
tweak that this change to Cabal requires. And it will be a continual
source of compile failures going forward when the code is modified and
the cabal file not also updated.

DRY DRY DRY, I repeat myself, but: DRY! Sigh..

(Also, had to remove all Build.* that are standalone programs from the
Other-Modules list, because since cabal passes those modules to ghc when
building git-annex, it complains that they use module Main. Those
modules are only used when building with the Makefile anyway, so this
change shouldn't break anything.)

This commit was sponsored by Thomas Hochstein on Patreon.
2017-08-18 11:08:58 -04:00
Joey Hess
df11e54788
avoid the dashed ssh hostname class of security holes
Security fix: Disallow hostname starting with a dash, which would get
passed to ssh and be treated an option. This could be used by an attacker
who provides a crafted ssh url (for eg a git remote) to execute arbitrary
code via ssh -oProxyCommand.

No CVE has yet been assigned for this hole.
The same class of security hole recently affected git itself,
CVE-2017-1000117.

Method: Identified all places where ssh is run, by git grep '"ssh"'
Converted them all to use a SshHost, if they did not already, for
specifying the hostname.

SshHost was made a data type with a smart constructor, which rejects
hostnames starting with '-'.

Note that git-annex already contains extensive use of Utility.SafeCommand,
which fixes a similar class of problem where a filename starting with a
dash gets passed to a program which treats it as an option.

This commit was sponsored by Jochen Bartl on Patreon.
2017-08-17 22:11:31 -04:00
Joey Hess
d39c120afa
add annex-ignore-command and annex-sync-command configs
Added remote configuration settings annex-ignore-command and
annex-sync-command, which are dynamic equivilants of the annex-ignore
and annex-sync configurations.

For this I needed a new DynamicConfig infrastructure. Its implementation
should be as fast as before when there is no dynamic config, and it caches
so shell commands are only run once.

Note that annex-ignore-command exits nonzero when the remote should be ignored.
While that may seem backwards, it allows using the same command for it as
for annex-sync-command when you want to disable both.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 13:54:14 -04:00
Joey Hess
4173decf27
Windows: Win32 package has subsumed Win32-extras; update dependency. 2017-08-16 17:43:38 -04:00
Joey Hess
2cecc8d2a3
Added GIT_ANNEX_VECTOR_CLOCK environment variable
Can be used to override the default timestamps used in log files in the
git-annex branch. This is a dangerous environment variable; use with
caution.

Note that this only affects writing to the logs on the git-annex branch.
It is not used for metadata in git commits (other env vars can be set for
that).

There are many other places where timestamps are still used, that don't
get committed to git, but do touch disk. Including regular timestamps
of files, and timestamps embedded in some files in .git/annex/, including
the last fsck timestamp and timestamps in transfer log files.

A good way to find such things in git-annex is to get for getPOSIXTime and
getCurrentTime, although some of the results are of course false positives
that never hit disk (unless git-annex gets swapped out..)

So this commit does NOT necessarily make git-annex comply with some HIPPA
privacy regulations; it's up to the user to determine if they can use it in
a way compliant with such regulations.

Benchmarking: It takes 0.00114 milliseconds to call getEnv
"GIT_ANNEX_VECTOR_CLOCK" when that env var is not set. So, 100 thousand log
files can be written with an added overhead of only 0.114 seconds. That
should be by far swamped by the actual overhead of writing the log files
and making the commit containing them.

This commit was supported by the NSF-funded DataLad project.
2017-08-14 14:19:58 -04:00
Joey Hess
e4100fd60e
releasing package git-annex version 6.20170520 2017-06-12 13:55:00 -04:00
Joey Hess
c6079c3ce8
releasing package git-annex version 6.20170519 2017-05-19 10:58:03 -04:00
Joey Hess
a1730cd6af
adeiu, MissingH
Removed dependency on MissingH, instead depending on the split
library.

After laying groundwork for this since 2015, it
was mostly straightforward. Added Utility.Tuple and
Utility.Split. Eyeballed System.Path.WildMatch while implementing
the same thing.

Since MissingH's progress meter display was being used, I re-implemented
my own. Bonus: Now progress is displayed for transfers of files of
unknown size.

This commit was sponsored by Shane-o on Patreon.
2017-05-16 01:03:52 -04:00
Joey Hess
6dd806f1ad
stop using MissingH for MD5
Cryptonite is faster and allocates less, and I want to get rid of
MissingH use.

Note that the new dependency on memory is free; it's a dependency of
cryptonite.

This commit was supported by the NSF-funded DataLad project.
2017-05-15 21:36:03 -04:00
Joey Hess
e40d9a1b12
add Other-Extensions: TemplateHaskell
Fixes a build failure of stack build --profile.
See https://ghc.haskell.org/trac/ghc/ticket/8443
2017-05-15 16:50:03 -04:00
Joey Hess
884505279a
releasing package git-annex version 6.20170510 2017-05-10 15:37:16 -04:00
Joey Hess
76c63a4a66
avoiding depending on latest version of process except on Windows 2017-04-10 12:14:24 -04:00
Joey Hess
e3184e54c9
version: Added "dependency versions" line.
This commit was sponsored by Anthony DeRobertis on Patreon.
2017-04-07 18:16:11 -04:00
Joey Hess
c3970f6c1a
multicast: New command, uses uftp to multicast annexed files, for eg a classroom setting.
This commit was supported by the NSF-funded DataLad project.
2017-03-30 19:35:30 -04:00
Joey Hess
464291243c
releasing package git-annex version 6.20170321 2017-03-21 13:46:20 -04:00
Joey Hess
faecd73f32
Support GIT_SSH and GIT_SSH_COMMAND
They are handled close the same as they are by git. However, unlike git,
git-annex sometimes needs to pass the -n parameter when using these.

So, this has the potential for breaking some setup, and perhaps there ought
to be a ANNEX_USE_GIT_SSH=1 needed to use these. But I'd rather avoid that
if possible, so let's see if anyone complains.

Almost all places where "ssh" was run have been changed to support the env
vars. Anything still calling sshOptions does not support them. In
particular, rsync special remotes don't. Seems that annex-rsync-transport
already gives sufficient control there.

(Fixed in passing: Remote.Helper.Ssh.toRepo used to extract
remoteAnnexSshOptions and pass them to sshOptions, which was redundant
since sshOptions also extracts those.)

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-03-17 16:20:37 -04:00
Joey Hess
1c4e5f65fc
Drop support for building with old versions of directory, feed, and http-types. 2017-03-10 15:57:41 -04:00
Joey Hess
9ef7207d5a
Revert "Drop support for building without network-uri."
This reverts commit fc3925a1cd.

Need it in stable w/o backports for the ancient autobuilder.
2017-03-10 15:49:18 -04:00
Joey Hess
ca49a84ba5
Drop support for building with old versions of dns and http-conduit. 2017-03-10 15:49:14 -04:00
Joey Hess
fc3925a1cd
Drop support for building without network-uri.
network-uri is available in Debian stable (backports) and testing,
so no need to complicate the cabal file anymore
2017-03-10 15:38:15 -04:00
Joey Hess
a9e1e17d40
releasing package git-annex version 6.20170301.1 2017-03-01 12:46:26 -04:00
Joey Hess
254b57aef7
6.20170301 version for hackage
No changes from 6.20170228; a new version number was needed due to a problem with Hackage.
2017-03-01 12:06:10 -04:00
Joey Hess
444278156c
releasing package git-annex version 6.20170228 2017-02-28 14:41:57 -04:00
Joey Hess
40327cab6e
Removed support for building with the old cryptohash library.
Building with that library made git-annex not support SHA3; it's time for
that to always be supported in case SHA2 dominoes.
2017-02-24 20:56:26 -04:00
Joey Hess
ca0daa8bb8
factor non-type stuff out of Key 2017-02-24 13:42:30 -04:00
Joey Hess
c5cf5cf03a
git-annex.cabal: Make crypto-api a dependency even when built w/o webapp and test suite.
The p2p code made it always be needed.

This commit was sponsored by Anthony DeRobertis on Patreon.
2017-02-20 12:21:35 -04:00
Joey Hess
e6857e75a6
sync hack to make updateInstead work on eg FAT
sync: When syncing with a local repository located on a crippled
filesystem, run the post-receive hook there, since it wouldn't get run
otherwise. This makes pushing to repos on FAT-formatted removable drives
update them when receive.denyCurrentBranch=updateInstead.

Made Remote.Git export onLocal, which was cleaned up to not have so many
caveats about its use.

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-02-17 15:21:52 -04:00
Joey Hess
d074532aff
post-recive hook to make updateInstead work in direct mode and adjusted branches
* Added post-recieve hook, which makes updateInstead work with direct
  mode and adjusted branches.
* init: Set up the post-receive hook.

This commit was sponsored by Fernando Jimenez on Patreon.
2017-02-17 14:04:43 -04:00
Joey Hess
95390f0c27
releasing package git-annex version 6.20170214 2017-02-14 14:56:11 -04:00
Joey Hess
3b22ad9f47
Work around sqlite's incorrect handling of umask when creating databases.
Refactored some common code into initDb.

This only deals with the problem when creating new databases. If a repo
got bad permissions into it, it's up to the user to deal with it.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2017-02-13 17:39:16 -04:00
Joey Hess
ed56dba868
annex.autocommit can be configured via git-annex config
... to control the default behavior in all clones of a repository.

This includes a new Configurable data type, so the GitConfig type indicates
which values can be configured this way.

The implementation should be quite efficient; the config log is only read
once, and only when a Configurable value has not already been set by
git-config.

Indeed, it would be nice in the future to extend this, so that git-config
is itself only read on demand. Some commands may not need to look at the
git configuration at all.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-02-03 13:58:53 -04:00
Joey Hess
a130bc4f9b
add new modules to annoying list 2017-01-31 17:27:44 -04:00
Joey Hess
8740cd9716
releasing package git-annex version 6.20170101 2016-12-31 23:59:56 -04:00
Joey Hess
b68d2a4b68
webapp: full wormhole pairing UI (untested)
This commit was sponsored by Riku Voipio.
2016-12-27 16:41:35 -04:00
Joey Hess
9e0aae036b
webapp: check that tor and magic wormhole are installed 2016-12-24 17:08:03 -04:00
Joey Hess
a196260924
mocked up wormhole pairing interface in webapp 2016-12-24 16:55:36 -04:00
Joey Hess
ab66bbfeb6
Merge branch 'master' into no-xmpp 2016-12-24 15:01:55 -04:00
Joey Hess
f7ca2b92fb
enable-tor: No longer needs to be run as root.
When run by not root, su's to root automatically.

This commit was sponsored by Brock Spratlen on Patreon.
2016-12-20 17:40:36 -04:00
Joey Hess
ccde0932a5
p2p --pair with magic wormhole (untested)
It builds. I have not tried to run it yet. :)

This commit was sponsored by Jake Vosloo on Patreon.
2016-12-18 16:51:41 -04:00
Joey Hess
fe6f36d9f3
magic wormhole module
This interacts with it using stdio, which is surprisingly hard.

sendFile does not currently work, due to
https://github.com/warner/magic-wormhole/issues/108

Parsing the output to find the magic code is done as robustly as
possible, and should continue to work unless wormhole radically changes
the format of its codes. Presumably it will never output something that
looks like a wormhole code before the actual wormhole code; that would
also break this. It would be better if there was a way to make
wormhole not mix the code with other output, as requested in
https://github.com/warner/magic-wormhole/issues/104

Only exchange of files/directories is supported. To exchange messages,
https://github.com/warner/magic-wormhole/issues/99 would need to be resolved.
I don't need message exchange however.
2016-12-17 16:58:05 -04:00
Joey Hess
73a79147b1
releasing package git-annex version 6.20161210 2016-12-10 12:23:18 -04:00
Alper Nebi Yasak
93a22a1c97
Remove http-conduit (<2.2.0) constraint
Since https://github.com/aristidb/aws/issues/206 is resolved, this
constraint is no longer necessary. However, http-conduit (>=2.2.0)
requires http-client (>=0.5.0) which introduces some breaking changes.
This commit also implements those changes depending on the version.
Fixes: https://git-annex.branchable.com/bugs/Build_with_aws_head_fails/

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
2016-12-10 10:45:52 -04:00
Joey Hess
e152c322f8
refactor ref change watching
Added to change notification to P2P protocol.

Switched to a TBChan so that a single long-running thread can be
started, and serve perhaps intermittent requests for change
notifications, without buffering all changes in memory.

The P2P runner currently starts up a new thread each times it waits
for a change, but that should allow later reusing a thread. Although
each connection from a peer will still need a new watcher thread to run.

The dependency on stm-chans is more or less free; some stuff in yesod
uses it, so it was already indirectly pulled in when building with the
webapp.

This commit was sponsored by Francois Marier on Patreon.
2016-12-09 15:01:09 -04:00
Joey Hess
d8f1a0d95c
move byteable to main dep list
Only the webapp had pulled it in, but the authtoken code uses it now.
2016-12-08 16:31:08 -04:00
Joey Hess
f744bd5391
refactor 2016-12-06 15:43:03 -04:00
Joey Hess
b29088b8dc
stub Remote.P2P
Similar to GCrypt remotes, P2P remotes have an url, so Remote.Git has to
separate them out and handle them, passing off to Remote.P2P.

This commit was sponsored by Ignacio on Patreon.
2016-12-06 12:27:58 -04:00
Joey Hess
881274d021
make remote-daemon able to send and receive objects over tor
Each worker thread needs to run in the Annex monad, but the
remote-daemon's liftAnnex can only run 1 action at a time. Used
Annex.Concurrent to deal with that.

P2P.Annex is incomplete as of yet.
2016-12-02 13:52:43 -04:00
Joey Hess
bfc8305814
implement p2p command 2016-11-30 14:35:24 -04:00
Joey Hess
38425fdc39
finish git-annex enable-tor
Make it stash the address away for git-annex p2p to use later, rather
than outputting it. And, look up the UUID itself.
2016-11-29 17:30:27 -04:00
Joey Hess
158ef45d76
add P2P.Auth 2016-11-22 14:37:50 -04:00
Joey Hess
b08799893f
reorg 2016-11-22 14:37:09 -04:00
Joey Hess
af4d919793
unified AuthToken type between webapp and tor 2016-11-22 14:18:34 -04:00
Joey Hess
070fb9e624
Added git-remote-tor-annex, which allows git pull and push to the tor hidden service.
Almost working, but there's a bug in the relaying.

Also, made tor hidden service setup pick a random port, to make it harder
to port scan.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2016-11-21 17:27:38 -04:00
Joey Hess
74691ddf0e
remotedaemon: serve tor hidden service 2016-11-20 15:48:12 -04:00
Joey Hess
d50b0f3bb3
implement p2p protocol for Handle
This is most of the way to having the p2p protocol working over tor
hidden services, at least enough to do git push/pull.

The free monad was split into two, one for network operations and the
other for local (Annex) operations. This will allow git-remote-tor-annex
to run only an IO action, not needing the Annex monad.

This commit was sponsored by Remy van Elst on Patreon.
2016-11-20 12:16:32 -04:00
Joey Hess
9d9d1fdcd4
Merge branch 'master' into tor 2016-11-18 20:05:34 -04:00
Joey Hess
5680565122
releasing package git-annex version 6.20161118 2016-11-18 11:59:49 -04:00