Commit graph

56 commits

Author SHA1 Message Date
Joey Hess
1b63132ca3
add searchPathContents
And rename related functions for consistency.
2021-02-02 19:06:15 -04:00
Joey Hess
5cfcf1f05f
cache remote.log
Unlikely to speed up any of the existing uses much, but I want to use it
in a message that might be displayed many times.
2020-09-22 13:52:26 -04:00
Joey Hess
6ea511beb4
Removed the S3 and WebDAV build flags
So these special remotes are always supported.

IIRC these build flags were added because the dep chains were a bit too
long, or perhaps because the libraries were not available in Debian stable,
or something like that. That was long ago, those reasons no longer apply,
and users get confused when builtin special remotes are not available, so
it seems best to remove the build flags now.

If this does cause a problem it can be reverted of course..

This commit was sponsored by Jochen Bartl on Patreon.
2020-09-08 12:42:59 -04:00
Joey Hess
41952204ce
S3: The REDUCED_REDUNDANCY storage class is no longer cheaper
So stop documenting it, and stop offering it as a choice in the assistant.

Removed the code that parses it into S3.ReducedRedundancy, because
S3.OtherStorageClass with the value will work just the same and avoids a
special case for a deprecated this.
2020-06-16 12:04:29 -04:00
Joey Hess
8af6d2c3c5
fix encryption of content to gcrypt and git-lfs
Fix serious regression in gcrypt and encrypted git-lfs remotes.
Since version 7.20200202.7, git-annex incorrectly stored content
on those remotes without encrypting it.

Problem was, Remote.Git enumerates all git remotes, including git-lfs
and gcrypt. It then dispatches to those. So, Remote.List used the
RemoteConfigParser from Remote.Git, instead of from git-lfs or gcrypt,
and that parser does not know about encryption fields, so did not
include them in the ParsedRemoteConfig. (Also didn't include other
fields specific to those remotes, perhaps chunking etc also didn't
get through.)

To fix, had to move RemoteConfig parsing down into the generate methods
of each remote, rather than doing it in Remote.List.

And a consequence of that was that ParsedRemoteConfig had to change to
include the RemoteConfig that got parsed, so that testremote can
generate a new remote based on an existing remote.

(I would have rather fixed this just inside Remote.Git, but that was not
practical, at least not w/o re-doing work that Remote.List already did.
Big ugly mostly mechanical patch seemed preferable to making git-annex
slower.)
2020-02-26 18:05:36 -04:00
Joey Hess
99cb3e75f1
add LISTCONFIGS to external special remote protocol
Special remote programs that use GETCONFIG/SETCONFIG are recommended
to implement it.

The description is not yet used, but will be useful later when adding a way
to make initremote list all accepted configs.

configParser now takes a RemoteConfig parameter. Normally, that's not
needed, because configParser returns a parter, it does not parse it
itself. But, it's needed to look at externaltype and work out what
external remote program to run for LISTCONFIGS.

Note that, while externalUUID is changed to a Maybe UUID, checkExportSupported
used to use NoUUID. The code that now checks for Nothing used to behave
in some undefined way if the external program made requests that
triggered it.

Also, note that in externalSetup, once it generates external,
it parses the RemoteConfig strictly. That generates a
ParsedRemoteConfig, which is thrown away. The reason it's ok to throw
that away, is that, if the strict parse succeeded, the result must be
the same as the earlier, lenient parse.

initremote of an external special remote now runs the program three
times. First for LISTCONFIGS, then EXPORTSUPPORTED, and again
LISTCONFIGS+INITREMOTE. It would not be hard to eliminate at least
one of those, and it should be possible to only run the program once.
2020-01-17 16:07:17 -04:00
Joey Hess
fa28b51206
fix build of webapp 2020-01-15 13:47:31 -04:00
Joey Hess
71ecfbfccf
be stricter about rejecting invalid configurations for remotes
This is a first step toward that goal, using the ProposedAccepted type
in RemoteConfig lets initremote/enableremote reject bad parameters that
were passed in a remote's configuration, while avoiding enableremote
rejecting bad parameters that have already been stored in remote.log

This does not eliminate every place where a remote config is parsed and a
default value is used if the parse false. But, I did fix several
things that expected foo=yes/no and so confusingly accepted foo=true but
treated it like foo=no. There are still some fields that are parsed with
yesNo but not not checked when initializing a remote, and there are other
fields that are parsed in other ways and not checked when initializing a
remote.

This also lays groundwork for rejecting unknown/typoed config keys.
2020-01-10 14:52:48 -04:00
Joey Hess
d1130ea04a
get rid of hardcoded "name" lookups
Support "sameas-name" being set instead.

In RenameRemote, rename which ever of the two is set.
2019-10-10 13:25:10 -04:00
Joey Hess
9dc7a10448
Drop support for building with aws older than 0.14.
debian stable has 0.14 so lose the complexity for old versions
2019-04-19 14:27:59 -04:00
Joey Hess
0a4479b8ec
Avoid backtraces on expected failures when built with ghc 8; only use backtraces for unexpected errors.
ghc 8 added backtraces on uncaught errors. This is great, but git-annex was
using error in many places for a error message targeted at the user, in
some known problem case. A backtrace only confuses such a message, so omit it.

Notably, commands like git annex drop that failed due to eg, numcopies,
used to use error, so had a backtrace.

This commit was sponsored by Ethan Aubin.
2016-11-15 21:29:54 -04:00
Joey Hess
b0e5c09408
fix various build warnings, mostly on Windows
And some when S3 is disabled
2015-10-13 13:24:44 -04:00
Joey Hess
16d7035ef7 Allow building with S3 disabled again. 2015-10-02 12:12:02 -04:00
Joey Hess
20205b6073 avoid hard dependency on new version of aws 2015-09-22 11:04:26 -04:00
Joey Hess
26d6566307 S3 storage classes expansion
Added support for storageclass=STANDARD_IA to use Amazon's
new Infrequently Accessed storage.

Also allows using storageclass=NEARLINE to use Google's NearLine storage.

The necessary changes to aws to support this are in
https://github.com/aristidb/aws/pull/176
2015-09-17 17:20:01 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
31a1f061c6 typo 2014-10-23 20:25:31 -04:00
Joey Hess
fa1318479e rename isIA to configIA
Already done on s3-aws branch, so reduce divergence.
2014-10-23 15:56:35 -04:00
Joey Hess
1b90838bbd add internet archive item url to info 2014-10-21 15:34:32 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
32e4368377 S3: support chunking
The assistant defaults to 1MiB chunk size for new S3 special remotes.
Which will work around a couple of bugs:
  http://git-annex.branchable.com/bugs/S3_memory_leaks/
  http://git-annex.branchable.com/bugs/S3_upload_not_using_multipart/
2014-08-02 15:51:58 -04:00
Sören Brunk
00c1cd0db1 bootstrap3 forms 2014-04-21 19:47:05 +02:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
958312885f webapp: Improve UI around remote that have no annex.uuid set, either because setup of them is incomplete, or because the remote git repository is not a git-annex repository.
Complicated by such repositories potentially being repos that should have
an annex.uuid, but it failed to be gotten, perhaps due to the past ssh repo
setup bugs. This is handled now by an Upgrade Repository button.
2013-11-07 18:02:00 -04:00
Joey Hess
26d95e86d7 rename module 2013-10-28 11:33:14 -04:00
Joey Hess
267f8b0bb5 switch to runFormPostNoToken to work around strange yesod bug
I am not happy about disabling yesod's XSRF tokens, but the webapp has two
guards of its own that should suffice: Listening only to localhost
(normally) and requiring its own auth token on every single request
(always).
2013-10-14 12:19:11 -04:00
Joey Hess
dfdaa649d0 hlint 2013-10-02 01:06:59 -04:00
Joey Hess
d83a244986 UI for making encrypted ssh remotes with gcrypt
Improved probing the remote server, so it gathers a list of the
capabilities it has. From that list, we can determine which types
of remotes are supported, and display an appropriate UI.

The new buttons for making gcrypt repos don't work yet, but the old buttons
for unencrypted git repo and encrypted rsync repo have been adapted to the
new data types and are working.

This commit was sponsored by David Schmitt.
2013-09-29 15:14:09 -04:00
Joey Hess
7665773593 refactor 2013-09-27 00:27:59 -04:00
Joey Hess
588494cbce webapp: Support storing encrypted git repositories on rsync.net.
Does not yet support re-enabling such a repository though.

This commit was sponsored by Jan Pieper.
2013-09-26 16:43:00 -04:00
Joey Hess
9fc1448947 webapp: Differentiate between creating a new S3/Glacier/WebDav remote, and initializing an existing remote. When creating a new remote, avoid conflicts with other existing (or deleted) remotes with the same name. 2013-07-20 18:15:16 -04:00
Joey Hess
ff4f008591 clean up build warnings with yesod 1.2, while still building with 1.1 2013-06-27 01:15:28 -04:00
Joey Hess
b42fe2283a remove unnecessary haskell extensions 2013-06-04 21:13:20 -04:00
Joey Hess
1198b5444d now builds with both yesod 1.2 and 1.1 2013-06-03 18:14:22 -04:00
Joey Hess
31753bad46 add liftH shim between yesod versions, to avoid needing zillions of ifdefs 2013-06-03 13:51:54 -04:00
Joey Hess
79fd677805 WIP yesod 1.2 2013-06-03 11:25:23 -04:00
Joey Hess
d9e288b0fd fix build with S3 disabled (for Android) 2013-05-01 13:27:51 -04:00
Joey Hess
c3498042fd webapp: Now automatically fills in any creds used by an existing remote when creating a new remote of the same type. Done for Internet Archive, S3, Glacier, and Box.com remotes. 2013-04-27 15:16:06 -04:00
Joey Hess
8a7e059f54 fix build w/o s3 2013-04-26 16:12:23 -04:00
Joey Hess
e3ea36174b webapp: Display some additional information about a repository on its edit page. 2013-04-25 16:42:17 -04:00
Joey Hess
8284b310a7 support enabling IA repositories 2013-04-25 13:14:49 -04:00
Joey Hess
2810807ca5 Internet Archive!
* Add public repository group.
* webapp: Can now set up Internet Archive repositories.

TODO: Enabling IA repositories.
2013-04-25 12:23:36 -04:00
Joey Hess
1f1cddbaa7 syncNewRemote is not only used for new remotes; rename to syncRemote 2013-04-08 15:43:43 -04:00
Joey Hess
9a5f421768 detect when unwanted remote is empty and remove it
Needs fixes to build when the webapp is disabled.
2013-04-03 17:01:40 -04:00
Joey Hess
140774a8c8 webapp: Switch all forms to POST. 2013-03-16 18:48:26 -04:00
Joey Hess
c94c99942b make liftAnnex and liftAssistant polymorphic, like liftIO 2013-03-16 00:19:56 -04:00
Joey Hess
907b0c0d78 better liftAnnex, avoid using runAnnex undefined 2013-03-04 16:36:38 -04:00
Joey Hess
0d2ac184b5 webapp: Avoid illegal characters in hostname when creating S3 or Glacier remote. 2013-01-14 12:54:29 -04:00
Joey Hess
7f7c31df1c type based git config handling
Now there's a Config type, that's extracted from the git config at startup.
Note that laziness means that individual config values are only looked up
and parsed on demand, and so we get implicit memoization for all of them.
So this is not only prettier and more type safe, it optimises several
places that didn't have explicit memoization before. As well as getting rid
of the ugly explicit memoization code.

Not yet done for annex.<remote>.* configuration settings.
2012-12-29 23:10:18 -04:00
Joey Hess
c9fd389fbf webapp: Encryption can be disabled when setting up remotes. 2012-12-04 13:28:22 -04:00