That's the version in Debian stable now. And this removes a lot of ifdefs.
Also I'm pretty sure a recent commit broke building with older versions of
aws, although that could be fixed with sufficent testing.
S3: When initremote is given the name of a bucket that already exists,
automatically set datacenter to the right value, rather than needing it to
be explicitly set.
This needs aws-0.23. But, initremote stores the datacenter value, so
a remote set up this way can be used with git-annex built with an older aws.
This is not done when signature=anonymous, because in that case,
using AWS.defaultRegion works fine for accessing buckets on other
datacenters.
It feels a bit round-about to need to do this probing. But without it,
the problem seems to be that, with a v4 signature, the location constraint
is included in the Authorization header. When that is the wrong location,
AWS S3 rejects it. I do wonder though if there is an easier way that I
am currently missing.
Sponsored-by: Dartmouth College's DANDI project
Commit 215640096f caused the default
region for S3 to change to us-east-2. This was due to regionInfo having
an undocumented property that the first item in the list is for the
default region.
Avoid relying on regionInfo for defaultRegion.
Sponsored-by: Dartmouth College's DANDI project
* S3: Default to signature=v4 when using an AWS endpoint, since some
AWS regions need v4 and all support it. When host= is used to specify
a different S3 host, the default remains signature=v2.
* webapp: Support setting up S3 buckets in regions that need v4
signatures.
For the webapp, went ahead and added all current S3 regions
(except govcloud, which is not usable by everyone).
Sponsored-by: Dartmouth College's DANDI project
RemoteDaemon.Transport.Tor was refactored into this, and most of the
code is reused between them.
getSocketFile does not yet deal with repositories on crippled
filesystems that don't support sockets. Annex.Ssh detects that and
allows the user to set an environment variable, and something similar
could be done here.
And it does not deal with a situation where there is no path to the
socket file that is not too long. In that situation it would crash out
I suppose. Probably though, remotedaemon is ran from the top of the
repo, and in that case the path is just ".git/annex/p2p/<md5>" so nice
and short.
This seems to mostly work. But I don't yet have a working git-annex-p2p-
command to test it with.
And with my not quite working git-annex-p2p-foo test script, running
remotedaemon results in an ever-growing number of zombie processes
that it's not waiting on.
Having the git-annex-p2p-<netname> command output the socket filename
left git-annex scrambling to listen to it in order to not miss incoming
connections. And if the command uses something like socat UNIX-CONNECT,
that expects the socket to be accepting connections and errors out when
it's not, that would be a problem.
Rather than complicating the protocol with git-annex needing to send
back a message when it's listening to the socket, simplified it by
having git-annex provide the socket path to the command.
This does mean that, if a P2P network has its own place it expects to
find a socket file, the git-annex-p2p-<netname> command would need to
somehow arrange for it to use the git-annex socket path. A symlink would
be one way to handle that situation.
Added git-remote-p2p-annex, which allows git pull and push to P2P networks
provided by external commands.
This is a refactor of git-remote-tor-annex, and should just work. Except
possibly for quirks with the address parsing. I've checked that the address
parsing basically works.
One thing I don't understand is why git-remote-tor-annex removes "/*" from
the end of the address. The git history does not provide any hints. So I
didn't make git-remote-p2p-annex do the same. Maybe that is needed in some
situation? But, a P2P address could contain "/", so removing it would be a
problem. I can't see anything in gitremote-helpers(7) about why the url
might get such a thing added to the end of it. My guess is that is not
needed for tor either (but does no harm there since onion addresses never
contain "/").
At this point, the implementation of generic P2P transports needs only
remotedaemon support.
These are another sort of external addon process, and this makes several
things work including shell scripts on windows. And it makes for nicer
error messages when the command is not in the path.
Note that the refactored startExternalAddonProcess used by this
does not use propGitEnv to set git environment variables in the
environment. Unlike startExternalAddonProcessProtocol which does.
This is because it runs in IO and does not have access to that
information. But also, I don't think that P2P.Generic processes need
that.
p2p: Added --enable option, which can be used to enable P2P networks
provided by external commands git-annex-p2p-<netname>
Made git-annex p2p --enable tor behave the same as git-annex enable-tor,
to make tor a bit less of a special case. However, it canot be run as root,
since it cannot take the user id parameter.
This is probably enough to support accessing remotes using p2p-annex:: urls.
Not tested yet of course since there is not yet support for serving the
other side of such a connection, or for setting up such a connection.
P2P.Generic has an implementation of the whole interface to the
git-annex-p2p-<netname> commands.
When using the new generic P2P transport to open an outgoing connection
to a peer, this will hold the pid of the git-annex-p2p-<netname>
command.
closeConnection simply waits for it. Rather than relying on garbage
collection of the closed handles to close it.
In Remote.Helper.Ssh, connProcess is set to Nothing, even though there
is a similar process being used there. That code stores the pid in
OpenConnection instead, and handles waiting for it itself. A bit ugly,
but not worth cleaning up at this point, maybe later.
This is for p2p-annex:: urls that will use the new generic P2P
transport.
In addressCredsFile, threw in an url encoding of any non-alphanumeric
characters that are in the address. This is to avoid any possible path
traversal attacks via a p2p-annex:: url, since the address part of it
could contain any characters. And, went ahead and did the same url
encoding of tor-annex:: urls, even though tor onion addresses are all
alphanumerics, on the off chance that might avoid a similar problem.
(It does not seem likely enough to treat it as a security hole.)
