mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/git-annex-p2phttp.mdwn

173 lines
5 KiB
Text
Raw Permalink Normal View History

started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
# NAME
annex+http urls
2024-07-23 12:42:33 +00:00
git-annex-p2phttp - HTTP server for the git-annex API
started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
# SYNOPSIS
started on auth
2024-07-09 21:30:55 +00:00
git-annex p2phttp
started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
# DESCRIPTION
annex+http urls
2024-07-23 12:42:33 +00:00
This is a HTTP server for the git-annex API.
started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
It is the git-annex equivilant of git-http-backend(1), for serving
a repository over HTTP with write access for authenticated users.
annex+http urls
2024-07-23 12:42:33 +00:00
This does not serve the git repository over HTTP, only the git-annex
API.
Typically a remote will have `remote.name.url` set to a http url
as usual, and `remote.name.annexUrl` set to an annex+http url such as
remove uuid from annex+http urls Not needed it turns out.
2024-07-29 00:29:42 +00:00
"annex+http://example.com/git-annex/". The annex+http url is
require a valid uuid at the end of an annex+http url
2024-07-23 16:30:27 +00:00
served by this server, and uses port 9417 by default.
annex+http urls
2024-07-23 12:42:33 +00:00
only base64 non-utf8
2024-07-11 19:47:16 +00:00
As well as serving the git-annex HTTP API, this server provides a
convenient way to download the content of any key, by using the path
"/git-annex/$uuid/$key". For example:
annex+http urls
2024-07-23 12:42:33 +00:00
$ curl http://example.com:9417/git-annex/f11773f0-11e1-45b2-9805-06db16768efe/key/SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03
only base64 non-utf8
2024-07-11 19:47:16 +00:00
hello
started on auth
2024-07-09 21:30:55 +00:00
# OPTIONS
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
* `--jobs=N` `-JN`
add Annex worker pool to P2PHttp This will be needed for get and store, since those need to run Annex actions. withLocalP2PConnections will also probably use it.
2024-07-10 16:19:47 +00:00
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
This or annex.jobs must be set to configure the number of worker
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
threads that serve connections to the webserver.
add Annex worker pool to P2PHttp This will be needed for get and store, since those need to run Annex actions. withLocalP2PConnections will also probably use it.
2024-07-10 16:19:47 +00:00
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
Since the webserver itself also uses one of these threads,
this needs to be set to 2 or more.
add Annex worker pool to P2PHttp This will be needed for get and store, since those need to run Annex actions. withLocalP2PConnections will also probably use it.
2024-07-10 16:19:47 +00:00
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
A good choice is often one worker per CPU core: `--jobs=cpus`
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
implement proxy connection pool removeOldestProxyConnectionPool will be innefficient the larger the pool is. A better data structure could be more efficient. Eg, make each value in the pool include the timestamp of its oldest element, then the oldest value can be found and modified, rather than rebuilding the whole Map. But, for pools of a few hundred items, this should be fine. It's O(n*n log n) or so. Also, when more than 1 connection with the same pool key exists, it's efficient even for larger pools, since removeOldestProxyConnectionPool is not needed. The default of 1 idle connection could perhaps be larger.. like the number of jobs? Otoh, it seems good to ramp up and down the number of connections, which does happen. With 1, there is at most one stale connection, which might cause a request to fail.
2024-07-26 20:34:08 +00:00
* `--proxyconnections=N`
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
When this command is run in a repository that is configured to act as a
implement proxy connection pool removeOldestProxyConnectionPool will be innefficient the larger the pool is. A better data structure could be more efficient. Eg, make each value in the pool include the timestamp of its oldest element, then the oldest value can be found and modified, rather than rebuilding the whole Map. But, for pools of a few hundred items, this should be fine. It's O(n*n log n) or so. Also, when more than 1 connection with the same pool key exists, it's efficient even for larger pools, since removeOldestProxyConnectionPool is not needed. The default of 1 idle connection could perhaps be larger.. like the number of jobs? Otoh, it seems good to ramp up and down the number of connections, which does happen. With 1, there is at most one stale connection, which might cause a request to fail.
2024-07-26 20:34:08 +00:00
proxy for some of its remotes, this is the maximum number of idle
connections to keep open to proxied remotes.
The default is 1.
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
* `--clusterjobs=N`
When this command is run in a repository that is a gateway for a cluster,
this is the number of concurrent jobs to use to access nodes of the
cluster, per connection to the webserver.
The default is 1.
A good choice for this will be a balance between the number of nodes
in the cluster and the value of `--jobs`.
For example, if the cluster has 4 nodes, and `--jobs=4`, using
`--clusterjobs=4` will make all nodes in the cluster be accessed
concurrently, which is often optimal. But around 20 cores can be needed
when the webserver is busy.
git-annex p2phttp options
2024-07-10 03:44:40 +00:00
* `--port=N`
annex+http urls
2024-07-23 12:42:33 +00:00
Port to listen on. The default is port 9417, which is the default
port used for an annex+http or annex+https url.
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
It is not recommended to run this command as root in order to
update
2024-07-23 00:59:45 +00:00
use a low port like port 80. It will not drop permissions when run as
root.
git-annex p2phttp options
2024-07-10 03:44:40 +00:00
add --bind option and listen to both ipv4 and ipv6 by default
2024-07-23 19:19:56 +00:00
* `--bind=address`
What address to bind to. The default is to bind to all addresses.
p2phttp support https
2024-07-23 19:37:36 +00:00
* `--certfile=filename`
TLS certificate file to use. Combining this with `--privatekeyfile`
makes the server use HTTPS.
* `--privatekeyfile=filename`
TLS private key file to use. Combining this with `--certfile`
makes the server use HTTPS.
* `--chainfile=filename`
TLS chain file to use. This option can be repeated any number of times.
started on auth
2024-07-09 21:30:55 +00:00
* `--authenv`
Allows users to be authenticated with a username and password.
For security, this only allows authentication when the user connects over
HTTPS.
To configure the passwords, set environment variables
like `GIT_ANNEX_P2PHTTP_PASSWORD_alice=foo123`
The permissions of users can also be configured by setting
environment variables like
`GIT_ANNEX_P2PHTTP_PERMISSIONS_alice=readonly`. The value
can be either "readonly" or "appendonly". When this is not set,
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
the default is to give the user full read+write+remove access.
started on auth
2024-07-09 21:30:55 +00:00
* `--authenv-http`
Like `--authenv`, but allows authentication when the user connects
over HTTP. This is not secure, since HTTP basic authentication is not
encrypted.
git-annex p2phttp options
2024-07-10 03:44:40 +00:00
* `--unauth-readonly`
started on auth
2024-07-09 21:30:55 +00:00
Allows unauthenticated users to read the repository, but not make
modifications to it.
p2phttp combining unauth and auth options p2phttp: Support serving unauthenticated users while requesting authentication for operations that need it. Eg, --unauth-readonly can be combined with --authenv. Drop locking currently needs authentication so it will prompt for that. That still needs to be addressed somehow.
2024-10-17 15:10:28 +00:00
This can be combined with `--authenv` or `--authenv-http` to allow
anonymous readonly access, and authenticated write access.
git-annex p2phttp options
2024-07-10 03:44:40 +00:00
* `--unauth-appendonly`
started on auth
2024-07-09 21:30:55 +00:00
Allows unauthenticated users to read the repository, and store data in
it, but not remove data from it.
p2phttp combining unauth and auth options p2phttp: Support serving unauthenticated users while requesting authentication for operations that need it. Eg, --unauth-readonly can be combined with --authenv. Drop locking currently needs authentication so it will prompt for that. That still needs to be addressed somehow.
2024-10-17 15:10:28 +00:00
This can be combined with `--authenv` or `--authenv-http` to allow
anonymous appendonly access, and authenticated remove access.
started on auth
2024-07-09 21:30:55 +00:00
p2phttp: Allow unauthenticated users to lock content by default * p2phttp: Allow unauthenticated users to lock content by default. * p2phttp: Added --unauth-nolocking option to prevent unauthenticated users from locking content. The rationalle for this is that locking is not really a write operation, so makes sense to allow in a repository that only allows read-only access. Not supporting locking in that situation will prevent the user from dropping content from a special remote they control in cases where the other copy of the content is on the p2phttp server. Also, when p2phttp is configured to also allow authenticated access, lockcontent was resulting in a password prompt for users who had no way to authenticate. And there is no good way to distinguish between the two types of users client side. --unauth-nolocking anticipates that this might be abused, and seems better than disabling unauthenticated access entirely if a server is being attacked. It may be that rate limiting locking by IP address or similar would be an effective measure in such a situation. Or just limiting the number of locks by anonymous users that can be live at any one time. Since the impact of such an DOS attempt is limited to preventing dropping content from the server, it seems not a very appealing target anyway.
2024-10-21 14:02:12 +00:00
* `--unauth-nolocking`
By default, when `--unauth-readonly` or `--unauth-appendonly` is used,
unauthenticated users are allowed to lock content in the repository.
This option prevents that.
Locking content prevents it from being dropped from the repository
so it may be that an unauthenticated user abuses that, and this option
can be used in such a situation.
Note that enabling this option will prevent unauthenticated users from
dropping content from their other remotes in some cases.
started on auth
2024-07-09 21:30:55 +00:00
* `--wideopen`
change default port to 9417 Port 80 would need root, not a good idea, so pick something that might work by default. 9418 is git protocol's port. 9419 is used by something, but nothing known uses 9417, so it's as good a default as any.
2024-07-23 00:44:48 +00:00
Gives unauthenticated users full read+write+remove access to the
repository.
started on auth
2024-07-09 21:30:55 +00:00
Please think carefully before enabling this option.
started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
# SEE ALSO
[[git-annex]](1)
git-http-backend(1)
[[git-annex-shell]](1)
implement proxy connection pool removeOldestProxyConnectionPool will be innefficient the larger the pool is. A better data structure could be more efficient. Eg, make each value in the pool include the timestamp of its oldest element, then the oldest value can be found and modified, rather than rebuilding the whole Map. But, for pools of a few hundred items, this should be fine. It's O(n*n log n) or so. Also, when more than 1 connection with the same pool key exists, it's efficient even for larger pools, since removeOldestProxyConnectionPool is not needed. The default of 1 idle connection could perhaps be larger.. like the number of jobs? Otoh, it seems good to ramp up and down the number of connections, which does happen. With 1, there is at most one stale connection, which might cause a request to fail.
2024-07-26 20:34:08 +00:00
[[git-annex-updateproxy]](1)
add --clusterjobs option and default to 1 The default of 1 is not ideal at all, but it avoids an accidental M*N causing so much concurrency it becomes unusable.
2024-07-28 14:36:22 +00:00
[[git-annex-initcluster]](1)
[[git-annex-updatecluster]](1)
started servant implementation of HTTP P2P protocol
2024-07-07 16:08:10 +00:00
<https://git-annex.branchable.com/design/p2p_protocol_over_http/>
# AUTHOR
Joey Hess <id@joeyh.name>
<http://git-annex.branchable.com/>
Warning: Automatically converted into a man page by mdwn2man. Edit with care
Reference in a new issue Copy permalink