Commit graph

46888 commits

Author SHA1 Message Date
Joey Hess
020b47c98a
stack.yaml: Update to lts-24.2. 2025-08-04 14:28:47 -04:00
Joey Hess
786fa037c8
comment 2025-08-01 13:57:50 -04:00
Joey Hess
655750cffb
reword 2025-08-01 13:55:38 -04:00
Joey Hess
3b2f09d6a2
link to new page 2025-08-01 13:52:29 -04:00
Joey Hess
a690c02361
add section on security 2025-08-01 13:51:25 -04:00
Joey Hess
398664d160
inline didn't work due to extension 2025-08-01 13:46:53 -04:00
Joey Hess
8d78682387
layout 2025-08-01 13:45:39 -04:00
Joey Hess
3b6b3416d9
add example git-annex-p2p-unix-sockets program and end-user docs 2025-08-01 13:44:00 -04:00
Joey Hess
c15dad6040
improve AuthToken display in P2P --debug
Using an empty string to obscure the AuthToken made it impossible to
tell if one was really being presented or not.
2025-08-01 12:58:03 -04:00
Joey Hess
05e945fdf9
Merge branch 'master' into genericp2p 2025-08-01 12:10:33 -04:00
Joey Hess
4958d74163
Merge branch 'master' of ssh://git-annex.branchable.com 2025-07-31 15:21:22 -04:00
Joey Hess
59bcfc838e
update design doc with changes from genericp2p branch
That branch is basically ready to merge, but needs more testing in a
chicken and egg situation.
2025-07-31 15:20:28 -04:00
yarikoptic
6578d323a1 Added a comment 2025-07-31 19:07:57 +00:00
Joey Hess
7e0802e951
avoid broken pipe zombies
hClose crashes if the pipe is broken, preventing waiting for the process
2025-07-31 14:57:51 -04:00
Joey Hess
f1781d01d8
remotedaemon support for generic P2P transports
RemoteDaemon.Transport.Tor was refactored into this, and most of the
code is reused between them.

getSocketFile does not yet deal with repositories on crippled
filesystems that don't support sockets. Annex.Ssh detects that and
allows the user to set an environment variable, and something similar
could be done here.

And it does not deal with a situation where there is no path to the
socket file that is not too long. In that situation it would crash out
I suppose. Probably though, remotedaemon is ran from the top of the
repo, and in that case the path is just ".git/annex/p2p/<md5>" so nice
and short.

This seems to mostly work. But I don't yet have a working git-annex-p2p-
command to test it with.

And with my not quite working git-annex-p2p-foo test script, running
remotedaemon results in an ever-growing number of zombie processes
that it's not waiting on.
2025-07-31 14:45:32 -04:00
Joey Hess
7643c716cd
changed design for p2p generic socket
Having the git-annex-p2p-<netname> command output the socket filename
left git-annex scrambling to listen to it in order to not miss incoming
connections. And if the command uses something like socat UNIX-CONNECT,
that expects the socket to be accepting connections and errors out when
it's not, that would be a problem.

Rather than complicating the protocol with git-annex needing to send
back a message when it's listening to the socket, simplified it by
having git-annex provide the socket path to the command.

This does mean that, if a P2P network has its own place it expects to
find a socket file, the git-annex-p2p-<netname> command would need to
somehow arrange for it to use the git-annex socket path. A symlink would
be one way to handle that situation.
2025-07-31 13:18:30 -04:00
Yann Büchau
4c652de8ee
Add yann's TdF talk about life in git annex 2025-07-31 18:25:31 +02:00
Joey Hess
86e9e88530
add git-remote-p2p-annex
Added git-remote-p2p-annex, which allows git pull and push to P2P networks
provided by external commands.

This is a refactor of git-remote-tor-annex, and should just work. Except
possibly for quirks with the address parsing. I've checked that the address
parsing basically works.

One thing I don't understand is why git-remote-tor-annex removes "/*" from
the end of the address. The git history does not provide any hints. So I
didn't make git-remote-p2p-annex do the same. Maybe that is needed in some
situation? But, a P2P address could contain "/", so removing it would be a
problem. I can't see anything in gitremote-helpers(7) about why the url
might get such a thing added to the end of it. My guess is that is not
needed for tor either (but does no harm there since onion addresses never
contain "/").

At this point, the implementation of generic P2P transports needs only
remotedaemon support.
2025-07-30 15:25:56 -04:00
Joey Hess
7403aeb95f
use Annex.ExternalAddonProcess for P2P.Generic processes
These are another sort of external addon process, and this makes several
things work including shell scripts on windows. And it makes for nicer
error messages when the command is not in the path.

Note that the refactored startExternalAddonProcess used by this
does not use propGitEnv to set git environment variables in the
environment. Unlike startExternalAddonProcessProtocol which does.
This is because it runs in IO and does not have access to that
information. But also, I don't think that P2P.Generic processes need
that.
2025-07-30 14:46:37 -04:00
Joey Hess
d3fbda13e4
p2p --enable
p2p: Added --enable option, which can be used to enable P2P networks
provided by external commands git-annex-p2p-<netname>

Made git-annex p2p --enable tor behave the same as git-annex enable-tor,
to make tor a bit less of a special case. However, it canot be run as root,
since it cannot take the user id parameter.
2025-07-30 14:08:59 -04:00
Joey Hess
4fb9b7cb67
support P2PAnnex in connectPeer
This is probably enough to support accessing remotes using p2p-annex:: urls.
Not tested yet of course since there is not yet support for serving the
other side of such a connection, or for setting up such a connection.

P2P.Generic has an implementation of the whole interface to the
git-annex-p2p-<netname> commands.
2025-07-30 13:23:23 -04:00
Joey Hess
a6f8248465
add connProcess to P2PConnection
When using the new generic P2P transport to open an outgoing connection
to a peer, this will hold the pid of the git-annex-p2p-<netname>
command.

closeConnection simply waits for it. Rather than relying on garbage
collection of the closed handles to close it.

In Remote.Helper.Ssh, connProcess is set to Nothing, even though there
is a similar process being used there. That code stores the pid in
OpenConnection instead, and handles waiting for it itself. A bit ugly,
but not worth cleaning up at this point, maybe later.
2025-07-30 12:35:16 -04:00
nobodyinperson
b57842addd Added a comment: Use an older version e.g. from archive.org 2025-07-30 16:17:44 +00:00
Joey Hess
f631bc9e56
add P2PAnnex constructor
This is for p2p-annex:: urls that will use the new generic P2P
transport.

In addressCredsFile, threw in an url encoding of any non-alphanumeric
characters that are in the address. This is to avoid any possible path
traversal attacks via a p2p-annex:: url, since the address part of it
could contain any characters. And, went ahead and did the same url
encoding of tor-annex:: urls, even though tor onion addresses are all
alphanumerics, on the off chance that might avoid a similar problem.
(It does not seem likely enough to treat it as a security hole.)
2025-07-30 12:09:17 -04:00
h0b0
3d7951abce Added a comment: Cataline build missing 2025-07-30 15:19:06 +00:00
Joey Hess
2a81b26e8e
document output as a single line 2025-07-29 14:26:10 -04:00
Joey Hess
d70a8de5c5
rename design page 2025-07-29 14:24:05 -04:00
Joey Hess
c4a0ecaad1
documentation for generic P2P transports 2025-07-29 14:22:25 -04:00
Joey Hess
05c016084d
design for p2p socket transport 2025-07-29 14:00:21 -04:00
apoelstra
b691575e14 Added a comment: fsck can do this 2025-07-24 14:44:52 +00:00
Joey Hess
b415d5d1f3
add news item for git-annex 10.20250721 2025-07-22 14:11:53 -04:00
Joey Hess
5fb0a7267a
releasing package git-annex version 10.20250721 2025-07-22 14:11:47 -04:00
mih
0de42855c5 Added a comment: Works! 2025-07-22 13:31:21 +00:00
Joey Hess
8867e7590a
prep for release of 10.20250721 2025-07-21 14:24:22 -04:00
Joey Hess
8767771901
Merge branch 'master' of ssh://git-annex.branchable.com 2025-07-21 14:16:35 -04:00
nobodyinperson
5a5332faab Added a comment: Workaround for default wanted content? 2025-07-21 18:12:59 +00:00
Joey Hess
cf449837ea
run reconcileStaged even in smudge clean filter, using alternate code path
Improved workaround for git 2.50 bug, avoding an occasional test suite
failure, as well as some situations where an unlocked file did not get
populated when adding another file to the repository with the same content.

This uses the alternate code path that was already using when there was
a conflict. Since that code path is not able to record its work,
it will redo the same work next time. If the only way reconcileStaged
is getting run is via the smudge clean filter, that could result in
more and more changes getting processed redundantly each time. Once
some other git-annex command runs and calls reconcileStaged, it
will stop redoing that work. I don't think the extra work will be a
problem.
2025-07-21 14:10:49 -04:00
Joey Hess
65a1cf54ce
make --keep-failures show full path to test repo
Otherwise, had to guess which of several subdirectories was the right
one.
2025-07-21 12:51:57 -04:00
Joey Hess
f4fdab0ffa
fixed build failure 2025-07-21 12:27:39 -04:00
Joey Hess
ba24f78626
fix build with OsPath build flag 2025-07-21 12:26:45 -04:00
Joey Hess
de6c5b6885
fix build warning 2025-07-21 12:25:17 -04:00
Joey Hess
d9bd90e598
response 2025-07-21 12:20:48 -04:00
Joey Hess
d364e434c8
Add --url option and url= preferred content expression
To match content that is recorded as present in an url.

Note that, this cannot ask remotes to provide an url using whereisKey, like
whereis does. Because preferred content expressions need to match the same
from multiple perspectives, and the remote would not always be available.

That's why the docs say "recorded as present", but still this may be
surprising to some who see an url in whereis output and are surprised they
cannot match on it.

The use of getDownloader is to strip the downloader prefix from urls like
"yt:". Note that, when OtherDownloader is used, this strips the ":" prefix,
and allows matching on those urls too.
2025-07-21 12:13:40 -04:00
Joey Hess
549569533b
Merge branch 'master' of ssh://git-annex.branchable.com 2025-07-21 10:53:08 -04:00
Joey Hess
ac25e5a6e7
todo 2025-07-19 16:45:51 -04:00
jnkl
6b4eabb161 Added a comment 2025-07-19 09:05:46 +00:00
Joey Hess
70da89d5b8
todo 2025-07-18 09:49:21 -04:00
jose1711
24b7f40cac Added a comment: fs type? 2025-07-18 09:10:50 +00:00
jnkl
cd3f5af7ed 2025-07-17 08:27:22 +00:00
mih
c58aa2cf47 Added a comment: Thanks! 2025-07-17 06:58:01 +00:00