Merge branch 'master' into exportreeplus

CHANGELOG

@@ -10,6 +10,8 @@ git-annex (10.20240732) UNRELEASED; urgency=medium
     in the same paths used by annexobjects=yes.
   * When proxying an upload to a special remote, verify the hash.
   * Avoid loading cluster log at startup.
+  * When getting from a P2P HTTP remote, prompt for credentials when
+    required, instead of failing.
 
  -- Joey Hess <id@joeyh.name>  Wed, 31 Jul 2024 15:52:03 -0400
 

P2P/Http/Client.hs

@@ -9,6 +9,7 @@
 
 {-# LANGUAGE BangPatterns #-}
 {-# LANGUAGE DataKinds, TypeApplications #-}
+{-# LANGUAGE ScopedTypeVariables #-}
 {-# LANGUAGE CPP #-}
 
 module P2P.Http.Client (
@@ -99,7 +100,7 @@ p2pHttpClientVersions allowedversion rmt fallback clientaction =
 	versions = filter allowedversion allProtocolVersions
 	go clientenv mcred credcached mauth (v:vs) = do
 		myuuid <- getUUID
-		res <- clientaction clientenv v
+		res <- catchclienterror $ clientaction clientenv v
 			(B64UUID (uuid rmt))
 			(B64UUID myuuid)
 			[]
@@ -126,6 +127,8 @@ p2pHttpClientVersions allowedversion rmt fallback clientaction =
 					("git-annex HTTP API server returned an unexpected response: " ++ show clienterror)
 	go _ _ _ _ [] = return Nothing
 
+	catchclienterror a = a `catch` \(ex :: ClientError) -> pure (Left ex)
+
 	authrequired clientenv vs = do
 		cred <- prompt $ 
 			inRepo $ Git.getUrlCredential credentialbaseurl

debian/patches/standalone-build

@@ -6,7 +6,7 @@ Last-Update: 2015-04-20
 
 --- a/debian/control
 +++ b/debian/control
-@@ -86,6 +86,7 @@ Build-Depends:
+@@ -95,6 +95,7 @@ Build-Depends:
  	git (>= 1:2.22),
  	rsync,
  	curl,
@@ -14,7 +14,7 @@ Last-Update: 2015-04-20
  	openssh-client,
  	git-remote-gcrypt (>= 0.20130908-6),
  	gnupg,
-@@ -96,15 +97,15 @@ Vcs-Git: git://git.joeyh.name/git-annex
+@@ -105,15 +106,15 @@ Vcs-Git: git://git.joeyh.name/git-annex
  Homepage: http://git-annex.branchable.com/
  XS-Testsuite: autopkgtest
  
@@ -36,16 +36,16 @@ Last-Update: 2015-04-20
  Recommends: 
  	lsof,
  	gnupg,
-@@ -123,7 +124,7 @@ Suggests:
- 	libnss-mdns,
+@@ -133,7 +134,7 @@ Suggests:
  	uftp,
+ 	yt-dlp,
  Breaks: datalad (< 0.12.3~)
 -Description: manage files with git, without checking their contents into git
 +Description: manage files with git, without checking their contents into git -- standalone build
   git-annex allows managing large files with git, without storing the file 
   contents in git. It can sync, backup, and archive your data, offline
   and online. Checksums and encryption keep your data safe and secure. Bring
-@@ -142,3 +143,7 @@ Description: manage files with git, with
+@@ -152,3 +153,7 @@ Description: manage files with git, with
   noticing when files are changed, and automatically committing them
   to git and transferring them to other computers. The git-annex webapp
   makes it easy to set up and use git-annex this way.

doc/bugs/get_from_p2phttp_does_not_ask_for__47__use_credentials.mdwn

@@ -0,0 +1,60 @@
+### Please describe the problem.
+
+Attempting to `git annex get` from a p2phttp remote does not work with authentication. Instead of being asked to supply credentials like with `git annex copy` I instead get this:
+
+```
+$ git annex get file1.bin 
+get file1.bin (from origin...) 
+
+  FailureResponse (Request {requestPath = (BaseUrl {baseUrlScheme = Http, baseUrlHost = "localhost", baseUrlPort = 3001, baseUrlPath = ""},"/git-annex/d430acae-547f-4e90-8ed7-0e0c57071ae0/v3/key/MD5E-s321388544--0cfb0327388a04a212b45f7099191e1c.bin"), requestQueryString = fromList [("clientuuid",Just "a2aea5f6-9cf1-441a-9a40-061b62a1b078"),("associatedfile",Just "file1.bin")], requestBody = Nothing, requestAccept = fromList [application/octet-stream], requestHeaders = fromList [], requestHttpVersion = HTTP/1.1, requestMethod = "GET"}) (Response {responseStatusCode = Status {statusCode = 401, statusMessage = "Unauthorized"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Wed, 07 Aug 2024 11:59:48 GMT"),("Server","Warp/3.3.25"),("WWW-Authenticate","Basic realm=\"git-annex\", charset=\"UTF-8\"")], responseHttpVersion = HTTP/1.1, responseBody = ""})
+
+  Unable to access these remotes: origin
+
+  Maybe add some of these git remotes (git remote add ...):
+  	a84bd368-4b2f-4bea-88f6-9d01f6cf32a7 -- icg149@icg1911:~/Playground/test-repo4
+failed
+get: 1 failed
+```
+
+Even if I first `git annex copy` to the remote and set the credentials, `git annex get` doesn't seem to use them. `git annex get` seems to just abort on 401, instead of responding to the WWW-Authenticate header and retrying with credentials.
+
+
+### What steps will reproduce the problem?
+
+- set up a repository served with `git annex p2phttp`
+- clone it
+- try to get a file
+
+
+### What version of git-annex are you using? On what operating system?
+
+```
+git-annex version: 10.20240731-g76f31d59b034ef647d0c94bc42aa75f411992230
+build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Servant Benchmark Feeds Testsuite S3 WebDAV
+dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.33 DAV-1.3.4 feed-1.3.2.1 ghc-9.4.7 http-client-0.7.14 persistent-sqlite-2.13.2.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external
+operating system: linux x86_64
+supported repository versions: 8 9 10
+upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
+local repository version: 10
+```
+
+On Ubuntu 22.04
+
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+I am currently experimenting with supporting the p2phttp protocol in forgejo-aneksajo. So far I have a prototype working with public repositories and it was surprisingly straightforward, but private repositories do not work yet due to this bug.
+
+> [[fixed|done]] --[[Joey]]

doc/bugs/get_from_p2phttp_does_not_ask_for__47__use_credentials/comment_1_34c509a615a48f76b7fa4b2873fbf95e._comment

@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-08-07T14:35:14Z"
+ content="""
+Reproduced this, and also `git-annex copy --from` fails the same way. While
+`git-annex copy --to` does prompt for credentials.
+"""]]

doc/bugs/migrate_does_not_migrate_from_SHA256E_to_SHA256.mdwn

@@ -0,0 +1,22 @@
+### Please describe the problem.
+In my repository I have a mixture of SHA256 and SHA256E backed files due to old files and manually configuring a subset of the remotes to SHA256. After configuring SHA256 as the (only) backend, when I run git annex migrate (with or without giving a specific path, or specifying the target SHA256 backend), some files pointing to SHA256E files still point to SHA256E instead of, as expected, SHA256.
+
+Note: The migration actually migrated MOST of the files, but some remain. I don't see anything special about these files.
+
+Note: The issue is NOT that, after migration, the SHA256E files remain as "duplicates". This is annoying, but I'm aware of this and can mitigate the issue via dropunused.
+
+
+### What steps will reproduce the problem?
+Have SHA256E. Configure SHA256 as backend. Run migrate.
+
+
+### What version of git-annex are you using? On what operating system?
+10.20230126 (Debian Stable)
+
+
+### Please provide any additional information below.
+n/a
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+I love it! :)
+