Commit graph

45714 commits

Author SHA1 Message Date
Joey Hess
b83fdf66df
Allow enabling the servant build flag with older versions of stm
Allowing building with ghc 9.0.2 (debian stable).

Updated patch covering all uses of writeTMVar.
2024-10-17 20:55:31 -04:00
yarikoptic
facde8bf85 removed 2024-10-17 22:08:32 +00:00
yarikoptic
e584ad6f24 Added a comment 2024-10-17 21:44:50 +00:00
yarikoptic
96f8297ab1 Added a comment 2024-10-17 21:44:17 +00:00
yarikoptic
2b42622727 Added a comment 2024-10-17 21:44:04 +00:00
mih
b642800d8c Tag with project ID 2024-10-17 20:26:08 +00:00
yarikoptic
f0a86bc223 Added a comment 2024-10-17 18:37:57 +00:00
Joey Hess
3c75e333c5
comment 2024-10-17 14:05:20 -04:00
Joey Hess
3a53c60121
Allow enabling the servant build flag with older versions of stm
Allowing building with ghc 9.0.2 (debian stable).
2024-10-17 14:04:31 -04:00
yarikoptic
c738920d19 Added a comment 2024-10-17 17:04:07 +00:00
Joey Hess
0629219617
p2phttp combining unauth and auth options
p2phttp: Support serving unauthenticated users while requesting
authentication for operations that need it. Eg, --unauth-readonly can be
combined with --authenv.

Drop locking currently needs authentication so it will prompt for that.
That still needs to be addressed somehow.
2024-10-17 11:10:28 -04:00
Joey Hess
4e59828f4c
response 2024-10-17 10:10:26 -04:00
Joey Hess
b387ffef33
Merge branch 'master' of ssh://git-annex.branchable.com 2024-10-17 10:06:25 -04:00
Joey Hess
5adc54eaba
add IMN7 to projects and tag a relevant issue 2024-10-17 10:06:08 -04:00
Joey Hess
6c65de35f0
comment 2024-10-17 10:01:31 -04:00
yarikoptic
1fa10f9427 question about Servant 2024-10-17 13:57:43 +00:00
matrss
8f96f7b16b 2024-10-17 09:39:56 +00:00
matrss
20e7201e5d Make VURL a proper list item 2024-10-17 09:18:04 +00:00
yarikoptic
48fb37388f initial report on divergence in behavior on ACL 2024-10-16 19:13:08 +00:00
yarikoptic
460cdb5623 Added a comment 2024-10-16 18:58:04 +00:00
yarikoptic
bcc243f5b1 make freeze/thaw relative paths 2024-10-16 18:51:45 +00:00
yarikoptic
d672a4ccfe initial report on freeze/thaw exit code ignorance 2024-10-16 18:34:06 +00:00
matrss
287212c40e Added a comment 2024-10-16 12:09:02 +00:00
Joey Hess
c4dfeaef53
streaming uploads 2024-10-15 16:02:19 -04:00
yarikoptic
6eb32468bc initial report on not all files being saved 2024-10-15 18:15:16 +00:00
Joey Hess
9e8bbb3aac
remove change that was accidentially committed 2024-10-15 13:30:52 -04:00
Joey Hess
c6c794a27d
comment 2024-10-15 13:27:27 -04:00
Joey Hess
c1b0348307
Merge branch 'master' of ssh://git-annex.branchable.com 2024-10-15 12:27:01 -04:00
Joey Hess
930c078965
working in streamproxy branch 2024-10-15 12:26:53 -04:00
Joey Hess
76a1989a0e
implement openFileBeingWritten
This bypasses the usual haskell file locking used to prevent opening a
file for read that is being written to.

This is unfortunately a bit of a hack. But it seems fairly unlikely to
get broken by changes to ghc. I hope. Using fdToHandle' will also work.

This does not work on windows because it uses openFd from posix. It
would probably be possible to implement it for windows too, just opening
the FD using the Win32 library instead. However, whether windows will
allow reading from a file that is also being written to I don't know,
and since in the git-annex case the writer could be another process (eg
external special remote), that might be doing its own locking in
windows, that seems a can of worms I'd prefer not to open.
2024-10-15 11:56:42 -04:00
Joey Hess
57ac43e4f1
update 2024-10-15 10:31:42 -04:00
matrss
c7155366c7 Added a comment 2024-10-14 12:21:54 +00:00
matrss
872d97eb2a Added a comment 2024-10-14 12:01:40 +00:00
Joey Hess
9574e3a8bb
Merge branch 'master' of ssh://git-annex.branchable.com 2024-10-12 10:57:52 -04:00
Spencer
10b6539174 Added a comment: [FR] Remote Settings for All Clones 2024-10-09 23:10:17 +00:00
annex@9cc004f218c318a28099ff2645959be0fcbc6d94
4404cc4c8b Added a comment: Support for importtree 2024-10-09 06:42:40 +00:00
matrss
4e5dcf4207 2024-10-08 07:22:39 +00:00
Spencer
cbc88a878f 2024-10-08 03:58:55 +00:00
Spencer
0f7ba08e95 2024-10-08 03:31:05 +00:00
Spencer
d87d725b1c Correction: rclonelayout=lower is not synonymous with the directory remote, directory is. 2024-10-07 21:26:42 +00:00
Spencer
ae09255c05 Added a comment: How to Clone? 2024-10-07 20:00:24 +00:00
Joey Hess
8baa43ee12
tried a blind alley on streaming special remote download via proxy
This didn't work. In case I want to revisit, here's what I tried.

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 48222872c1..e4e526d3dd 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -26,16 +26,21 @@ import Logs.UUID
 import Logs.Location
 import Utility.Tmp.Dir
 import Utility.Metered
+import Utility.ThreadScheduler
+import Utility.OpenFd
 import Git.Types
 import qualified Database.Export as Export

 import Control.Concurrent.STM
 import Control.Concurrent.Async
+import Control.Concurrent.MVar
 import qualified Data.ByteString as B
+import qualified Data.ByteString as BS
 import qualified Data.ByteString.Lazy as L
 import qualified System.FilePath.ByteString as P
 import qualified Data.Map as M
 import qualified Data.Set as S
+import System.IO.Unsafe

 proxyRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide
 proxyRemoteSide clientmaxversion bypass r
@@ -240,21 +245,99 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go
 		writeVerifyChunk iv h b
 		storetofile iv h (n - fromIntegral (B.length b)) bs

-	proxyget offset af k = withproxytmpfile k $ \tmpfile -> do
+	proxyget offset af k = withproxytmpfile k $ \tmpfile ->
+		let retrieve = tryNonAsync $ Remote.retrieveKeyFile r k af
+			(fromRawFilePath tmpfile) nullMeterUpdate vc
+		in case fromKey keySize k of
+			Just size | size > 0 -> do
+				cancelv <- liftIO newEmptyMVar
+				donev <- liftIO newEmptyMVar
+				streamer <- liftIO $ async $
+					streamdata offset tmpfile size cancelv donev
+				retrieve >>= \case
+					Right _ -> liftIO $ do
+						putMVar donev ()
+						wait streamer
+					Left err -> liftIO $ do
+						putMVar cancelv ()
+						wait streamer
+						propagateerror err
+			_ -> retrieve >>= \case
+				Right _ -> liftIO $ senddata offset tmpfile
+				Left err -> liftIO $ propagateerror err
+	  where
 		-- Don't verify the content from the remote,
 		-- because the client will do its own verification.
-		let vc = Remote.NoVerify
-		tryNonAsync (Remote.retrieveKeyFile r k af (fromRawFilePath tmpfile) nullMeterUpdate vc) >>= \case
-			Right _ -> liftIO $ senddata offset tmpfile
-			Left err -> liftIO $ propagateerror err
+		vc = Remote.NoVerify

+	streamdata (Offset offset) f size cancelv donev = do
+		sendlen offset size
+		waitforfile
+		x <- tryNonAsync $ do
+			fd <- openFdWithMode f ReadOnly Nothing defaultFileFlags
+			h <- fdToHandle fd
+			hSeek h AbsoluteSeek offset
+			senddata' h (getcontents size)
+		case x of
+			Left err -> do
+				throwM err
+			Right res -> return res
+	  where
+		-- The file doesn't exist at the start.
+		-- Wait for some data to be written to it as well,
+		-- in case an empty file is first created and then
+		-- overwritten. When there is an offset, wait for
+		-- the file to get that large. Note that this is not used
+		-- when the size is 0.
+		waitforfile = tryNonAsync (fromIntegral <$> getFileSize f) >>= \case
+			Right sz | sz > 0 && sz >= offset -> return ()
+			_ -> ifM (isEmptyMVar cancelv)
+				( do
+					threadDelaySeconds (Seconds 1)
+					waitforfile
+				, do
+					return ()
+				)
+
+		getcontents n h = unsafeInterleaveIO $ do
+			isdone <- isEmptyMVar donev <||> isEmptyMVar cancelv
+			c <- BS.hGet h defaultChunkSize
+			let n' = n - fromIntegral (BS.length c)
+			let c' = L.fromChunks [BS.take (fromIntegral n) c]
+			if BS.null c
+				then if isdone
+					then return mempty
+					else do
+						-- Wait for more data to be
+						-- written to the file.
+						threadDelaySeconds (Seconds 1)
+						getcontents n h
+				else if n' > 0
+					then do
+						-- unsafeInterleaveIO causes
+						-- this to be deferred until
+						-- data is read from the lazy
+						-- ByteString.
+						cs <- getcontents n' h
+						return $ L.append c' cs
+					else return c'
+
 	senddata (Offset offset) f = do
 		size <- fromIntegral <$> getFileSize f
-		let n = max 0 (size - offset)
-		sendmessage $ DATA (Len n)
+		sendlen offset size
 		withBinaryFile (fromRawFilePath f) ReadMode $ \h -> do
 			hSeek h AbsoluteSeek offset
-			sendbs =<< L.hGetContents h
+			senddata' h L.hGetContents
+
+	senddata' h getcontents = do
+			sendbs =<< getcontents h
 			-- Important to keep the handle open until
 			-- the client responds. The bytestring
 			-- could still be lazily streaming out to
@@ -272,6 +355,11 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv oclosedv mexportdb = go
 				Just FAILURE -> return ()
 				Just _ -> giveup "protocol error"
 				Nothing -> return ()
+
+	sendlen offset size = do
+		let n = max 0 (size - offset)
+		sendmessage $ DATA (Len n)
+

 {- Check if this repository can proxy for a specified remote uuid,
  - and if so enable proxying for it. -}
2024-10-07 15:12:09 -04:00
Spencer
cb196337f4 additional question of spaces in URL 2024-10-07 19:10:19 +00:00
Spencer
abd56608cf 2024-10-07 19:02:17 +00:00
matrss
f650627b23 2024-10-07 14:40:19 +00:00
matrss
b0a6301cde Added a comment 2024-10-07 14:12:23 +00:00
Joey Hess
b501d23f9b
update 2024-10-07 10:06:12 -04:00
matrss
6b6ec39997 2024-10-07 13:59:56 +00:00
sng@353ca358075d9aa328f60a5439a3cee10f8301fe
b57677251b Added a comment 2024-10-06 21:42:13 +00:00
matrss
19f7b0e7d4 2024-10-02 15:07:54 +00:00