From 29d687dce982a816e61406db589433b4d5f62152 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 14 Oct 2021 12:45:05 -0400 Subject: [PATCH] When retrival from a chunked remote fails, display the error that occurred when downloading the chunk Rather than the error that occurred when trying to download the unchunked content, which is less likely to actually be stored in the remote. Sponsored-by: Boyd Stephen Smith Jr. on Patreon --- CHANGELOG | 4 ++++ Remote/Helper/Chunked.hs | 15 ++++++++++----- ...mprovements_to_S3_glacier_integration.mdwn | 2 ++ ..._9cbe83bbade15b9146d033ceb5d8b05d._comment | 19 +++++++++++++++++++ doc/special_remotes/S3.mdwn | 9 ++++++--- 5 files changed, 41 insertions(+), 8 deletions(-) create mode 100644 doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment diff --git a/CHANGELOG b/CHANGELOG index dfd6b555c8..e8e4ad1d08 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -10,6 +10,10 @@ git-annex (8.20211012) UNRELEASED; urgency=medium * test: Put gpg temp home directory in system temp directory, not filesystem being tested. * Avoid crashing tilde expansion on user who does not exist. + * When retrival from a chunked remote fails, display the error that + occurred when downloading the chunk, rather than the error that + occurred when trying to download the unchunked content, which is less + likely to actually be stored in the remote. -- Joey Hess Mon, 11 Oct 2021 14:09:13 -0400 diff --git a/Remote/Helper/Chunked.hs b/Remote/Helper/Chunked.hs index ade47f3f4c..b56d43389a 100644 --- a/Remote/Helper/Chunked.hs +++ b/Remote/Helper/Chunked.hs @@ -286,9 +286,9 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc firstavail Nothing _ [] = giveup "unable to determine the chunks to use for this remote" firstavail (Just e) _ [] = throwM e firstavail pe currsize ([]:ls) = firstavail pe currsize ls - firstavail _ currsize ((k:ks):ls) + firstavail pe currsize ((k:ks):ls) | k == basek = getunchunked - `catchNonAsync` (\e -> firstavail (Just e) currsize ls) + `catchNonAsync` (\e -> firstavail (Just (pickerr e)) currsize ls) | otherwise = do let offset = resumeOffset currsize k let p = maybe basep @@ -302,10 +302,15 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc fromMaybe 0 $ fromKey keyChunkSize k getrest p h iv sz sz ks case v of - Left e - | null ls -> throwM e - | otherwise -> firstavail (Just e) currsize ls + Left e -> firstavail (Just (pickerr e)) currsize ls Right r -> return r + where + -- Prefer an earlier exception to a later one, because the + -- more probable location is tried first and less probable + -- ones later. + pickerr e = case pe of + Just pe' -> pe' + Nothing -> e getrest _ _ iv _ _ [] = return (Right iv) getrest p h iv sz bytesprocessed (k:ks) = do diff --git a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn index 233acfdbed..983869528d 100644 --- a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn +++ b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn @@ -115,3 +115,5 @@ ongoing-request="false", expiry-date="Mon, 26 Apr 2021 00:00:00 GMT" ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) Yes, for loads of stuff. It's awesome, thanks! + +> [[closed|done]], see my comment --[[Joey]] diff --git a/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment new file mode 100644 index 0000000000..b0dfda3309 --- /dev/null +++ b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2021-10-14T16:17:14Z" + content=""" +I have added a note to the S3 documentation about `DEEP_ARCHIVE` and the +glacier special remote. + +I have made git-annex display the exception for the more likely chunked +location, rather than the less likely unchunked location, when retrieving +from both locations fails. Although it's still possible for there to be +situations where the exception if displays is not for the location where +the content actually is. Eg, if the chunk size of the remote has +changed over time. + +I think that todo is basically talking about the same desire to make the S3 +remote support these glacier-style storage classes, in one way or another, +and so I think this bug report can be closed as otherwise a duplicate of it. +"""]] diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn index 71d74a533a..e34cbb7f9d 100644 --- a/doc/special_remotes/S3.mdwn +++ b/doc/special_remotes/S3.mdwn @@ -44,14 +44,17 @@ the S3 remote. When using Amazon S3, if the remote will be used for backup or archival, - and so its files are Infrequently Accessed, "STANDARD_IA" is a + and so its files are Infrequently Accessed, `STANDARD_IA` is a good choice to save money (requires a git-annex built with aws-0.13.0). If you have configured git-annex to preserve - multiple [[copies]], also consider setting this to "ONEZONE_IA" + multiple [[copies]], also consider setting this to `ONEZONE_IA` to save even more money. + Amazon S3's `DEEP_ARCHIVE` is similar to Amazon Glacier. For that, + use the [[glacier]] special remote, rather than this one. + When using Google Cloud Storage, to make a nearline bucket, set this to - "NEARLINE". (Requires a git-annex built with aws-0.13.0) + `NEARLINE`. (Requires a git-annex built with aws-0.13.0) Note that changing the storage class of an existing S3 remote will affect new objects sent to the remote, but not objects already