diff --git a/CHANGELOG b/CHANGELOG index dfd6b555c8..e8e4ad1d08 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -10,6 +10,10 @@ git-annex (8.20211012) UNRELEASED; urgency=medium * test: Put gpg temp home directory in system temp directory, not filesystem being tested. * Avoid crashing tilde expansion on user who does not exist. + * When retrival from a chunked remote fails, display the error that + occurred when downloading the chunk, rather than the error that + occurred when trying to download the unchunked content, which is less + likely to actually be stored in the remote. -- Joey Hess Mon, 11 Oct 2021 14:09:13 -0400 diff --git a/Remote/Helper/Chunked.hs b/Remote/Helper/Chunked.hs index ade47f3f4c..b56d43389a 100644 --- a/Remote/Helper/Chunked.hs +++ b/Remote/Helper/Chunked.hs @@ -286,9 +286,9 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc firstavail Nothing _ [] = giveup "unable to determine the chunks to use for this remote" firstavail (Just e) _ [] = throwM e firstavail pe currsize ([]:ls) = firstavail pe currsize ls - firstavail _ currsize ((k:ks):ls) + firstavail pe currsize ((k:ks):ls) | k == basek = getunchunked - `catchNonAsync` (\e -> firstavail (Just e) currsize ls) + `catchNonAsync` (\e -> firstavail (Just (pickerr e)) currsize ls) | otherwise = do let offset = resumeOffset currsize k let p = maybe basep @@ -302,10 +302,15 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc fromMaybe 0 $ fromKey keyChunkSize k getrest p h iv sz sz ks case v of - Left e - | null ls -> throwM e - | otherwise -> firstavail (Just e) currsize ls + Left e -> firstavail (Just (pickerr e)) currsize ls Right r -> return r + where + -- Prefer an earlier exception to a later one, because the + -- more probable location is tried first and less probable + -- ones later. + pickerr e = case pe of + Just pe' -> pe' + Nothing -> e getrest _ _ iv _ _ [] = return (Right iv) getrest p h iv sz bytesprocessed (k:ks) = do diff --git a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn index 233acfdbed..983869528d 100644 --- a/doc/bugs/Improvements_to_S3_glacier_integration.mdwn +++ b/doc/bugs/Improvements_to_S3_glacier_integration.mdwn @@ -115,3 +115,5 @@ ongoing-request="false", expiry-date="Mon, 26 Apr 2021 00:00:00 GMT" ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) Yes, for loads of stuff. It's awesome, thanks! + +> [[closed|done]], see my comment --[[Joey]] diff --git a/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment new file mode 100644 index 0000000000..b0dfda3309 --- /dev/null +++ b/doc/bugs/Improvements_to_S3_glacier_integration/comment_5_9cbe83bbade15b9146d033ceb5d8b05d._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2021-10-14T16:17:14Z" + content=""" +I have added a note to the S3 documentation about `DEEP_ARCHIVE` and the +glacier special remote. + +I have made git-annex display the exception for the more likely chunked +location, rather than the less likely unchunked location, when retrieving +from both locations fails. Although it's still possible for there to be +situations where the exception if displays is not for the location where +the content actually is. Eg, if the chunk size of the remote has +changed over time. + +I think that todo is basically talking about the same desire to make the S3 +remote support these glacier-style storage classes, in one way or another, +and so I think this bug report can be closed as otherwise a duplicate of it. +"""]] diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn index 71d74a533a..e34cbb7f9d 100644 --- a/doc/special_remotes/S3.mdwn +++ b/doc/special_remotes/S3.mdwn @@ -44,14 +44,17 @@ the S3 remote. When using Amazon S3, if the remote will be used for backup or archival, - and so its files are Infrequently Accessed, "STANDARD_IA" is a + and so its files are Infrequently Accessed, `STANDARD_IA` is a good choice to save money (requires a git-annex built with aws-0.13.0). If you have configured git-annex to preserve - multiple [[copies]], also consider setting this to "ONEZONE_IA" + multiple [[copies]], also consider setting this to `ONEZONE_IA` to save even more money. + Amazon S3's `DEEP_ARCHIVE` is similar to Amazon Glacier. For that, + use the [[glacier]] special remote, rather than this one. + When using Google Cloud Storage, to make a nearline bucket, set this to - "NEARLINE". (Requires a git-annex built with aws-0.13.0) + `NEARLINE`. (Requires a git-annex built with aws-0.13.0) Note that changing the storage class of an existing S3 remote will affect new objects sent to the remote, but not objects already