mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions 1
git-annex/Utility/Base64.hs

26 lines
552 B
Haskell
Raw Permalink Normal View History

switch MetaValue to ByteString and MetaField to Text MetaField was already limited to alphanumerics, so it makes sense to use Text for it. Note that technically a UUID can contain invalid UTF-8, and so remoteMetaDataPrefix's use of T.pack . fromUUID could replace non-UTF8 values with '?' or whatever. In practice, a UUID is usually also text, I only kept open the possibility of it containing invalid UTF-8 to avoid breaking parsing of strange UUIDs in git-annex branch files. So, I decided to let this edge case slip by. Have not updated the rest of the code base yet for this change, as the change took 2.5 hours longer than I expected to get working properly.
2019-01-07 14:18:24 -04:00
{- Simple Base64 encoding
Fix setting/setting/viewing metadata that contains unicode or other special characters, when in a non-unicode locale. Oh boy, not again. So, another place that the filesystem encoding needs to be applied. Yay. In passing, I changed decodeBS so if a NUL is embedded in the input, the resulting FilePath doesn't get truncated at that NUL. This was needed to make prop_b64_roundtrips pass, and on reviewing the callers of decodeBS, I didn't see any where this wouldn't make sense. When a FilePath is used to operate on the filesystem, it'll get truncated at a NUL anyway, whereas if a String is being used for something else, it might conceivably have a NUL in it, and we wouldn't want it to get truncated when going through decodeBS. (NB: There may be a speed impact from this change.)
2015-08-11 18:40:59 -04:00
-
simplify base64 to only use ByteString Note the use of fromString and toString from Data.ByteString.UTF8 dated back to commit 9b93278e8abe1163d53fbf56909d0fe6d7de69e9. Back then it was using the dataenc package for base64, which operated on Word8 and String. But with the switch to sandi, it uses ByteString, and indeed fromB64' and toB64' were already using ByteString without that complication. So I think there is no risk of such an encoding related breakage. I also tested the case that 9b93278e8abe1163d53fbf56909d0fe6d7de69e9 fixed: git-annex metadata -s foo='a …' x git-annex metadata x metadata x foo=a … In Remote.Helper.Encryptable, it was avoiding using Utility.Base64 because of that UTF8 conversion. Since that's no longer done, it can just use it now.
2023-10-26 12:42:32 -04:00
- Copyright 2011-2023 Joey Hess <id@joeyh.name>
factor out base64 code
2011-05-01 14:27:40 -04:00
-
relicense general utility library code to BSD Omitted a couple of files what have had significant contributions from others.
2014-05-10 11:01:27 -03:00
- License: BSD-2-clause
factor out base64 code
2011-05-01 14:27:40 -04:00
-}
switch MetaValue to ByteString and MetaField to Text MetaField was already limited to alphanumerics, so it makes sense to use Text for it. Note that technically a UUID can contain invalid UTF-8, and so remoteMetaDataPrefix's use of T.pack . fromUUID could replace non-UTF8 values with '?' or whatever. In practice, a UUID is usually also text, I only kept open the possibility of it containing invalid UTF-8 to avoid breaking parsing of strange UUIDs in git-annex branch files. So, I decided to let this edge case slip by. Have not updated the rest of the code base yet for this change, as the change took 2.5 hours longer than I expected to get working properly.
2019-01-07 14:18:24 -04:00
module Utility.Base64 where
factor out base64 code
2011-05-01 14:27:40 -04:00
filter out control characters in error messages giveup changed to filter out control characters. (It is too low level to make it use StringContainingQuotedPath.) error still does not, but it should only be used for internal errors, where the message is not attacker-controlled. Changed a lot of existing error to giveup when it is not strictly an internal error. Of course, other exceptions can still be thrown, either by code in git-annex, or a library, that include some attacker-controlled value. This does not guard against those. Sponsored-by: Noam Kremen on Patreon
2023-04-10 13:38:14 -04:00
import Utility.Exception
fix test suite fail in LANG=C This was caused by 23e9d3bb777d580931a5b4d7ee57aa604190d8b2 an Arbitrary String is not necessarily encoded using the filesystem encoding, and in a non-utf8 locale, encodeBS throws an exception on such a string. All I could think to do is limit test data to ascii. This shouldn't be a problem in practice, because the all Strings in git-annex that are not generated by Arbitrary should be loaded in a way that does apply the filesystem encoding.
2015-08-12 10:36:51 -04:00
simplify base64 to only use ByteString Note the use of fromString and toString from Data.ByteString.UTF8 dated back to commit 9b93278e8abe1163d53fbf56909d0fe6d7de69e9. Back then it was using the dataenc package for base64, which operated on Word8 and String. But with the switch to sandi, it uses ByteString, and indeed fromB64' and toB64' were already using ByteString without that complication. So I think there is no risk of such an encoding related breakage. I also tested the case that 9b93278e8abe1163d53fbf56909d0fe6d7de69e9 fixed: git-annex metadata -s foo='a …' x git-annex metadata x metadata x foo=a … In Remote.Helper.Encryptable, it was avoiding using Utility.Base64 because of that UTF8 conversion. Since that's no longer done, it can just use it now.
2023-10-26 12:42:32 -04:00
import Codec.Binary.Base64 as B64
tag xmpp pushes with jid This fixes the issue mentioned in the last commit. Turns out just collecting UUID of clients behind a XMPP remote is insufficient (although I should probably still do it for other reasons), because a single remote repo might be connected via both XMPP and local pairing. So a way is needed to know when a push was received from any client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
import Data.Maybe
switch MetaValue to ByteString and MetaField to Text MetaField was already limited to alphanumerics, so it makes sense to use Text for it. Note that technically a UUID can contain invalid UTF-8, and so remoteMetaDataPrefix's use of T.pack . fromUUID could replace non-UTF8 values with '?' or whatever. In practice, a UUID is usually also text, I only kept open the possibility of it containing invalid UTF-8 to avoid breaking parsing of strange UUIDs in git-annex branch files. So, I decided to let this edge case slip by. Have not updated the rest of the code base yet for this change, as the change took 2.5 hours longer than I expected to get working properly.
2019-01-07 14:18:24 -04:00
import qualified Data.ByteString as B
factor out base64 code
2011-05-01 14:27:40 -04:00
simplify base64 to only use ByteString Note the use of fromString and toString from Data.ByteString.UTF8 dated back to commit 9b93278e8abe1163d53fbf56909d0fe6d7de69e9. Back then it was using the dataenc package for base64, which operated on Word8 and String. But with the switch to sandi, it uses ByteString, and indeed fromB64' and toB64' were already using ByteString without that complication. So I think there is no risk of such an encoding related breakage. I also tested the case that 9b93278e8abe1163d53fbf56909d0fe6d7de69e9 fixed: git-annex metadata -s foo='a …' x git-annex metadata x metadata x foo=a … In Remote.Helper.Encryptable, it was avoiding using Utility.Base64 because of that UTF8 conversion. Since that's no longer done, it can just use it now.
2023-10-26 12:42:32 -04:00
toB64 :: B.ByteString -> B.ByteString
toB64 = B64.encode
switch MetaValue to ByteString and MetaField to Text MetaField was already limited to alphanumerics, so it makes sense to use Text for it. Note that technically a UUID can contain invalid UTF-8, and so remoteMetaDataPrefix's use of T.pack . fromUUID could replace non-UTF8 values with '?' or whatever. In practice, a UUID is usually also text, I only kept open the possibility of it containing invalid UTF-8 to avoid breaking parsing of strange UUIDs in git-annex branch files. So, I decided to let this edge case slip by. Have not updated the rest of the code base yet for this change, as the change took 2.5 hours longer than I expected to get working properly.
2019-01-07 14:18:24 -04:00
simplify base64 to only use ByteString Note the use of fromString and toString from Data.ByteString.UTF8 dated back to commit 9b93278e8abe1163d53fbf56909d0fe6d7de69e9. Back then it was using the dataenc package for base64, which operated on Word8 and String. But with the switch to sandi, it uses ByteString, and indeed fromB64' and toB64' were already using ByteString without that complication. So I think there is no risk of such an encoding related breakage. I also tested the case that 9b93278e8abe1163d53fbf56909d0fe6d7de69e9 fixed: git-annex metadata -s foo='a …' x git-annex metadata x metadata x foo=a … In Remote.Helper.Encryptable, it was avoiding using Utility.Base64 because of that UTF8 conversion. Since that's no longer done, it can just use it now.
2023-10-26 12:42:32 -04:00
fromB64Maybe :: B.ByteString -> Maybe (B.ByteString)
fromB64Maybe = either (const Nothing) Just . B64.decode
tag xmpp pushes with jid This fixes the issue mentioned in the last commit. Turns out just collecting UUID of clients behind a XMPP remote is insufficient (although I should probably still do it for other reasons), because a single remote repo might be connected via both XMPP and local pairing. So a way is needed to know when a push was received from any client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
simplify base64 to only use ByteString Note the use of fromString and toString from Data.ByteString.UTF8 dated back to commit 9b93278e8abe1163d53fbf56909d0fe6d7de69e9. Back then it was using the dataenc package for base64, which operated on Word8 and String. But with the switch to sandi, it uses ByteString, and indeed fromB64' and toB64' were already using ByteString without that complication. So I think there is no risk of such an encoding related breakage. I also tested the case that 9b93278e8abe1163d53fbf56909d0fe6d7de69e9 fixed: git-annex metadata -s foo='a …' x git-annex metadata x metadata x foo=a … In Remote.Helper.Encryptable, it was avoiding using Utility.Base64 because of that UTF8 conversion. Since that's no longer done, it can just use it now.
2023-10-26 12:42:32 -04:00
fromB64 :: B.ByteString -> B.ByteString
tag xmpp pushes with jid This fixes the issue mentioned in the last commit. Turns out just collecting UUID of clients behind a XMPP remote is insufficient (although I should probably still do it for other reasons), because a single remote repo might be connected via both XMPP and local pairing. So a way is needed to know when a push was received from any client using a given XMPP remote over XMPP, as opposed to via ssh.
2013-03-06 16:29:19 -04:00
fromB64 = fromMaybe bad . fromB64Maybe
where
filter out control characters in error messages giveup changed to filter out control characters. (It is too low level to make it use StringContainingQuotedPath.) error still does not, but it should only be used for internal errors, where the message is not attacker-controlled. Changed a lot of existing error to giveup when it is not strictly an internal error. Of course, other exceptions can still be thrown, either by code in git-annex, or a library, that include some attacker-controlled value. This does not guard against those. Sponsored-by: Noam Kremen on Patreon
2023-04-10 13:38:14 -04:00
bad = giveup "bad base64 encoded data"
Reference in a new issue Copy permalink