importfeed: Fix a crash when used in a non-unicode locale
See comment for analysis. At first I thought I'd need to convert all T.unpack in git-annex, but luckily not -- so long as the Text is read from a file, the filesystem encoding is applied and T.unpack is fine. It's only when using Feed that the filesystem encoding is not applied. While this fixes the crash, it does result in some mojibake, eg: itemid=http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-���-questions/ Have not tracked that down, but it must be unrelated, because I've verified that it roundtrips when using encodeUf8: joey@darkstar:~/src/git-annex>LANG=C ghci Utility/FileSystemEncoding.hs ghci> useFileSystemEncoding ghci> Just f <- Text.Feed.Import.parseFeedFromFile "/home/joey/tmp/career_tools_podcasts.xml" ghci> Just (_, x) = Text.Feed.Query.getItemId (Text.Feed.Query.feedItems f !! 0) ghci> decodeBS (Data.Text.Encoding.encodeUtf8 x) "http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-\56546\56448\56467-questions/" ghci> writeFile "foo" $ decodeBS (Data.Text.Encoding.encodeUtf8 x) Writes a file containing the ENDASH character. Sponsored-by: Jochen Bartl on Patreon
This commit is contained in:
parent
b12f7f84f7
commit
2bd778a46e
4 changed files with 92 additions and 13 deletions
|
@ -69,3 +69,5 @@ $ echo $?
|
|||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
I've been using git annex for a few years now primarily as a podcatcher. I've also been using it to manage my ebooks and devices. I'm slowly starting to use it for managing my personal (non-text) records and documents.
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
|
@ -0,0 +1,60 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2021-11-15T16:04:31Z"
|
||||
content="""
|
||||
To reproduce this, I had to set LANG=C. Using a unicode locale avoids
|
||||
the problem.
|
||||
|
||||
The `.met` file indicates it's a problem with encoding of metadata
|
||||
that is being imported from the feed, and so it must be the
|
||||
itemid that is causing the problem.
|
||||
|
||||
<item>
|
||||
...
|
||||
<guid>http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-–-questions/</guid>
|
||||
</item>
|
||||
|
||||
A file with the feed edited down to just that item is enough to reproduce it.
|
||||
Notice the unicode in the guid "chapter-7-–-questions".
|
||||
That ENDASH character is causing the crash.
|
||||
|
||||
Also I noticed that the next time it runs, it skips the item, since it got
|
||||
far enough to add the file for it and record the url before the metadata itemid
|
||||
write crashed it. Explains why it's failing on different items in different runs.
|
||||
|
||||
While this looks like one of the old Handle output encoding problems, it is not,
|
||||
because a) the itemid is written as a ByteString so encoding does not matter,
|
||||
b) those were fixed comprehensively by forcing all handles to use filesystem
|
||||
enconding, and c) just printing out the length of the itemid also causes a crash:
|
||||
|
||||
+ liftIO $ print (L.length (journalableByteString content))
|
||||
|
||||
git-annex: recoverEncode: invalid argument (invalid character)
|
||||
|
||||
Looking at what the feed library parses:
|
||||
|
||||
LANG=C ghci Utility/FileSystemEncoding.hs
|
||||
ghci> Just f <- Text.Feed.Import.parseFeedFromFile "career_tools_podcasts.xml"
|
||||
ghci> Just (_, x) = Text.Feed.Query.getItemId (Text.Feed.Query.feedItems f !! 0)
|
||||
ghci> x
|
||||
"http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-\8211-questions/"
|
||||
ghci> encodeBS (Data.Text.unpack x)
|
||||
"*** Exception: recoverEncode: invalid argument (invalid character)
|
||||
|
||||
So the problem is that Text parses the feed as unicode, leading to this
|
||||
non-ascii Char that is not encoded using the filesystem encoding
|
||||
(which would encode it as "\56546\56448\56467").
|
||||
And `encodeBS "\8211"` crashes in LANG=C.
|
||||
|
||||
Which is a reversion of sorts; before [[!commit fa62c98910]] encodeBS did
|
||||
not crash. Although it also didn't round-trip this value properly,
|
||||
producing "M" for it. Since this only affects strings that are not input in
|
||||
the filesystem encoding, I think the new encodeBS is still ok to use
|
||||
generally; I'm not going to revert that commit.
|
||||
|
||||
Instead, Text values originating from Feed need to be converted to
|
||||
String in some other way, producing a value encoded
|
||||
using the filesystem encoding. encodeUtf8 looks like it will
|
||||
do the right thing in this case.
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue