importfeed: Fix a crash when used in a non-unicode locale

See comment for analysis.

At first I thought I'd need to convert all T.unpack in git-annex, but
luckily not -- so long as the Text is read from a file, the filesystem
encoding is applied and T.unpack is fine. It's only when using Feed
that the filesystem encoding is not applied.

While this fixes the crash, it does result in some mojibake, eg:
itemid=http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-���-questions/

Have not tracked that down, but it must be unrelated, because
I've verified that it roundtrips when using encodeUf8:

joey@darkstar:~/src/git-annex>LANG=C ghci  Utility/FileSystemEncoding.hs
ghci> useFileSystemEncoding
ghci> Just f <- Text.Feed.Import.parseFeedFromFile "/home/joey/tmp/career_tools_podcasts.xml"
ghci> Just (_, x) = Text.Feed.Query.getItemId (Text.Feed.Query.feedItems f !! 0)
ghci> decodeBS (Data.Text.Encoding.encodeUtf8 x)
"http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-\56546\56448\56467-questions/"
ghci> writeFile "foo" $ decodeBS (Data.Text.Encoding.encodeUtf8 x)
Writes a file containing the ENDASH character.

Sponsored-by: Jochen Bartl on Patreon
This commit is contained in:
Joey Hess 2021-11-15 13:32:31 -04:00
parent b12f7f84f7
commit 2bd778a46e
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 92 additions and 13 deletions

View file

@ -69,3 +69,5 @@ $ echo $?
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've been using git annex for a few years now primarily as a podcatcher. I've also been using it to manage my ebooks and devices. I'm slowly starting to use it for managing my personal (non-text) records and documents.
> [[fixed|done]] --[[Joey]]

View file

@ -0,0 +1,60 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-11-15T16:04:31Z"
content="""
To reproduce this, I had to set LANG=C. Using a unicode locale avoids
the problem.
The `.met` file indicates it's a problem with encoding of metadata
that is being imported from the feed, and so it must be the
itemid that is causing the problem.
<item>
...
<guid>http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7--questions/</guid>
</item>
A file with the feed edited down to just that item is enough to reproduce it.
Notice the unicode in the guid "chapter-7--questions".
That ENDASH character is causing the crash.
Also I noticed that the next time it runs, it skips the item, since it got
far enough to add the file for it and record the url before the metadata itemid
write crashed it. Explains why it's failing on different items in different runs.
While this looks like one of the old Handle output encoding problems, it is not,
because a) the itemid is written as a ByteString so encoding does not matter,
b) those were fixed comprehensively by forcing all handles to use filesystem
enconding, and c) just printing out the length of the itemid also causes a crash:
+ liftIO $ print (L.length (journalableByteString content))
git-annex: recoverEncode: invalid argument (invalid character)
Looking at what the feed library parses:
LANG=C ghci Utility/FileSystemEncoding.hs
ghci> Just f <- Text.Feed.Import.parseFeedFromFile "career_tools_podcasts.xml"
ghci> Just (_, x) = Text.Feed.Query.getItemId (Text.Feed.Query.feedItems f !! 0)
ghci> x
"http://www.manager-tools.com/2014/01/choosing-a-company-work-chapter-7-\8211-questions/"
ghci> encodeBS (Data.Text.unpack x)
"*** Exception: recoverEncode: invalid argument (invalid character)
So the problem is that Text parses the feed as unicode, leading to this
non-ascii Char that is not encoded using the filesystem encoding
(which would encode it as "\56546\56448\56467").
And `encodeBS "\8211"` crashes in LANG=C.
Which is a reversion of sorts; before [[!commit fa62c98910]] encodeBS did
not crash. Although it also didn't round-trip this value properly,
producing "M" for it. Since this only affects strings that are not input in
the filesystem encoding, I think the new encodeBS is still ok to use
generally; I'm not going to revert that commit.
Instead, Text values originating from Feed need to be converted to
String in some other way, producing a value encoded
using the filesystem encoding. encodeUtf8 looks like it will
do the right thing in this case.
"""]]