Avoid errors on non UTF-8 Windows #566

igrep · 2017-01-03T13:16:27Z

Problem

haddock exits with errors like below:

(1)

haddock: internal error: <stderr>: hPutChar: invalid argument (invalid character)

(2)

haddock: internal error: Language\Haskell\HsColour\Anchors.hs: hGetContents: invalid argument (invalid byte sequence)

(1) is caused by printing the "bullet" character onto stderr.
For example, this warning contains it:

Language\Haskell\HsColour\ANSI.hs:62:10: warning: [-Wmissing-methods]
    • No explicit implementation for
        ‘toEnum’
    • In the instance declaration for ‘Enum Highlight’

(2) is caused when the input file of readFile contains some Unicode characters.
In the case above, '⇒' is the cause.

Environment

OS: Windows 10
haddock: 2.17.3
GHC: 8.0.1

Solution

Add hSetEncoding handle utf8 to avoid the errors.

Note

I found the detailed causes by these changes for debugging:
- 8f29edb
- 1dd23bf
These errors happen even after executing chcp 65001 on the console.
According to the debug code, hGetEncoding stderr returns CP932 regardless of the console encoding.

alexbiehl

Thank you for your PR. I am sorry for the late response. I am still catching up on the backlog.

alexbiehl · 2017-04-03T12:42:05Z

haddock-api/src/Haddock/Interface.hs

@@ -165,6 +167,7 @@ createIfaces verbosity flags instIfaceMap mods = do
 processModule :: Verbosity -> ModSummary -> [Flag] -> IfaceMap -> InstIfaceMap -> Ghc (Maybe Interface)
 processModule verbosity modsum flags modMap instIfaceMap = do
  out verbosity verbose $ "Checking module " ++ moduleString (ms_mod modsum) ++ "..."
+  liftIO $ hSetEncoding stderr utf8


I am not sure this is a good idea. We would need a graceful degradation mechanism if the terminal isn't utf8 enabled.

Okay, I'm gonna look into how GHC handles such encoding incompatibility.
Tell me if you have any hints.

I finally found a workaround:

import GHC.IO.Encoding.CodePage import GHC.IO.Encoding.Failure import System.IO hSetEncoding stderr $ mkLocaleEncoding TransliterateCodingFailure

I'm trying to update the patch and rebase, but I have a difficulty testing with GHC 8.2.
The latest master is now for GHC 8.2, but I failed to install GHC 8.2 on my Windows... 😞

Sorry, mistaken.
I failed to build cabal-install with GHC 8.2 on my Windows. Actually, I successfully installed GHC 8.2.

Nice @igrep!

cabal-install: Are you trying cabal-install HEAD from git? What is the error message do you get?

And for haddock: Unfortunately ghc-8.2 added a new function which isn't present in ghc-8.2-rc2 but haddock relies upon. If you experience this try reverting d5d8cd1. Then you should be able to compile.

Also you could make your change and push it here. Travis will indicate if it is good or not.

Okay, I got it. I just should have installed standalone cabal-install from https://www.haskell.org/cabal/download.html ...
Maybe fixed the problem! Thanks!

Great! I am eager to to review your change as I am currently experiencing those utf8 errors myself.

OMG! Another error when building haddock... cabal's bug?

$ cabal new-build In order, the following will be built (use -v for more details): - ghc-paths-0.1.0.9 (lib:ghc-paths) (requires build) - haddock-api-2.18.0 (lib) (first run) - haddock-2.18.0 (exe:haddock) (first run) Configuring ghc-paths-0.1.0.9 (all, legacy fallback)... Building ghc-paths-0.1.0.9 (all, legacy fallback)... Failed to build ghc-paths-0.1.0.9. The failure occurred during the final install step. Build log ( C:\Users\igrep\AppData\Roaming\cabal\logs\ghc-8.2.0.20170507\ghc-paths-0.1.0.9-2b1e83330a86c551a66824a766311413e1d9a984.log ): Installing library in C:\Users\igrep\AppData\Roaming\cabal\store\ghc-8.2.0.20170507\incoming\new-14172\Users\igrep\AppData\Roaming\cabal\store\ghc-8.2.0.20170507\ghc-paths-0.1.0.9-2b1e83330a86c551a66824a766311413e1d9a984\lib copyFile: does not exist (指定されたパスが見つかりません。)box\prg\foreign\haddock\dist-newstyle\tmp\src-14172\ghc-paths-0.1.0.9\dist\set up\setup.exe ... Configuring ghc-paths-0.1.0.9... cabal.exe: Failed to build ghc-paths-0.1.0.9 (which is required by exe:haddock from haddock-2.18.0). See the build log above for details. exit status 1

Looks like haskell/cabal#4515

Fine! 🎉

$ haddock-master.exe Language/Haskell/HsColour/ANSI.hs Language\Haskell\HsColour\ANSI.hs:62:10: warning: [-Wmissing-methods] ? No explicit implementation for ▒etoEnum▒f ? In the instance declaration for ▒eEnum Highlight▒f | 62 | instance Enum Highlight where | ^^^^^^^^^^^^^^ Haddock's resource directory does not exist! Haddock coverage: 88% ( 7 / 8) in 'Language.Haskell.HsColour.ColourHighlight' Missing documentation for: Module header 33% ( 1 / 3) in 'Language.Haskell.HsColour.Output' Missing documentation for: Module header TerminalType (.\Language\Haskell\HsColour\Output.hs:3) 37% ( 10 / 27) in 'Language.Haskell.HsColour.ANSI' Missing documentation for: highlightOnG (Language/Haskell/HsColour/ANSI.hs:91) highlightOff (Language/Haskell/HsColour/ANSI.hs:96) cleareol (Language/Haskell/HsColour/ANSI.hs:46) clearbol (Language/Haskell/HsColour/ANSI.hs:47) clearline (Language/Haskell/HsColour/ANSI.hs:48) clearDown (Language/Haskell/HsColour/ANSI.hs:49) clearUp (Language/Haskell/HsColour/ANSI.hs:50) cursorUp (Language/Haskell/HsColour/ANSI.hs:41) cursorDown (Language/Haskell/HsColour/ANSI.hs:42) cursorLeft (Language/Haskell/HsColour/ANSI.hs:44) cursorRight (Language/Haskell/HsColour/ANSI.hs:43) savePosition (Language/Haskell/HsColour/ANSI.hs:55) restorePosition (Language/Haskell/HsColour/ANSI.hs:56) scrollUp (Language/Haskell/HsColour/ANSI.hs:118) scrollDown (Language/Haskell/HsColour/ANSI.hs:116) lineWrap (Language/Haskell/HsColour/ANSI.hs:122) TerminalType (.\Language\Haskell\HsColour\Output.hs:3) Warning: Language.Haskell.HsColour.ColourHighlight: could not find link destinations for: Word8 Enum succ pred toEnum Int fromEnum enumFrom enumFromThen enumFromTo enumFromThenTo Eq == Bool /= Read readsPrec ReadS readList readPrec ReadPrec readListPrec Show showsPrec ShowS show String showList Integral Warning: Language.Haskell.HsColour.Output: could not find link destinations for: Eq == Bool /= Ord compare Ordering < <= > >= max min Show showsPrec Int ShowS show String showList Warning: Language.Haskell.HsColour.ANSI: could not find link destinations for: String Char Int Eq == Bool /= Read readsPrec ReadS readList readPrec ReadPrec readListPrec Show showsPrec ShowS show showList Word8 Enum succ pred toEnum fromEnum enumFrom enumFromThen enumFromTo enumFromThenTo Ord compare Ordering < <= > >= max min

$ haddock.exe Language/Haskell/HsColour/ANSI.hs Language\Haskell\HsColour\ANSI.hs:62:10: warning: [-Wmissing-methods] Haddock coverage: 88% ( 7 / 8) in 'Language.Haskell.HsColour.ColourHighlight' Missing documentation for: Module header 33% ( 1 / 3) in 'Language.Haskell.HsColour.Output' Missing documentation for: Module header TerminalType (.\Language\Haskell\HsColour\Output.hs:3) haddock: internal error: <stderr>: hPutChar: invalid argument (invalid character)

alexbiehl · 2017-04-03T12:43:55Z

haddock-api/src/Haddock/Interface/Create.hs

-    rawSrc = readFile $ msHsFilePath ms
+    rawSrc = readFileUtf8 $ msHsFilePath ms
+
+readFileUtf8 :: FilePath -> IO String


This one on the other hand is a very good idea!

Problem ==== haddock exits with errors like below: `(1)` ``` haddock: internal error: <stderr>: hPutChar: invalid argument (invalid character) ``` `(2)` ``` haddock: internal error: Language\Haskell\HsColour\Anchors.hs: hGetContents: invalid argument (invalid byte sequence) ``` `(1)` is caused by printing [the "bullet" character](http://www.fileformat.info/info/unicode/char/2022/index.htm) onto stderr. For example, this warning contains it: ``` Language\Haskell\HsColour\ANSI.hs:62:10: warning: [-Wmissing-methods] • No explicit implementation for ‘toEnum’ • In the instance declaration for ‘Enum Highlight’ ``` `(2)` is caused when the input file of `readFile` contains some Unicode characters. In the case above, '⇒' is the cause. Environment ---- OS: Windows 10 haddock: 2.17.3 GHC: 8.0.1 Solution ==== Add `hSetEncoding handle utf8` to avoid the errors. Note ==== - I found the detailed causes by these changes for debugging: - haskell@8f29edb - haskell@1dd23bf - These errors happen even after executing `chcp 65001` on the console. According to the debug code, `hGetEncoding stderr` returns `CP932` regardless of the console encoding.

alexbiehl · 2017-07-04T13:00:01Z

Yes! I think so! Please write in that ticket that you are also affected by this. This maybe raises the interested in fixing it. As a workaround I can point you to https://www.haskell.org/cabal/release/cabal-install-1.24.0.2/cabal-install-1.24.0.2-x86_64-unknown-mingw32.zip This cabal doesn't use the new store layout yet. And it works for me right now. Yuji Yamamoto <[email protected]> schrieb am Di., 4. Juli 2017 um 14:58 Uhr:

…

***@***.**** commented on this pull request. ------------------------------ In haddock-api/src/Haddock/Interface.hs <#566 (comment)>: > @@ -165,6 +167,7 @@ createIfaces verbosity flags instIfaceMap mods = do processModule :: Verbosity -> ModSummary -> [Flag] -> IfaceMap -> InstIfaceMap -> Ghc (Maybe Interface) processModule verbosity modsum flags modMap instIfaceMap = do out verbosity verbose $ "Checking module " ++ moduleString (ms_mod modsum) ++ "..." + liftIO $ hSetEncoding stderr utf8 Looks like haskell/cabal#4515 <haskell/cabal#4515> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#566 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByiiVuWJmLRdF4Gp8kM81sttZJoV9geks5sKjbagaJpZM4LZm8v> .

alexbiehl · 2017-07-04T13:31:11Z

haddock-api/src/Haddock/Interface.hs

@@ -165,6 +172,9 @@ createIfaces verbosity flags instIfaceMap mods = do
 processModule :: Verbosity -> ModSummary -> [Flag] -> IfaceMap -> InstIfaceMap -> Ghc (Maybe Interface)
 processModule verbosity modsum flags modMap instIfaceMap = do
  out verbosity verbose $ "Checking module " ++ moduleString (ms_mod modsum) ++ "..."
+#if defined(mingw32_HOST_OS)


Nice! But one last nit: Wouldn't it make sense to set this up somewhere upstream in the call hierarchie?

Like somewhere around here: https://github.com/haskell/haddock/blob/master/haddock-api/src/Haddock.hs#L141

And: Do we need the CPP here? Couldn't this potentially happen on non Windows systems too?

Ah I see that mkLocaleEncoding only exists on Windows, thats ok then

How about readPackagesAndProcessModules? Or processModules ?

Please put it in haddockWithGhc it is the toplevel entrypoint. So we are sure nothing slips through.

🤔 Hmmm... processModules looks better because it's one of the public API of Documentation.Haddock.

…character)' non UTF-8 Windows Better solution for 5941175's (1)

alexbiehl · 2017-07-04T13:53:10Z

Looks good to me! As soon as it is green I will merge and ping ben to make sure this is included in the release for ghc-8.2.

alexbiehl · 2017-07-04T14:29:22Z

Thanks again @igrep! ben just confirmed this will be part of ghc-8.2! So you can enjoy your fruits of work soon!

igrep · 2017-07-08T05:03:28Z

Thank you! I confirmed GHC 8.2.1 rc3's haddock contains my patch! 😄

Steps to reproduce and the error message ==== ``` > stack haddock basement ... snip ... Warning: 'A' is out of scope. Warning: 'haddock: internal error: <stdout>: commitBuffer: invalid argument (invalid character) ``` Environment ==== OS: Windows 10 ver. 1709 haddock: [HEAD of ghc-8.4 when I reproduce the error](haskell@532b209). (I had to use this version to avoid another probrem already fixed in HEAD) GHC: 8.4.3 stack: Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2 Related pull request ==== haskell#566

…892) Steps to reproduce and the error message ==== ``` > stack haddock basement ... snip ... Warning: 'A' is out of scope. Warning: 'haddock: internal error: <stdout>: commitBuffer: invalid argument (invalid character) ``` Environment ==== OS: Windows 10 ver. 1709 haddock: [HEAD of ghc-8.4 when I reproduce the error](532b209). (I had to use this version to avoid another probrem already fixed in HEAD) GHC: 8.4.3 stack: Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2 Related pull request ==== #566

…#892) Steps to reproduce and the error message ==== ``` > stack haddock basement ... snip ... Warning: 'A' is out of scope. Warning: 'haddock: internal error: <stdout>: commitBuffer: invalid argument (invalid character) ``` Environment ==== OS: Windows 10 ver. 1709 haddock: [HEAD of ghc-8.4 when I reproduce the error](haskell/haddock@532b209). (I had to use this version to avoid another probrem already fixed in HEAD) GHC: 8.4.3 stack: Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2 Related pull request ==== haskell/haddock#566

alexbiehl reviewed Apr 3, 2017

View reviewed changes

igrep force-pushed the windows-non-utf8 branch from d4009bd to 0333dcc Compare July 4, 2017 13:19

alexbiehl reviewed Jul 4, 2017

View reviewed changes

Avoid 'internal error: <stderr>: hPutChar: invalid argument (invalid …

855118e

…character)' non UTF-8 Windows Better solution for 5941175's (1)

igrep force-pushed the windows-non-utf8 branch from 0333dcc to 855118e Compare July 4, 2017 13:51

alexbiehl merged commit 22cbf4d into haskell:master Jul 4, 2017

igrep deleted the windows-non-utf8 branch July 4, 2017 14:16

matsubara0507 mentioned this pull request Feb 25, 2018

Avoid errors on non UTF-8 Windows decentral1se/tasty-discover#138

Merged

dfordivam mentioned this pull request May 21, 2018

windows-gotchas を翻訳 haskell-jp/blog#118

Merged

igrep mentioned this pull request Jul 4, 2018

hGetContents: invalid argument (invalid byte sequence) on Japanese Windows haskell/haskell-ide-engine#667

Closed

igrep mentioned this pull request Jul 23, 2018

Avoid "invalid argument (invalid character)" on non-unicode Windows #892

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid errors on non UTF-8 Windows #566

Avoid errors on non UTF-8 Windows #566

igrep commented Jan 3, 2017

alexbiehl left a comment

alexbiehl Apr 3, 2017

igrep Apr 4, 2017

igrep Jul 2, 2017

igrep Jul 2, 2017

alexbiehl Jul 3, 2017 •

edited

Loading

igrep Jul 4, 2017 •

edited

Loading

alexbiehl Jul 4, 2017

igrep Jul 4, 2017

igrep Jul 4, 2017

igrep Jul 4, 2017

alexbiehl Apr 3, 2017

alexbiehl commented Jul 4, 2017 via email

alexbiehl Jul 4, 2017

alexbiehl Jul 4, 2017

alexbiehl Jul 4, 2017

alexbiehl Jul 4, 2017

igrep Jul 4, 2017

alexbiehl Jul 4, 2017

igrep Jul 4, 2017

alexbiehl Jul 4, 2017

alexbiehl commented Jul 4, 2017

alexbiehl commented Jul 4, 2017

igrep commented Jul 8, 2017

Avoid errors on non UTF-8 Windows #566

Avoid errors on non UTF-8 Windows #566

Conversation

igrep commented Jan 3, 2017

Problem

Environment

Solution

Note

alexbiehl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbiehl Jul 3, 2017 • edited Loading

Choose a reason for hiding this comment

igrep Jul 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbiehl commented Jul 4, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbiehl commented Jul 4, 2017

alexbiehl commented Jul 4, 2017

igrep commented Jul 8, 2017

alexbiehl Jul 3, 2017 •

edited

Loading

igrep Jul 4, 2017 •

edited

Loading