Don't add TAB when folding headers

[th-h]

Summary

XanaNews will fold headers containing MIME encoded words to keep them from getting longer (which is correct), but does so (at least in corner cases) by adding a TAB after the CRLF. That can (and will) lead to added whitespace after unfolding in other (RFC compliant) clients.

RFCs

RFC 5536 (Netnews Article Format) mostly refers to RFC 5322 (Internet Message Format). In "2.2. Header Fields", it says:

All header fields in a Netnews article are compliant with [RFC5322]; this specification, however, is less permissive in what can be generated and accepted by agents. The syntax allowed for Netnews article headers is a strict subset of the Internet Message Format headers, making all headers compliant with this specification inherently compliant with [RFC5322].

In "2.1. Base", it refers to RFC 5322 for the definition of folding:

This document follows the terminology in Section 2 of [RFC5322] in using the terms "line", "header field", "header field name", "header field body", and "folding", based on a belief that consistent terminology among specifications that depend on each other makes the specifications easier to use in the long run.

RFC 5322 defines folding (and unfolding) of headers like this ("2.2.3. Long Header Fields"):

Each header field is logically a single line of characters comprising the field name, the colon, and the field body. For convenience however, and to deal with the 998/78 character limitations per line, the field body portion of a header field can be split into a multiple-line representation; this is called "folding". The general rule is that wherever this specification allows for folding white space (not simply WSP characters), a CRLF may be inserted before any WSP.
[...]
Unfolding is accomplished by simply removing any CRLF that is immediately followed by WSP.

(The same is true for the old RFC 1036 "Standard for USENET Messages" from 1987, which refers to the even older RFC 822 "Standard for the format of ARPA Internet Text Messages" from 1982 that has basically the same rules on folding.)

Bug

So XanaNews should just add a CRLF before any FWS (e.g. SPACE or TAB or whatever).
It does add CRLF+TAB after a SPACE.
Other clients will unfold that to SPACE+SPACE.

For example, a Subject of This is a test message to show header folding with encoded words -täst- will become

Subject: This is a test message to show header folding with encoded words 
	=?utf-8?Q?-t=C3=A4st-?=

(SPACE after "words", TAB before "=?utf-8?Q?-t=C3=A4st-?=".)

It should become

Subject: This is a test message to show header folding with encoded words
 =?utf-8?Q?-t=C3=A4st-?=

(No SPACE after "words", SPACE before "=?utf-8?Q?-t=C3=A4st-?=".)

Unfolding the Subject header folded by XanaNews would be done by removing the CRLF, so having SPACE and TAB between "words" and "-täst-", adding more whitespace than there was before.

Please see <[email protected]> and <[email protected]> in misc.test for a real life example, or the archived versions of those posts at http://al.howardknight.net/?ID=159195862400 and http://al.howardknight.net/?ID=159195864600, respectively.