Queued Mail Delivery Sets Wrong Charset

[treinhard]

Sending an email via pyramid_mailer's send_to_queue() and then delivering it with the qp console script results in an incorrect charset of us-ascii in the Content-Type on Python 3.6.2.

The initial Message with unicode content is correctly encoded with iso-8859-1 and the charset is set on the message:

>>> from pyramid_mailer.message import Message
>>> msg = Message(subject='Test', sender='[email protected]', recipients=['[email protected]'], body='Test französisches Email')
>>> msg = msg.to_message()
>>> msg.get_charset()
'iso-8859-1'
>>> msg.as_string()
'Content-Type: text/plain; charset="iso-8859-1"\nMIME-Version: 1.0\nContent-Transfer-Encoding: quoted-printable\nFrom: [email protected]\nSubject: Test\nTo: [email protected]\nContent-Disposition: inline\n\nTest=20franz=F6sisches=20Email'

The content is written to a file and later (during delivery) parsed in the QueueProcessor:

>>> from email.parser import Parser
>>> from io import StringIO
>>> msg = parser.parse(StringIO(msg.as_string()))
>>> print(msg.get_charset())
None
>>> msg.get_content_charset()
'iso-8859-1'
>>> msg.as_string()
'Content-Type: text/plain; charset="iso-8859-1"\nMIME-Version: 1.0\nContent-Transfer-Encoding: quoted-printable\nFrom: [email protected]\nSubject: Test\nTo: [email protected]\nContent-Disposition: inline\n\nTest=20franz=F6sisches=20Email'

Message looks ok (except that msg.get_charset() returns now None). SMTPMailer.send() runs this message through repoze.sendmail.encoding.cleanup_message() which replaces the initial message charset of iso-8859-1 with us-ascii:

>>> from repoze.sendmail.encoding import cleanup_message
>>> msg = cleanup_message(msg)
>>> msg.get_charset()
'us-ascii'
>>> msg.get_content_charset()
'us-ascii'
>>> msg.as_string()
'MIME-Version: 1.0\nContent-Transfer-Encoding: quoted-printable\nFrom: [email protected]\nSubject: Test\nTo: [email protected]\nContent-Disposition: inline\nContent-Type: text/plain; charset="us-ascii"\n\nTest=20franz=F6sisches=20Email'

The message.get_charset() call at https://github.com/repoze/repoze.sendmail/blob/master/repoze/sendmail/encoding.py#L74 returns None (because the charset is None after parsing the message from the file). The fallback on the following lines results in a us-ascii encoding because the message is already encoded.

The result is a message with Content-Type: text/plain; charset="us-ascii" containing iso-8859-1 encoded content.

[tseaver]

@treinhard Thanks for the careful analysis of the problem!

[unikmhz]

Experiencing the same issue.
Couple of differences in my case:

• Original message is multipart/alternative, consisting of one plaintext and one HTML parts.
• Both have charset="utf-8" appended to content-type.
• Sending is done via SMTPMailer+QueueProcessor directly, instead of using qp utility.
• Resulting delivered mail had charset="us-ascii" in content-type for both parts.
• When not using queued delivery (via pyramid_mailer's .send() method), headers remain unaltered.

[WuShell]

I've hit this same problem too.

Looking at pending pull requests for the project, found a couple of them that seem related to this issue:

#26 (specially this one seems good)

#25

But both seem to be there for a long, long time without reply :-(

[WuShell]

I've made a PR that fixes this bug, here: #43 @treinhard, @unikmhz I'd be glad if you can give it a try and confirm if it works for you (thanks!)