[work-in-progress] Decoding BufReader implementation

[dralley]

Very, very work-in-progress. Doesn't compile yet. Only posting it for transparency.

[codecov-commenter]

Codecov Report

Merging #441 (10038ca) into master (90be3ee) will increase coverage by 0.69%.
The diff coverage is 83.63%.

@@            Coverage Diff             @@
##           master     #441      +/-   ##
==========================================
+ Coverage   52.12%   52.81%   +0.69%     
==========================================
  Files          28       28              
  Lines       13480    13392      -88     
==========================================
+ Hits         7026     7073      +47     
+ Misses       6454     6319     -135     

Flag	Coverage Δ
unittests	52.81% <83.63%> (+0.69%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
benches/microbenches.rs	0.00% <0.00%> (ø)
examples/custom_entities.rs	0.00% <0.00%> (ø)
examples/nested_readers.rs	0.00% <0.00%> (ø)
examples/read_buffered.rs	0.00% <0.00%> (ø)
examples/read_texts.rs	0.00% <0.00%> (ø)
src/errors.rs	2.08% <0.00%> (+0.01%)	⬆️
src/reader/buffered_reader.rs	69.89% <0.00%> (-6.25%)	⬇️
src/writer.rs	48.30% <0.00%> (-0.28%)	⬇️
src/de/var.rs	75.00% <11.11%> (-10.11%)	⬇️
src/de/escape.rs	19.84% <40.00%> (-1.21%)	⬇️
... and 26 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[Mingun]

I like the idea to create a separate test file for encodings, so the commit 2 is good for me. The others lets hold until your finish your work. I'm not sure, that we should remove StartText event, because it still can be useful even if will not contain BOM. The text before the declaration is not considered as a part of XML model, so it should be treated differently, for example, the unescaping of that text has no meaning.

I suggest firstly to add tests for all possible encodings -- because the set of encodings is finite and not so big, we can create a bunch of documents in each encoding and test their parsing. The tests that I would like to see:

• detecting encoding (I found that we do not do that correctly, I'll plan to look at this before release, I already made some investigations and improvements, but not yet proved them with tests)
• reading events

The XMLs should looks like:

<?xml encoding="<encoding>"?>
<root attribute1=value1
      attribute2="value1"
      attribute3='value2'
>
  <?instruction?>
  <!--Comment-->
  text
  <ns:element ns:attribute="value1" xmlns:ns="namespace"/>
  <[[CDATA[[cdata]]>
</root>

Because almost all supported encodings are one-byte encodings, we could create XML files with all possible symbols (probably except that explicitly forbidden by the XML standard). The national symbols should be presented in all possible places (tag names, namespace names, attribute names and values, text and CDATA content, comments, processing instructions).

Only after that, I believe, we should made any improvements in encoding support.

[dralley]

The basic functionality is pretty much working. TODO:

• encoding detection regression for UTF-16 (failing tests)
• async support in Utf8BytesReader
• UTF-8 validation inside Utf8BytesReader when the encoding feature is turned off
• some other cleanup

Async support is the real pain unfortunately.

[dralley]

I'm thinking we really need a better solution than this. Either by tracking newlines during parsing or by automatically implementing something like this for &str and perhaps File.

[dralley]

I'm not sure about the overhead of the former approach, might be too expensive.

[Mingun]

Left some preliminary comments

[Mingun]

Yes, I think we could make this default.

[Mingun]

This test is intentionally uses from_reader (which its name say). Changing that you make it the same as the test just below it. You should either delete this change or the entire test

[dralley]

Then it needs to be removed, because this variant of this method cannot be implemented for SliceReader.

[dralley]

Status update: I'm still working on this, but making PRs against https://github.com/BurntSushi/encoding_rs_io/

[dralley]

And also hsivonen/encoding_rs#88

I'm not sure how long I'm going to be blocked on that, so if there is a portion of the work you think could be split off and merged, I'd appreciate that. Otherwise I'll probably work on the attribute normalization PR.

[dralley]

So, as a practical matter I now think we need to strip the BOM in all situations, no matter what.

Not because there's anything invalid about it at a purely technical level, but because buried deep down on page 157 out of 1054 of the Unicode standard it says that for UTF-16 the BOM "should not be considered part of the text". It actually says nothing about that for UTF-8 and in fact implies the opposite, but that line appears to have been the basis for encoding_rs to remove the BOM from the stream for both UTF-16 and UTF-8.

Realistically I don't think we should cut against the grain of what encoding_rs is doing, it's probably a sane decision for consistency purposes anyway, and we need to get the same output in both the encoding and not-encoding cases - so that means the BOM should never be present.

[Mingun]

Do what you think right, just explain decisions in the comments in the code

[dralley]

@Mingun I'm just going to rip out any changes to the serde code and do that absolutely last, so I don't need to keep redoing it

It's been a month without any further progress on getting BurntSushi/encoding_rs_io#11 merged, so I'm just going to vendor the code internally for now. It's not that much code and he was very skeptical about merging async support anyway, which we're kind of obligated to support now.