sending pure binary over datauri

[mofosyne]

This is just a brainstorm...

ASCII being 7bit is not binary safe. base64 is not compact. Hex is not compact.

maybe use ;charset=UTF-8 to send binary data safely? Even if it looks like UTF-8 gibberish, since it is UTF-8, it is 8bit, thus 8bit binary content is safe.

Or should we use a specialized token instead?

• http://dba.stackexchange.com/questions/61696/is-there-any-danger-using-binary-data-over-an-utf-8-mysql-connection

[xloem]

The charset you want is ISO-8859-1 . It preserves binary data exactly.
Search for all instances of "ISO-8859-1" at https://github.com/xloem/qrstream/blob/master/android/app/src/main/java/com/github/xloem/qrstream/Receive.java

[mofosyne]

Sweet Thanks. Btw, I'm a bit curious as to how you transmit your files. Do you just transmit binary? Or is there a manifest you also send?

[xloem]

For now QRStream just dumps the file content straight into the QR codes, in order, in ISO-8859-1. So if you're receiving a text file and do not have qr stream, you can read into the clipboard and paste the results one after the other into a text file and have back the exact same file. Similarly you can generate codes from the commandline with split. But the user needs to read them exactly in order and know what to do with the content already.

[mofosyne]

I see, so it's not fully autonomous, in the sense of telling the user what
media/type or filename it has. Right, well that's the next challenge to
tackle then.

But I like your approach of writing binary stream straight to a file. But
it does mean that you have to scan exactly in the correct order... again
not an issue in your use case. (But would be somewhat an issue for mine if
I want to implement something like a psudo-structured append system.)

I thought about mixing binary data into a datauri container for space
saving, and for potential compression like "lzma". But now that I thought
about it. It's probably a bad idea to make datauri parsing more complicated
than it needs to be.

So for both tagdrop and qrstream, we need a binary transmission system that
supports these feature at minimum:

• compression definition
• some form of structured append
  • Either by having a manifest QR code holding a sequence of md5 hash...
    but that would be a pretty big QR code.
  • A header on each code saying "1of3" in either text (or more likely in
    binary)
• mediatype & character encoding
• filename (extension should be derived from media-type)

A possible approach is to look at the NDEF standard, and see if that can be
adapted (since NDEF doesn't support structured append, filename, or
compression).

[xloem]

I consider ordering an issue but I think it is best solved by submitting a proper patch to zxing that implements structured append more fully. The QR code format already supports structured append & character encoding information.

But since some clients would be stuck in the situation of being unable to access structured append information from the codes they read (or unable to generate it with the library they use), it would be great to support them with a general format as you say.

Here are some ideas for manifest formats:

• MIME "Key: value\n"
• URI such as "qrencoded://name?key=value&key=value"
• json "{key:value,key:value}"
• Compact binary format (e.g. 1 byte key, key decides how next data is processed, repeat)

md5 hash sequence can maybe be shrunk to fit in the QR code by hashing larger chunks of the data. Additionally if a user is sending 100 codes they are less likely to care if there is more manifest data on some small number of extra codes.

Other links: https://en.wikipedia.org/wiki/YEnc https://en.wikipedia.org/wiki/Parity_file