Marshal and Unmarshal Binary Formats in Go

The Go standard library features packages for converting between structs and various message and file formats. Packages encoding/json and encoding/xml are well known for providing convenient functions Marshal() and Unmarshal() that leverage the declarative nature of struct tags. These functions are however missing from encoding/binary, leaving developers without an accessible and intuitive way to work with binary formats.

This module supplies the ubiquitous Marshal() and Unmarshal() function signatures so that developers can define custom binary formats using only struct tags and avoid custom bit manipulation.

User Story

As a Go developer implementing a binary message or file format,
I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
that convert a struct into a series of bits in a byte slice and vice versa,
so that I can avoid the complexities of custom bit manipulation.

See the rest of the behaviour-driven specifications of this module below.

Binary Formats

Message and file formats specify how bits are arranged to encode information. Control over individual bits or groups smaller than a byte is often required to put together and take apart these binary structures.

Message Format Examples

Describing the anatomy of TCP/IP headers at the beginning of every internet datagram ("packet") are

• Section 3.1 of RFC 791 Internet Protocol, and
• Section 3.1 of RFC 793 Transmission Control Protocol.

File Format Examples

Binary file formats are not significantly different from message formats from an application developer's perspective. RFC 1952 describes the GZIP File Format Specification.

Working with Binary Formats in Go

The smallest data structures Go provides are the basic type byte (alias of uint8, an unsigned 8-bit integer), and bool, both eight bits long. To manipulate data at a scale smaller than eight bits would require the use of bitwise logical and shift operators such as AND (&), OR (|), left shift (<<), and right shift (>>).

Relevant Questions Posted on StackOverflow

Suggestions on StackOverflow are limited to the use of bitwise operators.

• Golang: Parse bit values from a byte
• Creating 8 bit binary data from 4,3, and 1 bit data in Golang
• How to pack the C bit field struct via encoding package in GO?

Behaviour-Driven Specifications

Feature: Marshal and Unmarshal

    As a Go developer implementing a binary message or file format,
    I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
    that convert a struct into a series of bits in a byte slice and vice versa,
    so that I can avoid the complexities of custom bit manipulation.

    Background:
        # Ubiquitous language
        Given a message or file "format"
            """
            A format specifies how bits are arranged to encode information.
            """
        And the format is a series of "bit fields"
            """
            A bit field is one or more adjacent bits representing a value,
            and should not be confused with struct fields.
            """
        And adjacent bit fields are grouped into "words"
            """
            A word is a series of bits that can be simultaneously processed
            by a given computer architecture and programming language.
            """

        # Define format-structs
        And a format is represented by a type definition of a "format-struct"
        And the format-struct nests one or more exported "word-structs"
        And the words are tagged to indicate their lengths in number of bits

            type RFC791InternetHeaderFormatWithoutOptions struct {
                RFC791InternetHeaderFormatWord0 `word:"32"`
                RFC791InternetHeaderFormatWord1 `word:"32"`
                RFC791InternetHeaderFormatWord2 `word:"32"`
                RFC791InternetHeaderFormatWord3 `word:"32"`
                RFC791InternetHeaderFormatWord4 `word:"32"`
            }

        And the length of each word is a multiple of eight in the range [8, 64]

        # Define word-structs
        And each word-struct has exported field(s) corresponding to bit field(s)
        And the fields are of unsigned integer or boolean types
        And the fields are tagged to indicate the lengths of those bit fields

            type RFC791InternetHeaderFormatWord0 struct {
                Version     uint8  `bitfield:"4"`
                IHL         uint8  `bitfield:"4"`
                Precedence  uint8  `bitfield:"3"`
                Delay       bool   `bitfield:"1"`
                Throughput  bool   `bitfield:"1"`
                Reliability bool   `bitfield:"1"`
                Reserved    uint8  `bitfield:"2"`
                TotalLength uint16 `bitfield:"16"`
            }

        And the length of each bit field does not overflow the type of the field
            """
            A bit field overflows a type
            when it is long enough to represent values
            outside the set of values of the type.
            """
        And the sum of lengths of all fields is equal to the length of that word

Marshal

    Scenario: Marshal a struct into a byte slice
        Given a format-struct variable representing a binary message or file

            internetHeader = RFC791InternetHeaderFormatWithoutOptions{
                RFC791InternetHeaderFormatWord0{
                    Version: 4,
                    IHL:     5,
                    // ...
                },
                // ...
            }

        And the struct field values do not overflow corresponding bit fields
            """
            A struct field value overflows its corresponding bit field
            when it falls outside the range of values
            that can be represented by that bit field given its length.
            """
        When I pass to function Marshal() a pointer to that struct variable

            var (
                bytes []byte
                e     error
            )

            bytes, e = binary.Marshal(&internetHeader)

        Then Marshal() should return a slice of bytes and a nil error
        And I should see struct field values reflected as bits in those bytes

            log.Printf("%08b", bytes)
            // [01000101 ...]

            log.Println(e == nil)
            // true

        And I should see that the lengths of the slice and the format are equal
            """
            The length of a format is the sum of lengths of the words in it.
            The length of a word is the sum of lengths of the bit fields in it.
            """

Unmarshal

    Scenario: Unmarshal a byte slice into a struct
        Given a format-struct type representing a binary message or file format

            var internetHeader RFC791InternetHeaderFormatWithoutOptions

        And a slice of bytes containing a binary message or file

            var bytes []byte

            // ...

            log.Printf("%08b", bytes)
            // [01000101 ...]

        And the lengths of the slice and the format (measured in bits) are equal
        When I pass to function Unmarshal() the slice of bytes as an argument
        And I pass to the function a pointer to the struct as a second argument

            e = binary.Unmarshal(bytes, &internetHeader)

        Then Unmarshal() should return a nil error
        And I should see struct field values matching the bits in those bytes

            log.Println(e == nil)
            // true

            log.Println(internetHeader.RFC791InternetHeaderFormatWord0.Version)
            // 4

            log.Println(internetHeader.RFC791InternetHeaderFormatWord0.IHL)
            // 5

Performance and Optimisation

This module is optimised for performance.

$ go test -cpuprofile cpu.prof -memprofile mem.prof -bench . -benchmem

goos: linux
goarch: arm64
pkg: github.com/encodingx/binary
BenchmarkMarshal-2     	 3435181	       349.9 ns/op	      64 B/op	       6 allocs/op
BenchmarkUnmarshal-2   	 3004425	       396.4 ns/op	      64 B/op	       8 allocs/op
PASS
ok  	github.com/encodingx/binary	3.288s