Resp

Resp is a library for the MiniScript programming language that implements RESP serialization and deserialization.

The library can be used to exchange data with Redis or a Redis compatible database (if you find a way to connect to one from MiniScript), or alternatively it can be just used on its own for the serialization sake.

Your platform should have RawData class compatible with the one in Mini Micro.

Example

import "resp"

x = {"foo": 42, "bar": [null, 4.3]}

r = resp.dump(x)
print r.utf8
// prints:
//  %2
//  +bar
//  *2
//  _
//  ,4.3
//  +foo
//  :42

x2 = resp.load(r)
print x2                // prints: {"bar": [null, 4.3], "foo": 42}

Example 2. Talking to a Keydb server using unix domain sockets.

The MiniScript interpreter for this example is patched to support unix domain sockets via uds module.
keydb.conf contains this line: unixsocket /tmp/keydb.sock

import "resp"

connection = uds.connect("/tmp/keydb.sock")

connection.send resp.command("SET k hello")

rep = connection.receive

print resp.load(rep)  // prints: OK

connection.send resp.command("GET k")

rep = connection.receive

print resp.load(rep).utf8  // prints: hello

Install

You only need this file: lib/resp.ms.

Overview

In very simple cases, the load() and dump() functions (and sometimes loadMany()) should suffice (see High level API).

Since there is no one-to-one correspondence between MiniScript and RESP data types, the default conversion between them will lose info. There are three ways to overcome this:

Use wrapper classes to manipulate RESP types directly (see Wrappers).
Provide converter callbacks for the load() and dump() functions (see Writing converter callbacks).
Define ._toRESPWrp() for your classes (see _toRESPWrp() callback).

Serialization and deserialization problems (like corrupted data, reference cycles etc) will result in qa.abort() beeing called (they crash). To avoid the crashes, pass onError callback to the load() / dump() (see Writing onError callbacks).

If you're writing a stream-oriented code, there's a middle-level Loader class that is capable of consuming RESP from streams of RawData chunks (see Loader).

A low-level RawDataCollection class is a scary tree data type that is not meant to be used directly and is there to speed things up behind the scenes (at least I believe it should).

Finally, there is a couple of helper functions (see Helpers):

str() can replace the intrinsic str function and it calls ._str() method of objects (good for debugging).
stringToRawData() converts strings to RawData objects.
command() produces Redis-compatible requests.

High level API

load()

load(r, onError = null, wrptov = null) -> value

Deserializes RESP into a MiniScript value. The r param can be a string, a RawData object or a list of RawData objects.

Deserialization conversions:

Blob types (including streamed strings) become RawData.
Null type becomes null.
Simple string and simple error are converted to string.
Numeric and boolean types are converted to number.
Array and "push" types are converted to list.
Map type is returned as map.
"Set" type becomes a map of {elem1: true, elem2: true, elem3: true, ...}.
Any attribute objects are dropped.

loadMany()

loadMany(r, onError = null, wrptov = null) -> list of values

Deserializes RESP into a list of MiniScript values. The r param can be a string, a RawData object or a list of RawData objects.

If r contains several encoded values one after another, they all get deserialized and returned as a list.

If r doesn't contain any whole value, an empty list is returned.

Same default conversions are applied as for load().

dump()

dump(v, onError = null, vtowrp = null) -> RawData

Serializes a MiniScript value into RESP. Returns a RawData object.

(There are more functions that do the same: dumpToList() returns a list of RawData objects that being put together make the same RESP, and dumpToString() returns a string.)

Serialization conversions:

null is serialized as null type.
Integer numbers are serialized as number type.
Real numbers are serialized as double type.
Strings without <CR><LF> are serialized as simple strings.
Strings with <CR><LF> and RawData objects are serialized as blob strings.
Lists become array type.
Maps become map type.
If a map has _toRESPWrp() method, it is called and the result is used to produce RESP (see _toRESPWrp() callback).

Some values will not serialize: functions, unknown types (e.g handles) and maps with __isa (if they don't have _toRESPWrp()).

Wrappers

The high-level API doesn't preserve RESP types and attributes.

You can use wrapper classes (the descendants of Wrp class) to manipulate RESP types directly:

RESP type	Wrapper class	General type category
`$` blob string	`BlobStringWrp`	blob
`!` blob error	`BlobErrorWrp`	blob
`=` verbatim string	`VerbatimStringWrp`	blob
`+` simple string	`SimpleStringWrp`	line
`-` simple error	`SimpleErrorWrp`	line
`_` null	`NullWrp`	line
`:` number	`NumberWrp`	line (numeric)
`,` double	`DoubleWrp`	line (numeric)
`#` boolean	`BooleanWrp`	line (numeric)
`(` big number	`BigNumberWrp`	line (numeric)
`*` array	`ArrayWrp`	aggregate (list-like)
`~` set	`SetWrp`	aggregate (list-like)
`>` push	`PushWrp`	aggregate (list-like)
`%` map	`MapWrp`	aggregate (map-like)
`\|` attribute	`AttributeWrp`	aggregate (map-like)

Additional wrappers to support streamed strings:

RESP type	Wrapper class	General type category
`$?` streamed string	`StreamedStringWrp`	aggregate (list-like)
`;` blob chunk	`BlobChunkWrp`	blob

There are several ways to acquire a wrapper object in code:

Use a class method Wrp.fromRESP() (instead of a load() function) to deserialize it from RESP.
Use a class method Wrp.manyFromRESP() (instead of a loadMany() function) to deserialize a list of wrappers from RESP.
Use a class method Wrp.fromValue() to convert it from a MiniScript value using the default conversion rules.
Construct a simple type using a <class>.fromData() factory.
Construct an aggregate type using a <class>.make() factory and then add elements to it with its pushValue() or pushKeyValue() methods.
Use loader.getWrp().

When you've acquired a warapper, you can:

Use toRESP() / toRESPList() / toRESPString() methods to produce its RESP representation.
Use toValue() to convert it to a MiniScript value using the default conversion rules.
Extract the underlying data from simple types with their toRawData() / toString() / toNumber() methods.
Access elements of an aggregate type through its elements property.

All wrappers can have an optional RESP "attribute" (an instance of AttributeWrp class). You can get/set it through attribute / setAttribute().

Wrp.fromRESP()

Wrp.fromRESP(r, onError = null) -> wrapper

(class method) Deserializes RESP from a string, a RawData object or a list of RawData objects into a wrapper object.

Wrp.manyFromRESP()

Wrp.manyFromRESP(r, onError = null) -> list of wrappers

(class method) Deserializes RESP from a string, a RawData object or a list of RawData objects into a list of wrapper objects.

If r contains several encoded values one after another, they all get deserialized into wrappers and returned as a list.

If r doesn't contain any whole value, an empty list is returned.

<wrapper>.toRESP()

wrapper.toRESP() -> RawData

Serializes a wrapper object into a RawData containing RESP.

(There are also wrapper.toRESPList() analogous to dumpToList() and wrapper.toRESPString() analogous to dumpToString().)

Wrp.fromValue()

Wrp.fromValue(v, onError = null, vtowrp = null) -> wrapper

(class method) Creates a wrapper from a MiniScript value using the default conversion.

<wrapper>.toValue()

wrapper.toValue(wrptov = null) -> value

Creates a MiniScript value from a wrapper using the default conversion.

<blob or line class>.fromData()

<blob or line class>.fromData(d) -> wrapper

(class method) Create a wrapper with data d as its content.

import "resp"

w = resp.BigNumberWrp.fromData(42)
print w.toRESPString  // prints: (42<CR><LF>

w = resp.BlobErrorWrp.fromData("we're all doomed")
print w.toRESPString
// prints:
//  !16<CR><LF>
//  we're all doomed<CR><LF>

<aggregate class>.make()

<aggregate class>.make(isStreamed = false, hasHead = false, hasTail = false) -> wrapper

(class method) Create an empty aggregate type wrapper.

This method of StreamedStringWrp class doesn't have isStreamed parameter: StreamedStringWrp.make(hasHead = false, hasTail = false).

<list-like wrapper>.pushValue()

<list-like wrapper>.pushValue(v)

Adds an element to an list-like aggregate type wrapper (ArrayWrp, SetWrp, PushWrp or StreamedStringWrp).

v is expected to be a wrapper object.

In case of streamed strings, v can only be a BlobChunkWrp object.

import "resp"

w = resp.SetWrp.make
w.pushValue resp.NumberWrp.fromData(100)
w.pushValue resp.NumberWrp.fromData(200)
print w.toRESPString
// prints:
//  ~2<CR><LF>
//  :100<CR><LF>
//  :200<CR><LF>

<map-like wrapper>.pushKeyValue()

<map-like wrapper>.pushKeyValue(k, v)

Adds a pair to a map-like aggregate type wrapper (MapWrp or AttributeWrp).

k and v are both expected to be wrapper objects.

import "resp"

w = resp.MapWrp.make
w.pushKeyValue resp.SimpleStringWrp.fromData("foo"),
               resp.SimpleStringWrp.fromData("bar")
print w.toRESPString
// prints:
//  %1<CR><LF>
//  +foo<CR><LF>
//  +bar<CR><LF>

<blob or line wrapper>.toRawData()

<blob or line wrapper>.toRawData -> RawData

Extracts the content part of a blob or line type wrapper. Returns RawData.

<line wrapper>.toString()

<line wrapper>.toString -> string

Extracts the content part of a line type wrapper and returns it as string.

<numeric wrapper>.toNumber()

<numeric wrapper>.toNumber -> number

Extracts the content part of a numeric type wrapper and returns it as number.

<aggregate wrapper>.elements

This property is a list of previously pushed elements.

<wrapper>.attribute

This property is either null or an assigned AttributeWrp object.

<wrapper>.setAttribute()

<wrapper>.setAttribute(attribute)

Assigns an attribute property to the wrapper.

import "resp"

attr = resp.AttributeWrp.make
attr.pushKeyValue resp.SimpleStringWrp.fromData("attr1"),
                  resp.DoubleWrp.fromData("-1.23")

w = resp.BooleanWrp.fromData(true)
w.setAttribute attr
print w.toRESPString
// prints:
//  |1<CR><LF>
//  +attr1<CR><LF>
//  ,-1.23<CR><LF>
//  #t<CR><LF>

Writing `onError` callbacks

If a serialization or deserialization problem happens, load() and dump() functions will crash. To avoid uncaught errors, you can supply an onError callback.

onError(errCode, arg1, arg2, offset) -> ...

The errCode value denotes the problem (or event).

The meaning of arg1 and arg2 may differ for each errCode.

When deserializing, the offset is where in the subject the problem was encountered.

The return value of the callback becomes the return value of the caller.

D/S	errCode	arg1	arg2	Meaning
S	`"FROM_FUNC"`			Unable to serialize: the value is a function
S	`"FROM_CYCLES"`	value		Unable to serialize: the value has reference cycles
S	`"FROM_BAD_CALLBACK"`	result of `_toRESPWrp()`		Unable to serialize: `_toRESPWrp()` returned a value that is not a wrapper
S	`"FROM_ARB_INSTANCE"`	value		Unable to serialize: the value has `__isa`
S	`"FROM_ARB_TYPE"`	value		Unable to serialize: the value has unknown type
D	`"UNKNOWN_TYPE"`	type character	type character code	Unable to deserialize: value of unknown type
D	`"BAD_ELEM_TYPE"`	aggregate type character	element type character	Unable to deserialize: wrong type of an element in an aggregate
D	`"EMPTY_LENGTH"`			Unable to deserialize: Empty string instead of the value length
D	`"BAD_CHUNK"`	chunk length		Unable to deserialize: no `<CR><LF>` after a blob
D	`"MAX_DEPTH_EXCEEDED"`			Unable to deserialize a very deeply nested value
D	`"MAX_ELEMENTS_EXCEEDED"`			Unable to deserialize a very long aggregate
D	`"MAX_BLOB_LENGTH_EXCEEDED"`			Unable to deserialize a very long blob value
D	`"MAX_LINE_LENGTH_EXCEEDED"`			Unable to deserialize a very long line value
D	`"STREAM_STARTED"`	aggregate		(not an error) A start marker for a steramed aggregate type is read
D	`"STREAM_ELEMENT"`	aggregate	element	(not an error) An element in the current steramed aggregate is read
D	`"STREAM_STOPPED"`	aggregate		(not an error) An end marker for the current steramed aggregate is read

import "resp"

onError = function(errCode, arg1, arg2, offset)
	print "problem = " + errCode
end function

v = resp.load("@foo", @onError)  // prints: problem = UNKNOWN_TYPE

r = resp.dump(@str, @onError)  // prints: problem = FROM_FUNC

Writing converter callbacks

Functions load(), loadMany(), dump(), Wrp.toValue() and Wrp.fromValue() accept optional callback functions that can alter the types conversion process.

load(), loadMany() and Wrp.toValue() accept a wrptov() callback.
dump() and Wrp.fromValue() accept a vtowrp() callback.

(The actual function names are not important, but their signature and semantics differ.)

wrptov() // wrapper-to-value

wrptov(wrapper) -> value

Converts a wrapper object into a MiniScript value.

If the callback returns null, the default type conversion takes place.

import "resp"

wrptov = function(wrp)
	if wrp isa resp.NumericValueWrp then return "( " + wrp.toString + " )"
end function

v = resp.load("*2" + char(13) + char(10) +
              ":42" + char(13) + char(10) +
              "+foo" + char(13) + char(10), null, @wrptov)
print v  // prints: ["( 42 )", "foo"]

vtowrp() // value-to-wrapper

vtowrp(value) -> wrapper

Converts a MiniScript value into a wrapper object.

If the callback returns null, the default type conversion takes place.

import "resp"

vtowrp = function(v)
	if hasIndex(v, "foo") then return resp.SimpleStringWrp.fromData("( " + v.foo + " )")
end function

r = resp.dump([{"foo": 42}, {"bar": 43}], null, @vtowrp)
print r.utf8
// prints:
//  *2<CR><LF>
//  +( 42 )<CR><LF>
//  %1<CR><LF>
//  +bar<CR><LF>
//  :43<CR><LF>

`_toRESPWrp()` callback

Arbitrary maps with __isa properties don't serialize to RESP. However, if such objects have _toRESPWrp() method, it is called and its result is used in conversion.

import "resp"

A = {}
A._toRESPWrp = function
	return resp.SimpleStringWrp.fromData("( A )")
end function

r = resp.dump([new A, new A])
print r.utf8
// prints:
//  *2<CR><LF>
//  +( A )<CR><LF>
//  +( A )<CR><LF>

Loader

Loader is a class that consumes a stream of RawData objects and produces a stream of wrappers.

To create a loader use the make() factory.

Call pushData() and getWrp() to add RawData and read wrappers respectively.

Loader.make()

Loader.make -> loader

(class method)

<loader>.pushData()

<loader>.pushData(r, onError = null)

Adds a RawData chunk to the internal RawData collection.

<loader>.getWrp()

<loader>.getWrp() -> wrapper or null

Returns a wrapper object, or null if the internal RawData collection doesn't contain enough data for a whole value.

import "resp"

l = resp.Loader.make

l.pushData "+foo"
l.pushData char(13) + char(10)
l.pushData "+bar" + char(13) + char(10) + "+baz"

print l.getWrp.toValue  // prints: foo
print l.getWrp.toValue  // prints: bar
print l.getWrp == null  // prints: 1

Helpers

str()

str(x, depth = null) -> string

Replacement for built-in str() that calls ._str if available.

stringToRawData()

stringToRawData(s) -> RawData

Returns a RawData object containing data from the string.

command()

command(parts) -> RawData

Encodes a message containing a Redis command.

import "resp"

cmd = resp.command(["SET", "k", "foo"])
print cmd.utf8
// prints:
//  *3<CR><LF>
//  $3<CR><LF>
//  SET<CR><LF>
//  $1<CR><LF>
//  k<CR><LF>
//  $3<CR><LF>
//  foo<CR><LF>

cmd = resp.command("GET k")
print cmd.utf8
// prints:
//  *2<CR><LF>
//  $3<CR><LF>
//  GET<CR><LF>
//  $1<CR><LF>
//  k<CR><LF>

Changes

ver 0.2.0

new functions: loadMany() and Wrp.manyFromRESP()
Loader.push() renamed to Loader.pushData()
.push() of list-like wrappers renamed to .pushValue()
.push(v) of map-like wrappers renamed to .pushKeyValue(k, v) (and changed signature)
parsing is now done in Loader.pushData() (formerly known as .push()) and not in .getWrp(), so onError callback parameter also moved there.
deleted error codes NOT_ENOUGH_DATA and MORE_DATA
added limits Loader.maxDepth, .maxElements, .maxBlobLength and .maxLineLength and error codes MAX_DEPTH_EXCEEDED, MAX_ELEMENTS_EXCEEDED, MAX_BLOB_LENGTH_EXCEEDED and MAX_LINE_LENGTH_EXCEEDED
deleted offset parameter of load() and Wrp.fromRESP()

Files

README.md

Latest commit

History