diff --git a/build.sh b/build.sh
index 087e24b..b765c83 100755
--- a/build.sh
+++ b/build.sh
@@ -1,7 +1,7 @@
#!/bin/bash
set -e
-VERSION="0.2.2"
+VERSION="0.2.3"
PROTECTED_MODE="no"
export GO15VENDOREXPERIMENT=1
diff --git a/vendor/github.com/tidwall/gjson/README.md b/vendor/github.com/tidwall/gjson/README.md
index 1aa0765..79f3ec4 100644
--- a/vendor/github.com/tidwall/gjson/README.md
+++ b/vendor/github.com/tidwall/gjson/README.md
@@ -3,15 +3,14 @@
src="logo.png"
width="240" height="78" border="0" alt="GJSON">
-
+
get a json value quickly
-GJSON is a Go package the provides a [very fast](#performance) and simple way to get a value from a json document. The reason for this library it to give efficient json indexing for the [BuntDB](https://github.com/tidwall/buntdb) project. +GJSON is a Go package that provides a [fast](#performance) and [simple](#get-a-value) way to get values from a json document. +It has features such as [one line retrieval](#get-a-value), [dot notation paths](#path-syntax), [iteration](#iterate-through-an-object-or-array). Getting Started =============== @@ -27,7 +26,7 @@ $ go get -u github.com/tidwall/gjson This will retrieve the library. ## Get a value -Get searches json for the specified path. A path is in dot syntax, such as "name.last" or "age". This function expects that the json is well-formed and validates. Invalid json will not panic, but it may return back unexpected results. When the value is found it's returned immediately. +Get searches json for the specified path. A path is in dot syntax, such as "name.last" or "age". This function expects that the json is well-formed. Bad json will not panic, but it may return back unexpected results. When the value is found it's returned immediately. ```go package main @@ -47,6 +46,7 @@ This will print: ``` Prichard ``` +*There's also the [GetMany](#get-multiple-values-at-once) function to get multiple values at once, and [GetBytes](#working-with-bytes) for working with JSON byte slices.* ## Path Syntax @@ -54,7 +54,7 @@ A path is a series of keys separated by a dot. A key may contain special wildcard characters '\*' and '?'. To access an array value use the index as the key. To get the number of elements in an array or to access a child path, use the '#' character. -The dot and wildcard characters can be escaped with '\'. +The dot and wildcard characters can be escaped with '\\'. ```json { @@ -63,25 +63,33 @@ The dot and wildcard characters can be escaped with '\'. "children": ["Sara","Alex","Jack"], "fav.movie": "Deer Hunter", "friends": [ - {"first": "James", "last": "Murphy"}, - {"first": "Roger", "last": "Craig"} + {"first": "Dale", "last": "Murphy", "age": 44}, + {"first": "Roger", "last": "Craig", "age": 68}, + {"first": "Jane", "last": "Murphy", "age": 47} ] } ``` ``` "name.last" >> "Anderson" "age" >> 37 +"children" >> ["Sara","Alex","Jack"] "children.#" >> 3 "children.1" >> "Alex" "child*.2" >> "Jack" "c?ildren.0" >> "Sara" "fav\.movie" >> "Deer Hunter" -"friends.#.first" >> [ "James", "Roger" ] +"friends.#.first" >> ["Dale","Roger","Jane"] "friends.1.last" >> "Craig" ``` -To query an array: + +You can also query an array for the first match by using `#[...]`, or find all matches with `#[...]#`. +Queries support the `==`, `!=`, `<`, `<=`, `>`, `>=` comparison operators and the simple pattern matching `%` operator. + ``` -`friends.#[last="Murphy"].first` >> "James" +friends.#[last=="Murphy"].first >> "Dale" +friends.#[last=="Murphy"]#.first >> ["Dale","Jane"] +friends.#[age>45]#.last >> ["Craig","Murphy"] +friends.#[first%"D*"].last >> "Murphy" ``` ## Result Type @@ -105,25 +113,33 @@ result.Type // can be String, Number, True, False, Null, or JSON result.Str // holds the string result.Num // holds the float64 number result.Raw // holds the raw json -result.Multi // holds nested array values result.Index // index of raw value in original json, zero means index unknown ``` There are a variety of handy functions that work on a result: ```go +result.Exists() bool result.Value() interface{} result.Int() int64 +result.Uint() uint64 result.Float() float64 result.String() string result.Bool() bool +result.Time() time.Time result.Array() []gjson.Result result.Map() map[string]gjson.Result result.Get(path string) Result +result.ForEach(iterator func(key, value Result) bool) +result.Less(token Result, caseSensitive bool) bool ``` The `result.Value()` function returns an `interface{}` which requires type assertion and is one of the following Go types: +The `result.Array()` function returns back an array of values. +If the result represents a non-existent value, then an empty array will be returned. +If the result is not a JSON array, the return value will be an array containing one result. + ```go boolean >> bool number >> float64 @@ -151,14 +167,14 @@ Suppose you want all the last names from the following json: "lastName": "Harold", } ] -}` +} ``` You would use the path "programmers.#.lastName" like such: ```go result := gjson.Get(json, "programmers.#.lastName") -for _,name := range result.Array() { +for _, name := range result.Array() { println(name.String()) } ``` @@ -170,6 +186,20 @@ name := gjson.Get(json, `programmers.#[lastName="Hunter"].firstName`) println(name.String()) // prints "Elliotte" ``` +## Iterate through an object or array + +The `ForEach` function allows for quickly iterating through an object or array. +The key and value are passed to the iterator function for objects. +Only the value is passed for arrays. +Returning `false` from an iterator will stop iteration. + +```go +result := gjson.Get(json, "programmers") +result.ForEach(func(key, value gjson.Result) bool { + println(value.String()) + return true // keep iterating +}) +``` ## Simple Parse and Get @@ -185,7 +215,7 @@ gjson.Get(json, "name.last") ## Check for the existence of a value -Sometimes you just want to know you if a value exists. +Sometimes you just want to know if a value exists. ```go value := gjson.Get(json, "name.last") @@ -196,38 +226,132 @@ if !value.Exists() { } // Or as one step -if gjson.Get(json, "name.last").Exists(){ +if gjson.Get(json, "name.last").Exists() { println("has a last name") } ``` +## Unmarshalling + +There's a `gjson.Unmarshal` function which loads json data into a value. +It's a general replacement for `json.Unmarshal` and you can typically +see a 2-3x boost in performance without the need for external generators. + +This function works almost identically to `json.Unmarshal` except that +`gjson.Unmarshal` will automatically attempt to convert JSON values to any +Go type. For example, the JSON string "100" or the JSON number 100 can be +equally assigned to Go string, int, byte, uint64, etc. This rule applies to +all types. + + +```go +package main + +import ( + "fmt" + + "github.com/tidwall/gjson" +) + +type Animal struct { + Type string `json:"type"` + Sound string `json:"sound"` + Age int `json:"age"` +} + +var json = `{ + "type": "Dog", + "Sound": "Bark", + "Age": "11" +}` + +func main() { + var dog Animal + gjson.Unmarshal([]byte(json), &dog) + fmt.Printf("type: %s, sound: %s, age: %d\n", dog.Type, dog.Sound, dog.Age) +} +``` + +This will print: + +``` +type: Dog, sound: Bark, age: 11 +``` + ## Unmarshal to a map To unmarshal to a `map[string]interface{}`: ```go m, ok := gjson.Parse(json).Value().(map[string]interface{}) -if !ok{ +if !ok { // not a map } ``` +## Working with Bytes + +If your JSON is contained in a `[]byte` slice, there's the [GetBytes](https://godoc.org/github.com/tidwall/gjson#GetBytes) function. This is preferred over `Get(string(data), path)`. + +```go +var json []byte = ... +result := gjson.GetBytes(json, path) +``` + +If you are using the `gjson.GetBytes(json, path)` function and you want to avoid converting `result.Raw` to a `[]byte`, then you can use this pattern: + +```go +var json []byte = ... +result := gjson.GetBytes(json, path) +var raw []byte +if result.Index > 0 { + raw = json[result.Index:result.Index+len(result.Raw)] +} else { + raw = []byte(result.Raw) +} +``` + +This is a best-effort no allocation sub slice of the original json. This method utilizes the `result.Index` field, which is the position of the raw data in the original json. It's possible that the value of `result.Index` equals zero, in which case the `result.Raw` is converted to a `[]byte`. + +## Get multiple values at once + +The `GetMany` function can be used to get multiple values at the same time, and is optimized to scan over a JSON payload once. + +```go +results := gjson.GetMany(json, "name.first", "name.last", "age") +``` + +The return value is a `[]Result`, which will always contain exactly the same number of items as the input paths. + ## Performance Benchmarks of GJSON alongside [encoding/json](https://golang.org/pkg/encoding/json/), [ffjson](https://github.com/pquerna/ffjson), [EasyJSON](https://github.com/mailru/easyjson), -and [jsonparser](https://github.com/buger/jsonparser) +[jsonparser](https://github.com/buger/jsonparser), +and [json-iterator](https://github.com/json-iterator/go) + +``` +BenchmarkGJSONGet-8 3000000 372 ns/op 0 B/op 0 allocs/op +BenchmarkGJSONUnmarshalMap-8 900000 4154 ns/op 1920 B/op 26 allocs/op +BenchmarkJSONUnmarshalMap-8 600000 9019 ns/op 3048 B/op 69 allocs/op +BenchmarkJSONUnmarshalStruct-8 600000 9268 ns/op 1832 B/op 69 allocs/op +BenchmarkJSONDecoder-8 300000 14120 ns/op 4224 B/op 184 allocs/op +BenchmarkFFJSONLexer-8 1500000 3111 ns/op 896 B/op 8 allocs/op +BenchmarkEasyJSONLexer-8 3000000 887 ns/op 613 B/op 6 allocs/op +BenchmarkJSONParserGet-8 3000000 499 ns/op 21 B/op 0 allocs/op +BenchmarkJSONIterator-8 3000000 812 ns/op 544 B/op 9 allocs/op +``` + +Benchmarks for the `GetMany` function: ``` -BenchmarkGJSONGet-8 15000000 333 ns/op 0 B/op 0 allocs/op -BenchmarkGJSONUnmarshalMap-8 900000 4188 ns/op 1920 B/op 26 allocs/op -BenchmarkJSONUnmarshalMap-8 600000 8908 ns/op 3048 B/op 69 allocs/op -BenchmarkJSONUnmarshalStruct-8 600000 9026 ns/op 1832 B/op 69 allocs/op -BenchmarkJSONDecoder-8 300000 14339 ns/op 4224 B/op 184 allocs/op -BenchmarkFFJSONLexer-8 1500000 3156 ns/op 896 B/op 8 allocs/op -BenchmarkEasyJSONLexer-8 3000000 938 ns/op 613 B/op 6 allocs/op -BenchmarkJSONParserGet-8 3000000 442 ns/op 21 B/op 0 allocs/op +BenchmarkGJSONGetMany4Paths-8 4000000 303 ns/op 112 B/op 0 allocs/op +BenchmarkGJSONGetMany8Paths-8 8000000 208 ns/op 56 B/op 0 allocs/op +BenchmarkGJSONGetMany16Paths-8 16000000 156 ns/op 56 B/op 0 allocs/op +BenchmarkGJSONGetMany32Paths-8 32000000 127 ns/op 64 B/op 0 allocs/op +BenchmarkGJSONGetMany64Paths-8 64000000 117 ns/op 64 B/op 0 allocs/op +BenchmarkGJSONGetMany128Paths-8 128000000 109 ns/op 64 B/op 0 allocs/op ``` JSON document used: @@ -268,8 +392,22 @@ widget.image.hOffset widget.text.onMouseUp ``` +For the `GetMany` benchmarks these paths are used: + +``` +widget.window.name +widget.image.hOffset +widget.text.onMouseUp +widget.window.title +widget.image.alignment +widget.text.style +widget.window.height +widget.image.src +widget.text.data +widget.text.size +``` -*These benchmarks were run on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.7.* +*These benchmarks were run on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.8.* ## Contact Josh Baker [@tidwall](http://twitter.com/tidwall) diff --git a/vendor/github.com/tidwall/gjson/gjson.go b/vendor/github.com/tidwall/gjson/gjson.go index 01c15aa..1403506 100644 --- a/vendor/github.com/tidwall/gjson/gjson.go +++ b/vendor/github.com/tidwall/gjson/gjson.go @@ -2,8 +2,17 @@ package gjson import ( + "encoding/base64" + "encoding/json" + "errors" "reflect" "strconv" + "strings" + "sync" + "sync/atomic" + "time" + "unicode/utf16" + "unicode/utf8" "unsafe" "github.com/tidwall/match" @@ -65,7 +74,7 @@ type Result struct { func (t Result) String() string { switch t.Type { default: - return "null" + return "" case False: return "false" case Number: @@ -101,10 +110,45 @@ func (t Result) Int() int64 { case True: return 1 case String: - n, _ := strconv.ParseInt(t.Str, 10, 64) + n, _ := parseInt(t.Str) + return n + case Number: + // try to directly convert the float64 to int64 + n, ok := floatToInt(t.Num) + if !ok { + // now try to parse the raw string + n, ok = parseInt(t.Raw) + if !ok { + // fallback to a standard conversion + return int64(t.Num) + } + } + return n + } +} + +// Uint returns an unsigned integer representation. +func (t Result) Uint() uint64 { + switch t.Type { + default: + return 0 + case True: + return 1 + case String: + n, _ := parseUint(t.Str) return n case Number: - return int64(t.Num) + // try to directly convert the float64 to uint64 + n, ok := floatToUint(t.Num) + if !ok { + // now try to parse the raw string + n, ok = parseUint(t.Raw) + if !ok { + // fallback to a standard conversion + return uint64(t.Num) + } + } + return n } } @@ -123,16 +167,97 @@ func (t Result) Float() float64 { } } -// Array returns back an array of children. The result must be a JSON array. +// Time returns a time.Time representation. +func (t Result) Time() time.Time { + res, _ := time.Parse(time.RFC3339, t.String()) + return res +} + +// Array returns back an array of values. +// If the result represents a non-existent value, then an empty array will be returned. +// If the result is not a JSON array, the return value will be an array containing one result. func (t Result) Array() []Result { - if t.Type != JSON { + if !t.Exists() { return nil } + if t.Type != JSON { + return []Result{t} + } r := t.arrayOrMap('[', false) return r.a } -// Map returns back an map of children. The result should be a JSON array. +// ForEach iterates through values. +// If the result represents a non-existent value, then no values will be iterated. +// If the result is an Object, the iterator will pass the key and value of each item. +// If the result is an Array, the iterator will only pass the value of each item. +// If the result is not a JSON array or object, the iterator will pass back one value equal to the result. +func (t Result) ForEach(iterator func(key, value Result) bool) { + if !t.Exists() { + return + } + if t.Type != JSON { + iterator(Result{}, t) + return + } + json := t.Raw + var keys bool + var i int + var key, value Result + for ; i < len(json); i++ { + if json[i] == '{' { + i++ + key.Type = String + keys = true + break + } else if json[i] == '[' { + i++ + break + } + if json[i] > ' ' { + return + } + } + var str string + var vesc bool + var ok bool + for ; i < len(json); i++ { + if keys { + if json[i] != '"' { + continue + } + s := i + i, str, vesc, ok = parseString(json, i+1) + if !ok { + return + } + if vesc { + key.Str = unescape(str[1 : len(str)-1]) + } else { + key.Str = str[1 : len(str)-1] + } + key.Raw = str + key.Index = s + } + for ; i < len(json); i++ { + if json[i] <= ' ' || json[i] == ',' || json[i] == ':' { + continue + } + break + } + s := i + i, value, ok = parseAny(json, i, true) + if !ok { + return + } + value.Index = s + if !iterator(key, value) { + return + } + } +} + +// Map returns back an map of values. The result should be a JSON array. func (t Result) Map() map[string]Result { if t.Type != JSON { return map[string]Result{} @@ -254,7 +379,7 @@ end: return } -// Parse parses the json and returns a result +// Parse parses the json and returns a result. func Parse(json string) Result { var value Result for i := 0; i < len(json); i++ { @@ -292,6 +417,12 @@ func Parse(json string) Result { return value } +// ParseBytes parses the json and returns a result. +// If working with bytes, this method preferred over Parse(string(data)) +func ParseBytes(json []byte) Result { + return Parse(string(json)) +} + func squash(json string) string { // expects that the lead character is a '[' or '{' // squash the value, ignoring all nested arrays and objects. @@ -534,6 +665,7 @@ type arrayPathResult struct { path string op string value string + all bool } } @@ -564,8 +696,12 @@ func parseArrayPath(path string) (r arrayPathResult) { } s := i for ; i < len(path); i++ { - if path[i] <= ' ' || path[i] == '=' || - path[i] == '<' || path[i] == '>' || + if path[i] <= ' ' || + path[i] == '!' || + path[i] == '=' || + path[i] == '<' || + path[i] == '>' || + path[i] == '%' || path[i] == ']' { break } @@ -579,7 +715,11 @@ func parseArrayPath(path string) (r arrayPathResult) { } if i < len(path) { s = i - if path[i] == '<' || path[i] == '>' { + if path[i] == '!' { + if i < len(path)-1 && path[i+1] == '=' { + i++ + } + } else if path[i] == '<' || path[i] == '>' { if i < len(path)-1 && path[i+1] == '=' { i++ } @@ -624,6 +764,9 @@ func parseArrayPath(path string) (r arrayPathResult) { } } } else if path[i] == ']' { + if i+1 < len(path) && path[i+1] == '#' { + r.query.all = true + } break } } @@ -795,7 +938,7 @@ func parseObject(c *parseContext, i int, path string) (int, bool) { break } } - i, key, kesc, ok = i, c.json[s:], false, false + key, kesc, ok = c.json[s:], false, false parse_key_string_done: break } @@ -905,6 +1048,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool { switch rp.query.op { case "=": return value.Str == rpv + case "!=": + return value.Str != rpv case "<": return value.Str < rpv case "<=": @@ -913,12 +1058,16 @@ func queryMatches(rp *arrayPathResult, value Result) bool { return value.Str > rpv case ">=": return value.Str >= rpv + case "%": + return match.Match(value.Str, rpv) } case Number: rpvn, _ := strconv.ParseFloat(rpv, 64) switch rp.query.op { case "=": return value.Num == rpvn + case "!=": + return value.Num == rpvn case "<": return value.Num < rpvn case "<=": @@ -932,6 +1081,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool { switch rp.query.op { case "=": return rpv == "true" + case "!=": + return rpv != "true" case ">": return rpv == "false" case ">=": @@ -941,6 +1092,8 @@ func queryMatches(rp *arrayPathResult, value Result) bool { switch rp.query.op { case "=": return rpv == "false" + case "!=": + return rpv != "false" case "<": return rpv == "true" case "<=": @@ -955,10 +1108,11 @@ func parseArray(c *parseContext, i int, path string) (int, bool) { var h int var alog []int var partidx int + var multires []byte rp := parseArrayPath(path) if !rp.arrch { - n, err := strconv.ParseUint(rp.part, 10, 64) - if err != nil { + n, ok := parseUint(rp.part) + if !ok { partidx = -1 } else { partidx = int(n) @@ -1011,12 +1165,21 @@ func parseArray(c *parseContext, i int, path string) (int, bool) { res := Get(val, rp.query.path) if queryMatches(&rp, res) { if rp.more { - c.value = Get(val, rp.path) + res = Get(val, rp.path) + } else { + res = Result{Raw: val, Type: JSON} + } + if rp.query.all { + if len(multires) == 0 { + multires = append(multires, '[') + } else { + multires = append(multires, ',') + } + multires = append(multires, res.Raw...) } else { - c.value.Raw = val - c.value.Type = JSON + c.value = res + return i, true } - return i, true } } else if hit { if rp.alogok { @@ -1079,27 +1242,34 @@ func parseArray(c *parseContext, i int, path string) (int, bool) { if rp.alogok { var jsons = make([]byte, 0, 64) jsons = append(jsons, '[') - for j := 0; j < len(alog); j++ { + for j, k := 0, 0; j < len(alog); j++ { res := Get(c.json[alog[j]:], rp.alogkey) if res.Exists() { - if j > 0 { + if k > 0 { jsons = append(jsons, ',') } jsons = append(jsons, []byte(res.Raw)...) + k++ } } jsons = append(jsons, ']') c.value.Type = JSON c.value.Raw = string(jsons) return i + 1, true - } else { - if rp.alogok { - break - } - c.value.Raw = val - c.value.Type = Number - c.value.Num = float64(h - 1) - return i + 1, true + } + if rp.alogok { + break + } + c.value.Raw = val + c.value.Type = Number + c.value.Num = float64(h - 1) + c.calcd = true + return i + 1, true + } + if len(multires) > 0 && !c.value.Exists() { + c.value = Result{ + Raw: string(append(multires, ']')), + Type: JSON, } } return i + 1, false @@ -1113,6 +1283,7 @@ func parseArray(c *parseContext, i int, path string) (int, bool) { type parseContext struct { json string value Result + calcd bool } // Get searches json for the specified path. @@ -1121,7 +1292,7 @@ type parseContext struct { // Invalid json will not panic, but it may return back unexpected results. // When the value is found it's returned immediately. // -// A path is a series of keys seperated by a dot. +// A path is a series of keys searated by a dot. // A key may contain special wildcard characters '*' and '?'. // To access an array value use the index as the key. // To get the number of elements in an array or to access a child path, use the '#' character. @@ -1138,11 +1309,12 @@ type parseContext struct { // } // "name.last" >> "Anderson" // "age" >> 37 +// "children" >> ["Sara","Alex","Jack"] // "children.#" >> 3 // "children.1" >> "Alex" // "child*.2" >> "Jack" // "c?ildren.0" >> "Sara" -// "friends.#.first" >> [ "James", "Roger" ] +// "friends.#.first" >> ["James","Roger"] // func Get(json, path string) Result { var i int @@ -1159,7 +1331,7 @@ func Get(json, path string) Result { break } } - if len(c.value.Raw) > 0 { + if len(c.value.Raw) > 0 && !c.calcd { jhdr := *(*reflect.StringHeader)(unsafe.Pointer(&json)) rhdr := *(*reflect.StringHeader)(unsafe.Pointer(&(c.value.Raw))) c.value.Index = int(rhdr.Data - jhdr.Data) @@ -1169,6 +1341,43 @@ func Get(json, path string) Result { } return c.value } +func fromBytesGet(result Result) Result { + // safely get the string headers + rawhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Raw)) + strhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Str)) + // create byte slice headers + rawh := reflect.SliceHeader{Data: rawhi.Data, Len: rawhi.Len} + strh := reflect.SliceHeader{Data: strhi.Data, Len: strhi.Len} + if strh.Data == 0 { + // str is nil + if rawh.Data == 0 { + // raw is nil + result.Raw = "" + } else { + // raw has data, safely copy the slice header to a string + result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) + } + result.Str = "" + } else if rawh.Data == 0 { + // raw is nil + result.Raw = "" + // str has data, safely copy the slice header to a string + result.Str = string(*(*[]byte)(unsafe.Pointer(&strh))) + } else if strh.Data >= rawh.Data && + int(strh.Data)+strh.Len <= int(rawh.Data)+rawh.Len { + // Str is a substring of Raw. + start := int(strh.Data - rawh.Data) + // safely copy the raw slice header + result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) + // substring the raw + result.Str = result.Raw[start : start+strh.Len] + } else { + // safely copy both the raw and str slice headers to strings + result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) + result.Str = string(*(*[]byte)(unsafe.Pointer(&strh))) + } + return result +} // GetBytes searches json for the specified path. // If working with bytes, this method preferred over Get(string(data), path) @@ -1177,44 +1386,17 @@ func GetBytes(json []byte, path string) Result { if json != nil { // unsafe cast to string result = Get(*(*string)(unsafe.Pointer(&json)), path) - // safely get the string headers - rawhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Raw)) - strhi := *(*reflect.StringHeader)(unsafe.Pointer(&result.Str)) - // create byte slice headers - rawh := reflect.SliceHeader{Data: rawhi.Data, Len: rawhi.Len} - strh := reflect.SliceHeader{Data: strhi.Data, Len: strhi.Len} - if strh.Data == 0 { - // str is nil - if rawh.Data == 0 { - // raw is nil - result.Raw = "" - } else { - // raw has data, safely copy the slice header to a string - result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) - } - result.Str = "" - } else if rawh.Data == 0 { - // raw is nil - result.Raw = "" - // str has data, safely copy the slice header to a string - result.Str = string(*(*[]byte)(unsafe.Pointer(&strh))) - } else if strh.Data >= rawh.Data && - int(strh.Data)+strh.Len <= int(rawh.Data)+rawh.Len { - // Str is a substring of Raw. - start := int(strh.Data - rawh.Data) - // safely copy the raw slice header - result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) - // substring the raw - result.Str = result.Raw[start : start+strh.Len] - } else { - // safely copy both the raw and str slice headers to strings - result.Raw = string(*(*[]byte)(unsafe.Pointer(&rawh))) - result.Str = string(*(*[]byte)(unsafe.Pointer(&strh))) - } + result = fromBytesGet(result) } return result } +// runeit returns the rune from the the \uXXXX +func runeit(json string) rune { + n, _ := strconv.ParseUint(json[:4], 16, 64) + return rune(n) +} + // unescape unescapes a string func unescape(json string) string { //, error) { var str = make([]byte, 0, len(json)) @@ -1223,15 +1405,15 @@ func unescape(json string) string { //, error) { default: str = append(str, json[i]) case json[i] < ' ': - return "" //, errors.New("invalid character in string") + return string(str) case json[i] == '\\': i++ if i >= len(json) { - return "" //, errors.New("invalid escape sequence") + return string(str) } switch json[i] { default: - return "" //, errors.New("invalid escape sequence") + return string(str) case '\\': str = append(str, '\\') case '/': @@ -1250,29 +1432,27 @@ func unescape(json string) string { //, error) { str = append(str, '"') case 'u': if i+5 > len(json) { - return "" //, errors.New("invalid escape sequence") + return string(str) } - i++ - // extract the codepoint - var code int - for j := i; j < i+4; j++ { - switch { - default: - return "" //, errors.New("invalid escape sequence") - case json[j] >= '0' && json[j] <= '9': - code += (int(json[j]) - '0') << uint(12-(j-i)*4) - case json[j] >= 'a' && json[j] <= 'f': - code += (int(json[j]) - 'a' + 10) << uint(12-(j-i)*4) - case json[j] >= 'a' && json[j] <= 'f': - code += (int(json[j]) - 'a' + 10) << uint(12-(j-i)*4) + r := runeit(json[i+1:]) + i += 5 + if utf16.IsSurrogate(r) { + // need another code + if len(json) >= 6 && json[i] == '\\' && json[i+1] == 'u' { + // we expect it to be correct so just consume it + r = utf16.DecodeRune(r, runeit(json[i+2:])) + i += 6 } } - str = append(str, []byte(string(code))...) - i += 3 // only 3 because we will increment on the for-loop + // provide enough space to encode the largest utf8 possible + str = append(str, 0, 0, 0, 0, 0, 0, 0, 0) + n := utf8.EncodeRune(str[len(str)-8:], r) + str = str[:len(str)-8+n] + i-- // backtrack index by one } } } - return string(str) //, nil + return string(str) } // Less return true if a token is less than another token. @@ -1336,3 +1516,917 @@ func stringLessInsensitive(a, b string) bool { } return len(a) < len(b) } + +// parseAny parses the next value from a json string. +// A Result is returned when the hit param is set. +// The return values are (i int, res Result, ok bool) +func parseAny(json string, i int, hit bool) (int, Result, bool) { + var res Result + var val string + for ; i < len(json); i++ { + if json[i] == '{' || json[i] == '[' { + i, val = parseSquash(json, i) + if hit { + res.Raw = val + res.Type = JSON + } + return i, res, true + } + if json[i] <= ' ' { + continue + } + switch json[i] { + case '"': + i++ + var vesc bool + var ok bool + i, val, vesc, ok = parseString(json, i) + if !ok { + return i, res, false + } + if hit { + res.Type = String + res.Raw = val + if vesc { + res.Str = unescape(val[1 : len(val)-1]) + } else { + res.Str = val[1 : len(val)-1] + } + } + return i, res, true + case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': + i, val = parseNumber(json, i) + if hit { + res.Raw = val + res.Type = Number + res.Num, _ = strconv.ParseFloat(val, 64) + } + return i, res, true + case 't', 'f', 'n': + vc := json[i] + i, val = parseLiteral(json, i) + if hit { + res.Raw = val + switch vc { + case 't': + res.Type = True + case 'f': + res.Type = False + } + return i, res, true + } + } + } + return i, res, false +} + +var ( // used for testing + testWatchForFallback bool + testLastWasFallback bool +) + +// areSimplePaths returns true if all the paths are simple enough +// to parse quickly for GetMany(). Allows alpha-numeric, dots, +// underscores, and the dollar sign. It does not allow non-alnum, +// escape characters, or keys which start with a numbers. +// For example: +// "name.last" == OK +// "user.id0" == OK +// "user.ID" == OK +// "user.first_name" == OK +// "user.firstName" == OK +// "user.0item" == BAD +// "user.#id" == BAD +// "user\.name" == BAD +func areSimplePaths(paths []string) bool { + for _, path := range paths { + var fi int // first key index, for keys with numeric prefix + for i := 0; i < len(path); i++ { + if path[i] >= 'a' && path[i] <= 'z' { + // a-z is likely to be the highest frequency charater. + continue + } + if path[i] == '.' { + fi = i + 1 + continue + } + if path[i] >= 'A' && path[i] <= 'Z' { + continue + } + if path[i] == '_' || path[i] == '$' { + continue + } + if i > fi && path[i] >= '0' && path[i] <= '9' { + continue + } + return false + } + } + return true +} + +// GetMany searches json for the multiple paths. +// The return value is a Result array where the number of items +// will be equal to the number of input paths. +func GetMany(json string, paths ...string) []Result { + if len(paths) < 4 { + if testWatchForFallback { + testLastWasFallback = false + } + switch len(paths) { + case 0: + // return nil when no paths are specified. + return nil + case 1: + return []Result{Get(json, paths[0])} + case 2: + return []Result{Get(json, paths[0]), Get(json, paths[1])} + case 3: + return []Result{Get(json, paths[0]), Get(json, paths[1]), Get(json, paths[2])} + } + } + var results []Result + var ok bool + var i int + if len(paths) > 512 { + // we can only support up to 512 paths. Is that too many? + goto fallback + } + if !areSimplePaths(paths) { + // If there is even one path that is not considered "simple" then + // we need to use the fallback method. + goto fallback + } + // locate the object token. + for ; i < len(json); i++ { + if json[i] == '{' { + i++ + break + } + if json[i] <= ' ' { + continue + } + goto fallback + } + // use the call function table. + if len(paths) <= 8 { + results, ok = getMany8(json, i, paths) + } else if len(paths) <= 16 { + results, ok = getMany16(json, i, paths) + } else if len(paths) <= 32 { + results, ok = getMany32(json, i, paths) + } else if len(paths) <= 64 { + results, ok = getMany64(json, i, paths) + } else if len(paths) <= 128 { + results, ok = getMany128(json, i, paths) + } else if len(paths) <= 256 { + results, ok = getMany256(json, i, paths) + } else if len(paths) <= 512 { + results, ok = getMany512(json, i, paths) + } + if !ok { + // there was some fault while parsing. we should try the + // fallback method. This could result in performance + // degregation in some cases. + goto fallback + } + if testWatchForFallback { + testLastWasFallback = false + } + return results +fallback: + results = results[:0] + for i := 0; i < len(paths); i++ { + results = append(results, Get(json, paths[i])) + } + if testWatchForFallback { + testLastWasFallback = true + } + return results +} + +// GetManyBytes searches json for the specified path. +// If working with bytes, this method preferred over +// GetMany(string(data), paths...) +func GetManyBytes(json []byte, paths ...string) []Result { + if json == nil { + return GetMany("", paths...) + } + results := GetMany(*(*string)(unsafe.Pointer(&json)), paths...) + for i := range results { + results[i] = fromBytesGet(results[i]) + } + return results +} + +// parseGetMany parses a json object for keys that match against the callers +// paths. It's a best-effort attempt and quickly locating and assigning the +// values to the []Result array. If there are failures such as bad json, or +// invalid input paths, or too much recursion, the function will exit with a +// return value of 'false'. +func parseGetMany( + json string, i int, + level uint, kplen int, + paths []string, completed []bool, matches []uint64, results []Result, +) (int, bool) { + if level > 62 { + // The recursion level is limited because the matches []uint64 + // array cannot handle more the 64-bits. + return i, false + } + // At this point the last character read was a '{'. + // Read all object keys and try to match against the paths. + var key string + var val string + var vesc, ok bool +next_key: + for ; i < len(json); i++ { + if json[i] == '"' { + // read the key + i, val, vesc, ok = parseString(json, i+1) + if !ok { + return i, false + } + if vesc { + // the value is escaped + key = unescape(val[1 : len(val)-1]) + } else { + // just a plain old ascii key + key = val[1 : len(val)-1] + } + var hasMatch bool + var parsedVal bool + var valOrgIndex int + var valPathIndex int + for j := 0; j < len(key); j++ { + if key[j] == '.' { + // we need to look for keys with dot and ignore them. + if i, _, ok = parseAny(json, i, false); !ok { + return i, false + } + continue next_key + } + } + var usedPaths int + // loop through paths and look for matches + for j := 0; j < len(paths); j++ { + if completed[j] { + usedPaths++ + // ignore completed paths + continue + } + if level > 0 && (matches[j]>>(level-1))&1 == 0 { + // ignore unmatched paths + usedPaths++ + continue + } + + // try to match the key to the path + // this is spaghetti code but the idea is to minimize + // calls and variable assignments when comparing the + // key to paths + if len(paths[j])-kplen >= len(key) { + i, k := kplen, 0 + for ; k < len(key); k, i = k+1, i+1 { + if key[k] != paths[j][i] { + // no match + goto nomatch + } + } + if i < len(paths[j]) { + if paths[j][i] == '.' { + // matched, but there are still more keys in path + goto match_not_atend + } + } + if len(paths[j]) <= len(key) || kplen != 0 { + // matched and at the end of the path + goto match_atend + } + } + // no match, jump to the nomatch label + goto nomatch + match_atend: + // found a match + // at the end of the path. we must take the value. + usedPaths++ + if !parsedVal { + // the value has not been parsed yet. let's do so. + valOrgIndex = i // keep track of the current position. + i, results[j], ok = parseAny(json, i, true) + if !ok { + return i, false + } + parsedVal = true + valPathIndex = j + } else { + results[j] = results[valPathIndex] + } + // mark as complete + completed[j] = true + // jump over the match_not_atend label + goto nomatch + match_not_atend: + // found a match + // still in the middle of the path. + usedPaths++ + // mark the path as matched + matches[j] |= 1 << level + if !hasMatch { + hasMatch = true + } + nomatch: // noop label + } + + if !parsedVal { + if hasMatch { + // we found a match and the value has not been parsed yet. + // let's find out if the next value type is an object. + for ; i < len(json); i++ { + if json[i] <= ' ' || json[i] == ':' { + continue + } + break + } + if i < len(json) { + if json[i] == '{' { + // it's an object. let's go deeper + i, ok = parseGetMany(json, i+1, level+1, kplen+len(key)+1, paths, completed, matches, results) + if !ok { + return i, false + } + } else { + // not an object. just parse and ignore. + if i, _, ok = parseAny(json, i, false); !ok { + return i, false + } + } + } + } else { + // Since there was no matches we can just parse the value and + // ignore the result. + if i, _, ok = parseAny(json, i, false); !ok { + return i, false + } + } + } else if hasMatch && len(results[valPathIndex].Raw) > 0 && results[valPathIndex].Raw[0] == '{' { + // The value was already parsed and the value type is an object. + // Rewind the json index and let's parse deeper. + i = valOrgIndex + for ; i < len(json); i++ { + if json[i] == '{' { + break + } + } + i, ok = parseGetMany(json, i+1, level+1, kplen+len(key)+1, paths, completed, matches, results) + if !ok { + return i, false + } + } + if usedPaths == len(paths) { + // all paths have been used, either completed or matched. + // we should stop parsing this object to save CPU cycles. + if level > 0 && i < len(json) { + i, _ = parseSquash(json, i) + } + return i, true + } + } else if json[i] == '}' { + // reached the end of the object. end it here. + return i + 1, true + } + } + return i, true +} + +// Call table for GetMany. Using an isolated function allows for allocating +// arrays with know capacities on the stack, as opposed to dynamically +// allocating on the heap. This can provide a tremendous performance boost +// by avoiding the GC. +func getMany8(json string, i int, paths []string) ([]Result, bool) { + const max = 8 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany16(json string, i int, paths []string) ([]Result, bool) { + const max = 16 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany32(json string, i int, paths []string) ([]Result, bool) { + const max = 32 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany64(json string, i int, paths []string) ([]Result, bool) { + const max = 64 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany128(json string, i int, paths []string) ([]Result, bool) { + const max = 128 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany256(json string, i int, paths []string) ([]Result, bool) { + const max = 256 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} +func getMany512(json string, i int, paths []string) ([]Result, bool) { + const max = 512 + var completed = make([]bool, 0, max) + var matches = make([]uint64, 0, max) + var results = make([]Result, 0, max) + completed = completed[0:len(paths):max] + matches = matches[0:len(paths):max] + results = results[0:len(paths):max] + _, ok := parseGetMany(json, i, 0, 0, paths, completed, matches, results) + return results, ok +} + +var fieldsmu sync.RWMutex +var fields = make(map[string]map[string]int) + +func assign(jsval Result, goval reflect.Value) { + if jsval.Type == Null { + return + } + switch goval.Kind() { + default: + case reflect.Ptr: + if !goval.IsNil() { + newval := reflect.New(goval.Elem().Type()) + assign(jsval, newval.Elem()) + goval.Elem().Set(newval.Elem()) + } else { + newval := reflect.New(goval.Type().Elem()) + assign(jsval, newval.Elem()) + goval.Set(newval) + } + case reflect.Struct: + fieldsmu.RLock() + sf := fields[goval.Type().String()] + fieldsmu.RUnlock() + if sf == nil { + fieldsmu.Lock() + sf = make(map[string]int) + for i := 0; i < goval.Type().NumField(); i++ { + f := goval.Type().Field(i) + tag := strings.Split(f.Tag.Get("json"), ",")[0] + if tag != "-" { + if tag != "" { + sf[tag] = i + sf[f.Name] = i + } else { + sf[f.Name] = i + } + } + } + fields[goval.Type().String()] = sf + fieldsmu.Unlock() + } + jsval.ForEach(func(key, value Result) bool { + if idx, ok := sf[key.Str]; ok { + f := goval.Field(idx) + if f.CanSet() { + assign(value, f) + } + } + return true + }) + case reflect.Slice: + if goval.Type().Elem().Kind() == reflect.Uint8 && jsval.Type == String { + data, _ := base64.StdEncoding.DecodeString(jsval.String()) + goval.Set(reflect.ValueOf(data)) + } else { + jsvals := jsval.Array() + slice := reflect.MakeSlice(goval.Type(), len(jsvals), len(jsvals)) + for i := 0; i < len(jsvals); i++ { + assign(jsvals[i], slice.Index(i)) + } + goval.Set(slice) + } + case reflect.Array: + i, n := 0, goval.Len() + jsval.ForEach(func(_, value Result) bool { + if i == n { + return false + } + assign(value, goval.Index(i)) + i++ + return true + }) + case reflect.Map: + if goval.Type().Key().Kind() == reflect.String && goval.Type().Elem().Kind() == reflect.Interface { + goval.Set(reflect.ValueOf(jsval.Value())) + } + case reflect.Interface: + goval.Set(reflect.ValueOf(jsval.Value())) + case reflect.Bool: + goval.SetBool(jsval.Bool()) + case reflect.Float32, reflect.Float64: + goval.SetFloat(jsval.Float()) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + goval.SetInt(jsval.Int()) + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + goval.SetUint(jsval.Uint()) + case reflect.String: + goval.SetString(jsval.String()) + } + if len(goval.Type().PkgPath()) > 0 { + v := goval.Addr() + if v.Type().NumMethod() > 0 { + if u, ok := v.Interface().(json.Unmarshaler); ok { + u.UnmarshalJSON([]byte(jsval.Raw)) + } + } + } +} + +var validate uintptr = 1 + +// UnmarshalValidationEnabled provides the option to disable JSON validation +// during the Unmarshal routine. Validation is enabled by default. +func UnmarshalValidationEnabled(enabled bool) { + if enabled { + atomic.StoreUintptr(&validate, 1) + } else { + atomic.StoreUintptr(&validate, 0) + } +} + +// Unmarshal loads the JSON data into the value pointed to by v. +// +// This function works almost identically to json.Unmarshal except that +// gjson.Unmarshal will automatically attempt to convert JSON values to any Go +// type. For example, the JSON string "100" or the JSON number 100 can be equally +// assigned to Go string, int, byte, uint64, etc. This rule applies to all types. +func Unmarshal(data []byte, v interface{}) error { + if atomic.LoadUintptr(&validate) == 1 { + _, ok := validpayload(data, 0) + if !ok { + return errors.New("invalid json") + } + } + if v := reflect.ValueOf(v); v.Kind() == reflect.Ptr { + assign(ParseBytes(data), v) + } + return nil +} + +func validpayload(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + i, ok = validany(data, i) + if !ok { + return i, false + } + for ; i < len(data); i++ { + switch data[i] { + default: + return i, false + case ' ', '\t', '\n', '\r': + continue + } + } + return i, true + case ' ', '\t', '\n', '\r': + continue + } + } + return i, false +} +func validany(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + return i, false + case ' ', '\t', '\n', '\r': + continue + case '{': + return validobject(data, i+1) + case '[': + return validarray(data, i+1) + case '"': + return validstring(data, i+1) + case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': + return validnumber(data, i+1) + case 't': + return validtrue(data, i+1) + case 'f': + return validfalse(data, i+1) + case 'n': + return validnull(data, i+1) + } + } + return i, false +} +func validobject(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + return i, false + case ' ', '\t', '\n', '\r': + continue + case '}': + return i + 1, true + case '"': + key: + if i, ok = validstring(data, i+1); !ok { + return i, false + } + if i, ok = validcolon(data, i); !ok { + return i, false + } + if i, ok = validany(data, i); !ok { + return i, false + } + if i, ok = validcomma(data, i, '}'); !ok { + return i, false + } + if data[i] == '}' { + return i + 1, true + } + for ; i < len(data); i++ { + if data[i] == '"' { + goto key + } + } + return i, false + } + } + return i, false +} +func validcolon(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + return i, false + case ' ', '\t', '\n', '\r': + continue + case ':': + return i + 1, true + } + } + return i, false +} +func validcomma(data []byte, i int, end byte) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + return i, false + case ' ', '\t', '\n', '\r': + continue + case ',': + return i, true + case end: + return i, true + } + } + return i, false +} +func validarray(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + switch data[i] { + default: + for ; i < len(data); i++ { + if i, ok = validany(data, i); !ok { + return i, false + } + if i, ok = validcomma(data, i, ']'); !ok { + return i, false + } + if data[i] == ']' { + return i + 1, true + } + } + case ' ', '\t', '\n', '\r': + continue + case ']': + return i + 1, true + } + } + return i, false +} +func validstring(data []byte, i int) (outi int, ok bool) { + for ; i < len(data); i++ { + if data[i] < ' ' { + return i, false + } else if data[i] == '\\' { + i++ + if i == len(data) { + return i, false + } + switch data[i] { + default: + return i, false + case '"', '\\', '/', 'b', 'f', 'n', 'r', 't': + case 'u': + for j := 0; j < 4; j++ { + i++ + if i >= len(data) { + return i, false + } + if !((data[i] >= '0' && data[i] <= '9') || + (data[i] >= 'a' && data[i] <= 'f') || + (data[i] >= 'A' && data[i] <= 'F')) { + return i, false + } + } + } + } else if data[i] == '"' { + return i + 1, true + } + } + return i, false +} +func validnumber(data []byte, i int) (outi int, ok bool) { + i-- + // sign + if data[i] == '-' { + i++ + } + // int + if i == len(data) { + return i, false + } + if data[i] == '0' { + i++ + } else { + for ; i < len(data); i++ { + if data[i] >= '0' && data[i] <= '9' { + continue + } + break + } + } + // frac + if i == len(data) { + return i, true + } + if data[i] == '.' { + i++ + if i == len(data) { + return i, false + } + if data[i] < '0' || data[i] > '9' { + return i, false + } + i++ + for ; i < len(data); i++ { + if data[i] >= '0' && data[i] <= '9' { + continue + } + break + } + } + // exp + if i == len(data) { + return i, true + } + if data[i] == 'e' || data[i] == 'E' { + i++ + if i == len(data) { + return i, false + } + if data[i] == '+' || data[i] == '-' { + i++ + } + if i == len(data) { + return i, false + } + if data[i] < '0' || data[i] > '9' { + return i, false + } + i++ + for ; i < len(data); i++ { + if data[i] >= '0' && data[i] <= '9' { + continue + } + break + } + } + return i, true +} + +func validtrue(data []byte, i int) (outi int, ok bool) { + if i+3 <= len(data) && data[i] == 'r' && data[i+1] == 'u' && data[i+2] == 'e' { + return i + 3, true + } + return i, false +} +func validfalse(data []byte, i int) (outi int, ok bool) { + if i+4 <= len(data) && data[i] == 'a' && data[i+1] == 'l' && data[i+2] == 's' && data[i+3] == 'e' { + return i + 4, true + } + return i, false +} +func validnull(data []byte, i int) (outi int, ok bool) { + if i+3 <= len(data) && data[i] == 'u' && data[i+1] == 'l' && data[i+2] == 'l' { + return i + 3, true + } + return i, false +} + +// Valid returns true if the input is valid json. +func Valid(json string) bool { + _, ok := validpayload([]byte(json), 0) + return ok +} + +func parseUint(s string) (n uint64, ok bool) { + var i int + if i == len(s) { + return 0, false + } + for ; i < len(s); i++ { + if s[i] >= '0' && s[i] <= '9' { + n = n*10 + uint64(s[i]-'0') + } else { + return 0, false + } + } + return n, true +} + +func parseInt(s string) (n int64, ok bool) { + var i int + var sign bool + if len(s) > 0 && s[0] == '-' { + sign = true + i++ + } + if i == len(s) { + return 0, false + } + for ; i < len(s); i++ { + if s[i] >= '0' && s[i] <= '9' { + n = n*10 + int64(s[i]-'0') + } else { + return 0, false + } + } + if sign { + return n * -1, true + } + return n, true +} + +const minUint53 = 0 +const maxUint53 = 4503599627370495 +const minInt53 = -2251799813685248 +const maxInt53 = 2251799813685247 + +func floatToUint(f float64) (n uint64, ok bool) { + n = uint64(f) + if float64(n) == f && n >= minUint53 && n <= maxUint53 { + return n, true + } + return 0, false +} + +func floatToInt(f float64) (n int64, ok bool) { + n = int64(f) + if float64(n) == f && n >= minInt53 && n <= maxInt53 { + return n, true + } + return 0, false +} diff --git a/vendor/github.com/tidwall/gjson/gjson_test.go b/vendor/github.com/tidwall/gjson/gjson_test.go index 0740211..00ab1bd 100644 --- a/vendor/github.com/tidwall/gjson/gjson_test.go +++ b/vendor/github.com/tidwall/gjson/gjson_test.go @@ -7,11 +7,14 @@ import ( "fmt" "io" "math/rand" + "reflect" + "strconv" "strings" "testing" "time" "github.com/buger/jsonparser" + jsoniter "github.com/json-iterator/go" "github.com/mailru/easyjson/jlexer" fflib "github.com/pquerna/ffjson/fflib/v1" ) @@ -63,6 +66,17 @@ func TestRandomValidStrings(t *testing.T) { } } } + +func TestEmoji(t *testing.T) { + const input = `{"utf8":"Example emoji, KO: \ud83d\udd13, \ud83c\udfc3 OK: \u2764\ufe0f "}` + value := Get(input, "utf8") + var s string + json.Unmarshal([]byte(value.Raw), &s) + if value.String() != s { + t.Fatalf("expected '%v', got '%v'", s, value.String()) + } +} + func testEscapePath(t *testing.T, json, path, expect string) { if Get(json, path).String() != expect { t.Fatalf("expected '%v', got '%v'", expect, Get(json, path).String()) @@ -102,32 +116,86 @@ var basicJSON = `{"age":100, "name":{"here":"B\\\"R"}, "items":[1,2,3,{"tags":[1,2,3],"points":[[1,2],[3,4]]},4,5,6,7], "arr":["1",2,"3",{"hello":"world"},"4",5], "vals":[1,2,3,{"sadf":sdf"asdf"}],"name":{"first":"tom","last":null}, + "created":"2014-05-16T08:28:06.989Z", "loggy":{ "programmers": [ { - "firstName": "Brett", - "lastName": "McLaughlin", - "email": "aaaa" - }, + "firstName": "Brett", + "lastName": "McLaughlin", + "email": "aaaa", + "tag": "good" + }, { - "firstName": "Jason", - "lastName": "Hunter", - "email": "bbbb" - }, + "firstName": "Jason", + "lastName": "Hunter", + "email": "bbbb", + "tag": "bad" + }, { - "firstName": "Elliotte", - "lastName": "Harold", - "email": "cccc" + "firstName": "Elliotte", + "lastName": "Harold", + "email": "cccc", + "tag":, "good" }, { "firstName": 1002.3, "age": 101 } ] - } + }, + "lastly":{"yay":"final"} }` var basicJSONB = []byte(basicJSON) +func TestTimeResult(t *testing.T) { + assert(t, Get(basicJSON, "created").String() == Get(basicJSON, "created").Time().Format(time.RFC3339Nano)) +} + +func TestParseAny(t *testing.T) { + assert(t, Parse("100").Float() == 100) + assert(t, Parse("true").Bool()) + assert(t, Parse("valse").Bool() == false) +} + +func TestManyVariousPathCounts(t *testing.T) { + json := `{"a":"a","b":"b","c":"c"}` + counts := []int{3, 4, 7, 8, 9, 15, 16, 17, 31, 32, 33, 63, 64, 65, 127, 128, 129, 255, 256, 257, 511, 512, 513} + paths := []string{"a", "b", "c"} + expects := []string{"a", "b", "c"} + for _, count := range counts { + var gpaths []string + var gexpects []string + for i := 0; i < count; i++ { + if i < len(paths) { + gpaths = append(gpaths, paths[i]) + gexpects = append(gexpects, expects[i]) + } else { + gpaths = append(gpaths, fmt.Sprintf("not%d", i)) + gexpects = append(gexpects, "null") + } + } + results := GetMany(json, gpaths...) + for i := 0; i < len(paths); i++ { + if results[i].String() != expects[i] { + t.Fatalf("expected '%v', got '%v'", expects[i], results[i].String()) + } + } + } +} +func TestManyRecursion(t *testing.T) { + var json string + var path string + for i := 0; i < 100; i++ { + json += `{"a":` + path += ".a" + } + json += `"b"` + for i := 0; i < 100; i++ { + json += `}` + } + path = path[1:] + assert(t, GetMany(json, path)[0].String() == "b") +} func TestByteSafety(t *testing.T) { jsonb := []byte(`{"name":"Janet","age":38}`) mtok := GetBytes(jsonb, "name") @@ -147,17 +215,172 @@ func TestByteSafety(t *testing.T) { } func get(json, path string) Result { - return GetBytes([]byte(basicJSONB), path) + return GetBytes([]byte(json), path) } func TestBasic(t *testing.T) { var mtok Result + mtok = get(basicJSON, `loggy.programmers.#[tag="good"].firstName`) + if mtok.String() != "Brett" { + t.Fatalf("expected %v, got %v", "Brett", mtok.String()) + } + mtok = get(basicJSON, `loggy.programmers.#[tag="good"]#.firstName`) + if mtok.String() != `["Brett","Elliotte"]` { + t.Fatalf("expected %v, got %v", `["Brett","Elliotte"]`, mtok.String()) + } +} +func TestPlus53BitInts(t *testing.T) { + json := `{"IdentityData":{"GameInstanceId":634866135153775564}}` + value := Get(json, "IdentityData.GameInstanceId") + assert(t, value.Uint() == 634866135153775564) + assert(t, value.Int() == 634866135153775564) + assert(t, value.Float() == 634866135153775616) + + json = `{"IdentityData":{"GameInstanceId":634866135153775564.88172}}` + value = Get(json, "IdentityData.GameInstanceId") + assert(t, value.Uint() == 634866135153775616) + assert(t, value.Int() == 634866135153775616) + assert(t, value.Float() == 634866135153775616.88172) - mtok = get(basicJSON, `loggy.programmers.#[age=101].firstName`) + json = `{ + "min_uint64": 0, + "max_uint64": 18446744073709551615, + "overflow_uint64": 18446744073709551616, + "min_int64": -9223372036854775808, + "max_int64": 9223372036854775807, + "overflow_int64": 9223372036854775808, + "min_uint53": 0, + "max_uint53": 4503599627370495, + "overflow_uint53": 4503599627370496, + "min_int53": -2251799813685248, + "max_int53": 2251799813685247, + "overflow_int53": 2251799813685248 + }` + + assert(t, Get(json, "min_uint53").Uint() == 0) + assert(t, Get(json, "max_uint53").Uint() == 4503599627370495) + assert(t, Get(json, "overflow_uint53").Int() == 4503599627370496) + assert(t, Get(json, "min_int53").Int() == -2251799813685248) + assert(t, Get(json, "max_int53").Int() == 2251799813685247) + assert(t, Get(json, "overflow_int53").Int() == 2251799813685248) + assert(t, Get(json, "min_uint64").Uint() == 0) + assert(t, Get(json, "max_uint64").Uint() == 18446744073709551615) + // this next value overflows the max uint64 by one which will just + // flip the number to zero + assert(t, Get(json, "overflow_uint64").Int() == 0) + assert(t, Get(json, "min_int64").Int() == -9223372036854775808) + assert(t, Get(json, "max_int64").Int() == 9223372036854775807) + // this next value overflows the max int64 by one which will just + // flip the number to the negative sign. + assert(t, Get(json, "overflow_int64").Int() == -9223372036854775808) +} + +func TestTypes(t *testing.T) { + assert(t, (Result{Type: String}).Type.String() == "String") + assert(t, (Result{Type: Number}).Type.String() == "Number") + assert(t, (Result{Type: Null}).Type.String() == "Null") + assert(t, (Result{Type: False}).Type.String() == "False") + assert(t, (Result{Type: True}).Type.String() == "True") + assert(t, (Result{Type: JSON}).Type.String() == "JSON") + assert(t, (Result{Type: 100}).Type.String() == "") + // bool + assert(t, (Result{Type: String, Str: "true"}).Bool()) + assert(t, (Result{Type: True}).Bool()) + assert(t, (Result{Type: False}).Bool() == false) + assert(t, (Result{Type: Number, Num: 1}).Bool()) + // int + assert(t, (Result{Type: String, Str: "1"}).Int() == 1) + assert(t, (Result{Type: True}).Int() == 1) + assert(t, (Result{Type: False}).Int() == 0) + assert(t, (Result{Type: Number, Num: 1}).Int() == 1) + // uint + assert(t, (Result{Type: String, Str: "1"}).Uint() == 1) + assert(t, (Result{Type: True}).Uint() == 1) + assert(t, (Result{Type: False}).Uint() == 0) + assert(t, (Result{Type: Number, Num: 1}).Uint() == 1) + // float + assert(t, (Result{Type: String, Str: "1"}).Float() == 1) + assert(t, (Result{Type: True}).Float() == 1) + assert(t, (Result{Type: False}).Float() == 0) + assert(t, (Result{Type: Number, Num: 1}).Float() == 1) +} +func TestForEach(t *testing.T) { + Result{}.ForEach(nil) + Result{Type: String, Str: "Hello"}.ForEach(func(_, value Result) bool { + assert(t, value.String() == "Hello") + return false + }) + Result{Type: JSON, Raw: "*invalid*"}.ForEach(nil) + + json := ` {"name": {"first": "Janet","last": "Prichard"}, + "asd\nf":"\ud83d\udd13","age": 47}` + var count int + ParseBytes([]byte(json)).ForEach(func(key, value Result) bool { + count++ + return true + }) + assert(t, count == 3) + ParseBytes([]byte(`{"bad`)).ForEach(nil) + ParseBytes([]byte(`{"ok":"bad`)).ForEach(nil) +} +func TestMap(t *testing.T) { + assert(t, len(ParseBytes([]byte(`"asdf"`)).Map()) == 0) + assert(t, ParseBytes([]byte(`{"asdf":"ghjk"`)).Map()["asdf"].String() == "ghjk") + assert(t, len(Result{Type: JSON, Raw: "**invalid**"}.Map()) == 0) + assert(t, Result{Type: JSON, Raw: "**invalid**"}.Value() == nil) + assert(t, Result{Type: JSON, Raw: "{"}.Map() != nil) +} +func TestBasic1(t *testing.T) { + mtok := get(basicJSON, `loggy.programmers`) + var count int + mtok.ForEach(func(key, value Result) bool { + if key.Exists() { + t.Fatalf("expected %v, got %v", false, key.Exists()) + } + count++ + if count == 3 { + return false + } + if count == 1 { + i := 0 + value.ForEach(func(key, value Result) bool { + switch i { + case 0: + if key.String() != "firstName" || value.String() != "Brett" { + t.Fatalf("expected %v/%v got %v/%v", "firstName", "Brett", key.String(), value.String()) + } + case 1: + if key.String() != "lastName" || value.String() != "McLaughlin" { + t.Fatalf("expected %v/%v got %v/%v", "lastName", "McLaughlin", key.String(), value.String()) + } + case 2: + if key.String() != "email" || value.String() != "aaaa" { + t.Fatalf("expected %v/%v got %v/%v", "email", "aaaa", key.String(), value.String()) + } + } + i++ + return true + }) + } + return true + }) + if count != 3 { + t.Fatalf("expected %v, got %v", 3, count) + } +} +func TestBasic2(t *testing.T) { + mtok := get(basicJSON, `loggy.programmers.#[age=101].firstName`) if mtok.String() != "1002.3" { - t.Fatalf("expected %v, got %v", "1002,3", mtok.String()) + t.Fatalf("expected %v, got %v", "1002.3", mtok.String()) + } + mtok = get(basicJSON, `loggy.programmers.#[firstName != "Brett"].firstName`) + if mtok.String() != "Jason" { + t.Fatalf("expected %v, got %v", "Jason", mtok.String()) + } + mtok = get(basicJSON, `loggy.programmers.#[firstName % "Bre*"].email`) + if mtok.String() != "aaaa" { + t.Fatalf("expected %v, got %v", "aaaa", mtok.String()) } - mtok = get(basicJSON, `loggy.programmers.#[firstName == "Brett"].email`) if mtok.String() != "aaaa" { t.Fatalf("expected %v, got %v", "aaaa", mtok.String()) @@ -173,25 +396,27 @@ func TestBasic(t *testing.T) { if programmers.Array()[1].Map()["firstName"].Str != "Jason" { t.Fatalf("expected %v, got %v", "Jason", mtok.Map()["programmers"].Array()[1].Map()["firstName"].Str) } - +} +func TestBasic3(t *testing.T) { + var mtok Result if Parse(basicJSON).Get("loggy.programmers").Get("1").Get("firstName").Str != "Jason" { t.Fatalf("expected %v, got %v", "Jason", Parse(basicJSON).Get("loggy.programmers").Get("1").Get("firstName").Str) } var token Result if token = Parse("-102"); token.Num != -102 { - t.Fatal("expected %v, got %v", -102, token.Num) + t.Fatalf("expected %v, got %v", -102, token.Num) } if token = Parse("102"); token.Num != 102 { - t.Fatal("expected %v, got %v", 102, token.Num) + t.Fatalf("expected %v, got %v", 102, token.Num) } if token = Parse("102.2"); token.Num != 102.2 { - t.Fatal("expected %v, got %v", 102.2, token.Num) + t.Fatalf("expected %v, got %v", 102.2, token.Num) } if token = Parse(`"hello"`); token.Str != "hello" { - t.Fatal("expected %v, got %v", "hello", token.Str) + t.Fatalf("expected %v, got %v", "hello", token.Str) } if token = Parse(`"\"he\nllo\""`); token.Str != "\"he\nllo\"" { - t.Fatal("expected %v, got %v", "\"he\nllo\"", token.Str) + t.Fatalf("expected %v, got %v", "\"he\nllo\"", token.Str) } mtok = get(basicJSON, "loggy.programmers.#.firstName") if len(mtok.Array()) != 4 { @@ -204,12 +429,13 @@ func TestBasic(t *testing.T) { } mtok = get(basicJSON, "loggy.programmers.#.asd") if mtok.Type != JSON { - t.Fatal("expected %v, got %v", JSON, mtok.Type) + t.Fatalf("expected %v, got %v", JSON, mtok.Type) } if len(mtok.Array()) != 0 { t.Fatalf("expected 0, got %v", len(mtok.Array())) } - +} +func TestBasic4(t *testing.T) { if get(basicJSON, "items.3.tags.#").Num != 3 { t.Fatalf("expected 3, got %v", get(basicJSON, "items.3.tags.#").Num) } @@ -225,7 +451,7 @@ func TestBasic(t *testing.T) { if !get(basicJSON, "name.last").Exists() { t.Fatal("expected true, got false") } - token = get(basicJSON, "name.here") + token := get(basicJSON, "name.here") if token.String() != "B\\\"R" { t.Fatal("expecting 'B\\\"R'", "got", token.String()) } @@ -244,13 +470,15 @@ func TestBasic(t *testing.T) { } _ = token.Value().(string) token = get(basicJSON, "name.last") - if token.String() != "null" { - t.Fatal("expecting 'null'", "got", token.String()) + if token.String() != "" { + t.Fatal("expecting ''", "got", token.String()) } if token.Value() != nil { t.Fatal("should be nil") } - token = get(basicJSON, "age") +} +func TestBasic5(t *testing.T) { + token := get(basicJSON, "age") if token.String() != "100" { t.Fatal("expecting '100'", "got", token.String()) } @@ -314,7 +542,7 @@ func TestUnescape(t *testing.T) { } func assert(t testing.TB, cond bool) { if !cond { - t.Fatal("assert failed") + panic("assert failed") } } func TestLess(t *testing.T) { @@ -445,6 +673,384 @@ func TestUnmarshalMap(t *testing.T) { } } +func TestSingleArrayValue(t *testing.T) { + var json = `{"key": "value","key2":[1,2,3,4,"A"]}` + var result = Get(json, "key") + var array = result.Array() + if len(array) != 1 { + t.Fatal("array is empty") + } + if array[0].String() != "value" { + t.Fatalf("got %s, should be %s", array[0].String(), "value") + } + + array = Get(json, "key2.#").Array() + if len(array) != 1 { + t.Fatalf("got '%v', expected '%v'", len(array), 1) + } + + array = Get(json, "key3").Array() + if len(array) != 0 { + t.Fatalf("got '%v', expected '%v'", len(array), 0) + } + +} + +var manyJSON = ` { + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{ + "a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"hello":"world" + }}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}} + "position":{"type":"Point","coordinates":[-115.24,33.09]}, + "loves":["world peace"], + "name":{"last":"Anderson","first":"Nancy"}, + "age":31 + "":{"a":"emptya","b":"emptyb"}, + "name.last":"Yellow", + "name.first":"Cat", +}` + +func combine(results []Result) string { + return fmt.Sprintf("%v", results) +} +func TestManyBasic(t *testing.T) { + testWatchForFallback = true + defer func() { + testWatchForFallback = false + }() + testMany := func(shouldFallback bool, expect string, paths ...string) { + results := GetManyBytes( + []byte(manyJSON), + paths..., + ) + if len(results) != len(paths) { + t.Fatalf("expected %v, got %v", len(paths), len(results)) + } + if fmt.Sprintf("%v", results) != expect { + fmt.Printf("%v\n", paths) + t.Fatalf("expected %v, got %v", expect, results) + } + //if testLastWasFallback != shouldFallback { + // t.Fatalf("expected %v, got %v", shouldFallback, testLastWasFallback) + //} + } + testMany(false, "[Point]", "position.type") + testMany(false, `[emptya ["world peace"] 31]`, ".a", "loves", "age") + testMany(false, `[["world peace"]]`, "loves") + testMany(false, `[{"last":"Anderson","first":"Nancy"} Nancy]`, "name", "name.first") + testMany(true, `[]`, strings.Repeat("a.", 40)+"hello") + res := Get(manyJSON, strings.Repeat("a.", 48)+"a") + testMany(true, `[`+res.String()+`]`, strings.Repeat("a.", 48)+"a") + // these should fallback + testMany(true, `[Cat Nancy]`, "name\\.first", "name.first") + testMany(true, `[world]`, strings.Repeat("a.", 70)+"hello") +} +func testMany(t *testing.T, json string, paths, expected []string) { + testManyAny(t, json, paths, expected, true) + testManyAny(t, json, paths, expected, false) +} +func testManyAny(t *testing.T, json string, paths, expected []string, bytes bool) { + var result []Result + for i := 0; i < 2; i++ { + var which string + if i == 0 { + which = "Get" + result = nil + for j := 0; j < len(expected); j++ { + if bytes { + result = append(result, GetBytes([]byte(json), paths[j])) + } else { + result = append(result, Get(json, paths[j])) + } + } + } else if i == 1 { + which = "GetMany" + if bytes { + result = GetManyBytes([]byte(json), paths...) + } else { + result = GetMany(json, paths...) + } + } + for j := 0; j < len(expected); j++ { + if result[j].String() != expected[j] { + t.Fatalf("Using key '%s' for '%s'\nexpected '%v', got '%v'", paths[j], which, expected[j], result[j].String()) + } + } + } +} +func TestIssue20(t *testing.T) { + json := `{ "name": "FirstName", "name1": "FirstName1", "address": "address1", "addressDetails": "address2", }` + paths := []string{"name", "name1", "address", "addressDetails"} + expected := []string{"FirstName", "FirstName1", "address1", "address2"} + t.Run("SingleMany", func(t *testing.T) { testMany(t, json, paths, expected) }) +} + +func TestIssue21(t *testing.T) { + json := `{ "Level1Field1":3, + "Level1Field4":4, + "Level1Field2":{ "Level2Field1":[ "value1", "value2" ], + "Level2Field2":{ "Level3Field1":[ { "key1":"value1" } ] } } }` + paths := []string{"Level1Field1", "Level1Field2.Level2Field1", "Level1Field2.Level2Field2.Level3Field1", "Level1Field4"} + expected := []string{"3", `[ "value1", "value2" ]`, `[ { "key1":"value1" } ]`, "4"} + t.Run("SingleMany", func(t *testing.T) { testMany(t, json, paths, expected) }) +} + +func TestRandomMany(t *testing.T) { + var lstr string + defer func() { + if v := recover(); v != nil { + println("'" + hex.EncodeToString([]byte(lstr)) + "'") + println("'" + lstr + "'") + panic(v) + } + }() + rand.Seed(time.Now().UnixNano()) + b := make([]byte, 512) + for i := 0; i < 50000; i++ { + n, err := rand.Read(b[:rand.Int()%len(b)]) + if err != nil { + t.Fatal(err) + } + lstr = string(b[:n]) + paths := make([]string, rand.Int()%64) + for i := range paths { + var b []byte + n := rand.Int() % 5 + for j := 0; j < n; j++ { + if j > 0 { + b = append(b, '.') + } + nn := rand.Int() % 10 + for k := 0; k < nn; k++ { + b = append(b, 'a'+byte(rand.Int()%26)) + } + } + paths[i] = string(b) + } + GetMany(lstr, paths...) + } +} + +type ComplicatedType struct { + unsettable int + Tagged string `json:"tagged"` + NotTagged bool + Nested struct { + Yellow string `json:"yellow"` + } + NestedTagged struct { + Green string + Map map[string]interface{} + Ints struct { + Int int `json:"int"` + Int8 int8 + Int16 int16 + Int32 int32 + Int64 int64 `json:"int64"` + } + Uints struct { + Uint uint + Uint8 uint8 + Uint16 uint16 + Uint32 uint32 + Uint64 uint64 + } + Floats struct { + Float64 float64 + Float32 float32 + } + Byte byte + Bool bool + } `json:"nestedTagged"` + LeftOut string `json:"-"` + SelfPtr *ComplicatedType + SelfSlice []ComplicatedType + SelfSlicePtr []*ComplicatedType + SelfPtrSlice *[]ComplicatedType + Interface interface{} `json:"interface"` + Array [3]int + Time time.Time `json:"time"` + Binary []byte + NonBinary []byte +} + +var complicatedJSON = ` +{ + "tagged": "OK", + "Tagged": "KO", + "NotTagged": true, + "unsettable": 101, + "Nested": { + "Yellow": "Green", + "yellow": "yellow" + }, + "nestedTagged": { + "Green": "Green", + "Map": { + "this": "that", + "and": "the other thing" + }, + "Ints": { + "Uint": 99, + "Uint16": 16, + "Uint32": 32, + "Uint64": 65 + }, + "Uints": { + "int": -99, + "Int": -98, + "Int16": -16, + "Int32": -32, + "int64": -64, + "Int64": -65 + }, + "Uints": { + "Float32": 32.32, + "Float64": 64.64 + }, + "Byte": 254, + "Bool": true + }, + "LeftOut": "you shouldn't be here", + "SelfPtr": {"tagged":"OK","nestedTagged":{"Ints":{"Uint32":32}}}, + "SelfSlice": [{"tagged":"OK","nestedTagged":{"Ints":{"Uint32":32}}}], + "SelfSlicePtr": [{"tagged":"OK","nestedTagged":{"Ints":{"Uint32":32}}}], + "SelfPtrSlice": [{"tagged":"OK","nestedTagged":{"Ints":{"Uint32":32}}}], + "interface": "Tile38 Rocks!", + "Interface": "Please Download", + "Array": [0,2,3,4,5], + "time": "2017-05-07T13:24:43-07:00", + "Binary": "R0lGODlhPQBEAPeo", + "NonBinary": [9,3,100,115] +} +` + +func TestUnmarshal(t *testing.T) { + var s1 ComplicatedType + var s2 ComplicatedType + if err := json.Unmarshal([]byte(complicatedJSON), &s1); err != nil { + t.Fatal(err) + } + if err := Unmarshal([]byte(complicatedJSON), &s2); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(&s1, &s2) { + t.Fatal("not equal") + } + var str string + if err := json.Unmarshal([]byte(Get(complicatedJSON, "LeftOut").Raw), &str); err != nil { + t.Fatal(err) + } + assert(t, str == Get(complicatedJSON, "LeftOut").String()) +} + +func testvalid(json string, expect bool) { + _, ok := validpayload([]byte(json), 0) + if ok != expect { + panic("mismatch") + } +} + +func TestValidBasic(t *testing.T) { + testvalid("0", true) + testvalid("00", false) + testvalid("-00", false) + testvalid("-.", false) + testvalid("0.0", true) + testvalid("10.0", true) + testvalid("10e1", true) + testvalid("10EE", false) + testvalid("10E-", false) + testvalid("10E+", false) + testvalid("10E123", true) + testvalid("10E-123", true) + testvalid("10E-0123", true) + testvalid("", false) + testvalid(" ", false) + testvalid("{}", true) + testvalid("{", false) + testvalid("-", false) + testvalid("-1", true) + testvalid("-1.", false) + testvalid("-1.0", true) + testvalid(" -1.0", true) + testvalid(" -1.0 ", true) + testvalid("-1.0 ", true) + testvalid("-1.0 i", false) + testvalid("-1.0 i", false) + testvalid("true", true) + testvalid(" true", true) + testvalid(" true ", true) + testvalid(" True ", false) + testvalid(" tru", false) + testvalid("false", true) + testvalid(" false", true) + testvalid(" false ", true) + testvalid(" False ", false) + testvalid(" fals", false) + testvalid("null", true) + testvalid(" null", true) + testvalid(" null ", true) + testvalid(" Null ", false) + testvalid(" nul", false) + testvalid(" []", true) + testvalid(" [true]", true) + testvalid(" [ true, null ]", true) + testvalid(" [ true,]", false) + testvalid(`{"hello":"world"}`, true) + testvalid(`{ "hello": "world" }`, true) + testvalid(`{ "hello": "world", }`, false) + testvalid(`{"a":"b",}`, false) + testvalid(`{"a":"b","a"}`, false) + testvalid(`{"a":"b","a":}`, false) + testvalid(`{"a":"b","a":1}`, true) + testvalid(`{"a":"b","a": 1, "c":{"hi":"there"} }`, true) + testvalid(`{"a":"b","a": 1, "c":{"hi":"there", "easy":["going",{"mixed":"bag"}]} }`, true) + testvalid(`""`, true) + testvalid(`"`, false) + testvalid(`"\n"`, true) + testvalid(`"\"`, false) + testvalid(`"\\"`, true) + testvalid(`"a\\b"`, true) + testvalid(`"a\\b\\\"a"`, true) + testvalid(`"a\\b\\\uFFAAa"`, true) + testvalid(`"a\\b\\\uFFAZa"`, false) + testvalid(`"a\\b\\\uFFA"`, false) + testvalid(string(complicatedJSON), true) + testvalid(string(exampleJSON), true) +} + +var jsonchars = []string{"{", "[", ",", ":", "}", "]", "1", "0", "true", "false", "null", `""`, `"\""`, `"a"`} + +func makeRandomJSONChars(b []byte) { + var bb []byte + for len(bb) < len(b) { + bb = append(bb, jsonchars[rand.Int()%len(jsonchars)]...) + } + copy(b, bb[:len(b)]) +} +func TestValidRandom(t *testing.T) { + rand.Seed(time.Now().UnixNano()) + b := make([]byte, 100000) + start := time.Now() + for time.Since(start) < time.Second*3 { + n := rand.Int() % len(b) + rand.Read(b[:n]) + validpayload(b[:n], 0) + } + + start = time.Now() + for time.Since(start) < time.Second*3 { + n := rand.Int() % len(b) + makeRandomJSONChars(b[:n]) + validpayload(b[:n], 0) + } +} + type BenchStruct struct { Widget struct { Window struct { @@ -465,6 +1071,19 @@ var benchPaths = []string{ "widget.text.onMouseUp", } +var benchManyPaths = []string{ + "widget.window.name", + "widget.image.hOffset", + "widget.text.onMouseUp", + "widget.window.title", + "widget.image.alignment", + "widget.text.style", + "widget.window.height", + "widget.image.src", + "widget.text.data", + "widget.text.size", +} + func BenchmarkGJSONGet(t *testing.B) { t.ReportAllocs() t.ResetTimer() @@ -477,6 +1096,45 @@ func BenchmarkGJSONGet(t *testing.B) { } t.N *= len(benchPaths) // because we are running against 3 paths } +func BenchmarkGJSONGetMany4Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 4) +} +func BenchmarkGJSONGetMany8Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 8) +} +func BenchmarkGJSONGetMany16Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 16) +} +func BenchmarkGJSONGetMany32Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 32) +} +func BenchmarkGJSONGetMany64Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 64) +} +func BenchmarkGJSONGetMany128Paths(t *testing.B) { + benchmarkGJSONGetManyN(t, 128) +} +func benchmarkGJSONGetManyN(t *testing.B, n int) { + var paths []string + for len(paths) < n { + paths = append(paths, benchManyPaths...) + } + paths = paths[:n] + t.ReportAllocs() + t.ResetTimer() + for i := 0; i < t.N; i++ { + results := GetMany(exampleJSON, paths...) + if len(results) == 0 { + t.Fatal("did not find the value") + } + for j := 0; j < len(results); j++ { + if results[j].Type == Null { + t.Fatal("did not find the value") + } + } + } + t.N *= len(paths) // because we are running against 3 paths +} func BenchmarkGJSONUnmarshalMap(t *testing.B) { t.ReportAllocs() @@ -506,6 +1164,34 @@ func BenchmarkGJSONUnmarshalMap(t *testing.B) { t.N *= len(benchPaths) // because we are running against 3 paths } +func BenchmarkGJSONUnmarshalStruct(t *testing.B) { + t.ReportAllocs() + t.ResetTimer() + for i := 0; i < t.N; i++ { + for j := 0; j < len(benchPaths); j++ { + var s BenchStruct + if err := Unmarshal([]byte(exampleJSON), &s); err != nil { + t.Fatal(err) + } + switch benchPaths[j] { + case "widget.window.name": + if s.Widget.Window.Name == "" { + t.Fatal("did not find the value") + } + case "widget.image.hOffset": + if s.Widget.Image.HOffset == 0 { + t.Fatal("did not find the value") + } + case "widget.text.onMouseUp": + if s.Widget.Text.OnMouseUp == "" { + t.Fatal("did not find the value") + } + } + } + } + t.N *= len(benchPaths) // because we are running against 3 paths +} + func BenchmarkJSONUnmarshalMap(t *testing.B) { t.ReportAllocs() t.ResetTimer() @@ -655,22 +1341,85 @@ func BenchmarkFFJSONLexer(t *testing.B) { t.N *= len(benchPaths) // because we are running against 3 paths } -func BenchmarkEasyJSONLexer(t *testing.B) { - skipCC := func(l *jlexer.Lexer, n int) { - for i := 0; i < n; i++ { - l.Skip() +func skipCC(l *jlexer.Lexer, n int) { + for i := 0; i < n; i++ { + l.Skip() + l.WantColon() + l.Skip() + l.WantComma() + } +} +func skipGroup(l *jlexer.Lexer, n int) { + l.WantColon() + l.Delim('{') + skipCC(l, n) + l.Delim('}') + l.WantComma() +} +func easyJSONWindowName(t *testing.B, l *jlexer.Lexer) { + if l.String() == "window" { + l.WantColon() + l.Delim('{') + skipCC(l, 1) + if l.String() == "name" { l.WantColon() - l.Skip() - l.WantComma() + if l.String() == "" { + t.Fatal("did not find the value") + } } } - skipGroup := func(l *jlexer.Lexer, n int) { +} +func easyJSONImageHOffset(t *testing.B, l *jlexer.Lexer) { + if l.String() == "image" { l.WantColon() l.Delim('{') - skipCC(l, n) - l.Delim('}') - l.WantComma() + skipCC(l, 1) + if l.String() == "hOffset" { + l.WantColon() + if l.Int() == 0 { + t.Fatal("did not find the value") + } + } } +} +func easyJSONTextOnMouseUp(t *testing.B, l *jlexer.Lexer) { + if l.String() == "text" { + l.WantColon() + l.Delim('{') + skipCC(l, 5) + if l.String() == "onMouseUp" { + l.WantColon() + if l.String() == "" { + t.Fatal("did not find the value") + } + } + } +} +func easyJSONWidget(t *testing.B, l *jlexer.Lexer, j int) { + l.WantColon() + l.Delim('{') + switch benchPaths[j] { + case "widget.window.name": + skipCC(l, 1) + easyJSONWindowName(t, l) + case "widget.image.hOffset": + skipCC(l, 1) + if l.String() == "window" { + skipGroup(l, 4) + } + easyJSONImageHOffset(t, l) + case "widget.text.onMouseUp": + skipCC(l, 1) + if l.String() == "window" { + skipGroup(l, 4) + } + if l.String() == "image" { + skipGroup(l, 4) + } + easyJSONTextOnMouseUp(t, l) + } +} +func BenchmarkEasyJSONLexer(t *testing.B) { t.ReportAllocs() t.ResetTimer() for i := 0; i < t.N; i++ { @@ -678,58 +1427,7 @@ func BenchmarkEasyJSONLexer(t *testing.B) { l := &jlexer.Lexer{Data: []byte(exampleJSON)} l.Delim('{') if l.String() == "widget" { - l.WantColon() - l.Delim('{') - switch benchPaths[j] { - case "widget.window.name": - skipCC(l, 1) - if l.String() == "window" { - l.WantColon() - l.Delim('{') - skipCC(l, 1) - if l.String() == "name" { - l.WantColon() - if l.String() == "" { - t.Fatal("did not find the value") - } - } - } - case "widget.image.hOffset": - skipCC(l, 1) - if l.String() == "window" { - skipGroup(l, 4) - } - if l.String() == "image" { - l.WantColon() - l.Delim('{') - skipCC(l, 1) - if l.String() == "hOffset" { - l.WantColon() - if l.Int() == 0 { - t.Fatal("did not find the value") - } - } - } - case "widget.text.onMouseUp": - skipCC(l, 1) - if l.String() == "window" { - skipGroup(l, 4) - } - if l.String() == "image" { - skipGroup(l, 4) - } - if l.String() == "text" { - l.WantColon() - l.Delim('{') - skipCC(l, 5) - if l.String() == "onMouseUp" { - l.WantColon() - if l.String() == "" { - t.Fatal("did not find the value") - } - } - } - } + easyJSONWidget(t, l, j) } } } @@ -764,6 +1462,106 @@ func BenchmarkJSONParserGet(t *testing.B) { } t.N *= len(benchPaths) // because we are running against 3 paths } +func jsoniterWindowName(t *testing.B, iter *jsoniter.Iterator) { + var v string + for { + key := iter.ReadObject() + if key != "window" { + iter.Skip() + continue + } + for { + key := iter.ReadObject() + if key != "name" { + iter.Skip() + continue + } + v = iter.ReadString() + break + } + break + } + if v == "" { + t.Fatal("did not find the value") + } +} + +func jsoniterTextOnMouseUp(t *testing.B, iter *jsoniter.Iterator) { + var v string + for { + key := iter.ReadObject() + if key != "text" { + iter.Skip() + continue + } + for { + key := iter.ReadObject() + if key != "onMouseUp" { + iter.Skip() + continue + } + v = iter.ReadString() + break + } + break + } + if v == "" { + t.Fatal("did not find the value") + } +} +func jsoniterImageOffset(t *testing.B, iter *jsoniter.Iterator) { + var v int + for { + key := iter.ReadObject() + if key != "image" { + iter.Skip() + continue + } + for { + key := iter.ReadObject() + if key != "hOffset" { + iter.Skip() + continue + } + v = iter.ReadInt() + break + } + break + } + if v == 0 { + t.Fatal("did not find the value") + } +} +func jsoniterWidget(t *testing.B, iter *jsoniter.Iterator, j int) { + for { + key := iter.ReadObject() + if key != "widget" { + iter.Skip() + continue + } + switch benchPaths[j] { + case "widget.window.name": + jsoniterWindowName(t, iter) + case "widget.image.hOffset": + jsoniterImageOffset(t, iter) + case "widget.text.onMouseUp": + jsoniterTextOnMouseUp(t, iter) + } + break + } +} + +func BenchmarkJSONIterator(t *testing.B) { + t.ReportAllocs() + t.ResetTimer() + for i := 0; i < t.N; i++ { + for j := 0; j < len(benchPaths); j++ { + iter := jsoniter.ParseString(exampleJSON) + jsoniterWidget(t, iter, j) + } + } + t.N *= len(benchPaths) // because we are running against 3 paths +} var massiveJSON = func() string { var buf bytes.Buffer @@ -802,3 +1600,27 @@ func BenchmarkConvertGetBytes(t *testing.B) { GetBytes(data, "50.widget.text.onMouseUp") } } +func BenchmarkParseUintNumParser(t *testing.B) { + var s = "634866135153775564" + for i := 0; i < t.N; i++ { + parseUint(s) + } +} +func BenchmarkStdlibParseUintNumParser(t *testing.B) { + var s = "634866135153775564" + for i := 0; i < t.N; i++ { + strconv.ParseUint(s, 10, 64) + } +} +func BenchmarkParseIntNumParser(t *testing.B) { + var s = "-634866135153775564" + for i := 0; i < t.N; i++ { + parseInt(s) + } +} +func BenchmarkStdlibParseIntNumParser(t *testing.B) { + var s = "-634866135153775564" + for i := 0; i < t.N; i++ { + strconv.ParseInt(s, 10, 64) + } +} diff --git a/vendor/github.com/tidwall/sjson/README.md b/vendor/github.com/tidwall/sjson/README.md index 249556a..1a7c5c4 100644 --- a/vendor/github.com/tidwall/sjson/README.md +++ b/vendor/github.com/tidwall/sjson/README.md @@ -9,9 +9,11 @@set a json value quickly
-SJSON is a Go package that provides a very fast and simple way to set a value in a json document. The purpose for this library is to provide efficient json updating for the [SummitDB](https://github.com/tidwall/summitdb) project. +SJSON is a Go package that provides a [very fast](#performance) and simple way to set a value in a json document. The purpose for this library is to provide efficient json updating for the [SummitDB](https://github.com/tidwall/summitdb) project. For quickly retrieving json values check out [GJSON](https://github.com/tidwall/gjson). +For a command line interface check out [JSONed](https://github.com/tidwall/jsoned). + Getting Started =============== @@ -154,7 +156,7 @@ value, _ := sjson.Set(`{"name":{"last":"Anderson"}}`, "name.last", "Smith") println(value) // Output: -// {"name":{"first":"Sara","last":"Smith"}} +// {"name":{"last":"Smith"}} ``` Set a new array value: @@ -211,6 +213,63 @@ println(value) // {"friends":["Andy"]} ``` +## Performance + +Benchmarks of SJSON alongside [encoding/json](https://golang.org/pkg/encoding/json/), +[ffjson](https://github.com/pquerna/ffjson), +[EasyJSON](https://github.com/mailru/easyjson), +and [Gabs](https://github.com/Jeffail/gabs) + +``` +Benchmark_SJSON-8 3000000 805 ns/op 1077 B/op 3 allocs/op +Benchmark_SJSON_ReplaceInPlace-8 3000000 449 ns/op 0 B/op 0 allocs/op +Benchmark_JSON_Map-8 300000 21236 ns/op 6392 B/op 150 allocs/op +Benchmark_JSON_Struct-8 300000 14691 ns/op 1789 B/op 24 allocs/op +Benchmark_Gabs-8 300000 21311 ns/op 6752 B/op 150 allocs/op +Benchmark_FFJSON-8 300000 17673 ns/op 3589 B/op 47 allocs/op +Benchmark_EasyJSON-8 1500000 3119 ns/op 1061 B/op 13 allocs/op +``` + +JSON document used: + +```json +{ + "widget": { + "debug": "on", + "window": { + "title": "Sample Konfabulator Widget", + "name": "main_window", + "width": 500, + "height": 500 + }, + "image": { + "src": "Images/Sun.png", + "hOffset": 250, + "vOffset": 250, + "alignment": "center" + }, + "text": { + "data": "Click Here", + "size": 36, + "style": "bold", + "vOffset": 100, + "alignment": "center", + "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;" + } + } +} +``` + +Each operation was rotated though one of the following search paths: + +``` +widget.window.name +widget.image.hOffset +widget.text.onMouseUp +``` + +*These benchmarks were run on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.7.* + ## Contact Josh Baker [@tidwall](http://twitter.com/tidwall)