Schema Registry - a web service to manage schemas of Protocol Buffers, Avro, Thrift etc.
Prerequisites: perl JSON module, JDK, protobuf's compiler protoc and thrift's compiler thrift in PATH.
-
generate Makefile to download and compile schema files
perl src/main/scripts/gen-makefile.pl schemas.json > Makefile -
download and compile schema files into generated/ directory
mvn dependency:copy-dependencies make -
build this web service
mvn package
Edit schemas.json, its own schema is obvious, only a little explanation about "filename".
"filename" means the local relative path where the downloaded schema file is stored, usually it's just a file name without any directory part. But for protobuf it's a little complicated, protoc stores the .proto file path in command line into generated Java source file, and Descriptors.java[1] checks it against the "import" clause in .proto file, this means the "filename" field in schemas.json is decided by how other .proto files import a .proto file, this may be a bug, or a feature by design. For thrift, it's similar.
Example:
B.proto:
import "A.proto"; // "filename" for A.proto must be "A.proto"
C.proto:
import "protos/A.proto"; // "filename" for A.proto must be "protos/A.proto"
Put a "SchemaRegistry.properties" to /home/y/conf/SchemaRegistry/ or the home directory of Tomcat process owner.
schemaList=/path/to/schemas.json
rootDirectory=/path/to/generated/
reloadInterval=5 # seconds to check modification time of schema list and reload, default is 5
Then put the SchemaRegistry.war file into Tomcat's webapp directory and restart Tomcat if necessary.
-
list all schemas: http://localhost:8080/SchemaRegistry/i
-
get info for a schema: http://localhost:8080/SchemaRegistry/i/{ID}
-
get content for a schema: http://localhost:8080/SchemaRegistry/s/{ID}
-
serialize message: post data to http://localhost:8080/SchemaRegistry/e/{ID}?m={MessageName}&f={filters}
-
"m={MessageName}", optional, used to distinguish multiple messages defined in single .proto file.
-
"f={filters}", optional, filters applied to response, can be combination of base64,base64raw,bzip2,deflate,gzip,lz4,lzf,snappy separated by comma, the order is critical. Notice the filters are applied in reverse order.
-
"protobuf.delimited=true", optional, specific to Protobuf, indicate the input is separated by blank line and contains multiple messages.
-
"thrift.protocol=compact", optional, specific to Thrift, indicate to use compact protocol.
-
"avro.payload=file", optional, specific to Avro, indicate to write Avro DataFile format.
-
"avro.codec=null|deflate|bzip2|snappy", optional, specific to Avro, applied when avro.payload=file, indicate to choose which org.apache.avro.file.CodecFactory
curl -s --data-binary 'id: 1 name: "Jack"' 'http://localhost:8080/SchemaRegistry/e/protobuf-example-addressbook?m=Person'
-
-
deserialize message: post data to http://localhost:8080/SchemaRegistry/d/{ID}?m={MessageName}&f={filters}
-
"m={MessageName}", optional, used to distinguish multiple messages defined in single .proto file.
-
"f={filters}", optional, filters applied to request, can be combination of base64,base64raw,bzip2,deflate,gzip,lz4,lzf,snappy,skipN separated by comma, the order is critical.
- "skipN", N = 0 means skipping from beginning to first zero byte, N > 0 means skipping N bytes.
-
"protobuf.delimited=true", optional, specific to Protobuf, indicate the output is separated by blank line and contains multiple messages.
-
"thrift.protocol=compact", optional, specific to Thrift, indicate to use compact protocol.
-
"avro.payload=file", optional, specific to Avro, indicate to read Avro DataFile format.
curl -s --data-binary 'id: 1 name: "Jack"' 'http://localhost:8080/SchemaRegistry/e/protobuf-example-addressbook?m=Person' | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/protobuf-example-addressbook?m=Person' curl -s --data-binary 'id: 1 name: "Jack"' 'http://localhost:8080/SchemaRegistry/e/protobuf-example-addressbook?m=Person&f=base64' | base64 --decode | base64 | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/protobuf-example-addressbook?m=Person&f=base64' curl -s --data-binary 'id: 1 name: "Jack"' 'http://localhost:8080/SchemaRegistry/e/protobuf-example-addressbook?m=Person&f=base64,snappy' | base64 --decode | base64 | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/protobuf-example-addressbook?m=Person&f=base64,snappy' curl -s --data-binary 'id: 1 name: "Jack"' 'http://localhost:8080/SchemaRegistry/e/protobuf-example-addressbook?m=Person&f=base64,deflate,gzip,bzip2,lz4,lzf,snappy' | base64 --decode | base64 | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/protobuf-example-addressbook?m=Person&f=base64,deflate,gzip,bzip2,lz4,lzf,snappy' # Schema at http://avro.apache.org/docs/current/gettingstartedjava.html#Defining+a+schema curl -s --data-binary '{"name": "Alyssa", "favorite_number": {"int": 256}, "favorite_color": {"string": "red"}}' \ http://localhost:8080/SchemaRegistry/e/avro-example-user?f=base64,snappy | curl -s --data-binary @- http://localhost:8080/SchemaRegistry/d/avro-example-user?f=base64,snappy curl -s --data-binary '{"name": "Alyssa", "favorite_number": {"int": 256}, "favorite_color": {"string": "red"}} {"name": "Alyssa", "favorite_number": {"int": 256}, "favorite_color": {"string": "red"}}' \ 'http://localhost:8080/SchemaRegistry/e/avro-example-user?f=base64,snappy&avro.payload=file&avro.codec=deflate' | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/avro-example-user?f=base64,snappy&avro.payload=file' # Schema at https://git-wip-us.apache.org/repos/asf?p=thrift.git;a=blob_plain;f=tutorial/tutorial.thrift curl -s --data-binary '{"1":{"i32":111},"2":{"i32":222},"3":{"i32":4},"4":{"str":"hello world"}}' \ 'http://localhost:8080/SchemaRegistry/e/thrift-tutorial-tutorial?f=base64' | curl -s --data-binary @- 'http://localhost:8080/SchemaRegistry/d/thrift-tutorial-tutorial?f=base64'
-