Skip to content

mv cpp pb timeseries1

Matthew Von-Maszewski edited this page Dec 4, 2015 · 3 revisions

Status

  • merged to master -
  • code complete - December 4, 2015
  • development started - October, 2015

History / Context

This branch contains the Erlang layer of a C++ NIF for riak_pb_codec.erl. A detail discussion of the C++ portion is found here:

https://github.com/basho/riak_kv_ts/wiki/mv-cpp-pb-timeseries1

Engel Sanchez performed detail analysis of bottlenecks when streaming sample Riak CS data from a server to a client. The key bottleneck was the heap thrashing within Erlang during protocol buffer encoding/decoding. The thrashing is especially harsh when a given protocol buffer message contains many rows of data.

This branch ties processing of two Erlang protocol buffer messages within riak_pb_codec.erl to equivalent C++ functions in the riak_pb_codec.so shared library file.

This branch adds the riak_kv_ts repository as a subdirectory of riak_pb. The git submodule mechanism handles the creation of the subdirectory and retrieval of appropriate code. The intention of using a submodule is to allow the NIF to be easily included or excluded from various Riak products. The riak_kv_ts repository is private, not available as open source.

Branch Description

.gitmodules

This file activates or deactivates the use of riak_kv_ts in a product's build. A blank file does not retrieve or build the NIF.

Makefile

New rules exist at the end of the file to see whether or not riak_kv_ts exists in this build. If so, Makefile in riak_kv_ts is activated.

rebar.config

A rebar build does not use the Makefile directly. The pre-compile hook will let the riak_pb/Makefile decide whether or not riak_kv_ts exists and build it if it does.

Similarly the post-deps hook will activate and retrieve the riak_kv_ts repository if the .gitmodules lists it.

riak_pb_codec.erl

The new init()/0 function contains standard steps to potentially load a parallel NIF for this .erl file.

encode_tsputreq() contains the same steps that would have happened in a pure Erlang implementation. These steps still occur if the NIF is not included with the product. However, the equivalent to encode_tsputreq() within the C++ NIF is called INSTEAD if the NIF loads correctly via the init()/0 function.

Similarly decode_tsqueryresp() exists to let the NIF overlay only the decode of a TsQueryResp message.