Hello Teknek! Building your first application with SOL

Prereq

This example requires teknek to be set up. You can follow the instructions for teknek-stream-stack if you want a fast setup.

Stream Operator Language (SOL)

SOL (aka sh*t outta luck) was designed to allow real time development of streaming applications. SOL is accessible both by a command line interface and a web interface.

You can quickly fire up the web interface using jetty-runner:

cd teknek-web
java -jar jetty-runner-9.1.0.v20131115.jar target/teknek-web.war
2013-12-22 09:33:54.880:INFO:oejr.Runner:main: Runner
2013-12-22 09:33:58.745:INFO:oejs.ServerConnector:main: Started ServerConnector@1759817d{HTTP/1.1 {0.0.0.0:8080}

The important thing on this page is to enter your zookeeper information in the text box. In this case localhost:2181

After you submit the zookeeper information, click on the SOL Shell link.

This should have just blown your mind. SOL is an interactive shell over an ajax web page!

Real time development

Teknek enables real time development by allowing code to be plugged in at runtime. There are several ways to do this, inline groovy, url class loading, etc. For this example we will use tekneks ability to import groovy code from urls.

teknek> import https://raw.github.com/edwardcapriolo/teknek/master/teknek-core/src/test/resources/bundle_io.teknek_itests1.0.0.json

Imagine when everyone starts using teknek and you can load up things like twitter feeds or sentiment analysis operators from someone elses github!

This plan will consist of two parts GTry is a feed that produces 5 partitions of 1000 tuples each. groovy_identity is an operator that prints a tuple before passing it along.

teknek> create plan y
plan> load io.teknek GTry feed as GTry
feed> exit
plan> load io.teknek groovy_identity operator as groovy_identity
operator> exit
plan> set root groovy_identity
plan> set maxworkers 1
plan> set tupleRetry 1
plan> set offsetCommitInterval 1

You can always run show formatted plan to see what the Plan looks like in json form.

plan> show formatted plan
{
   "name" : "y",
   "feedDesc" : {
   "spec" : "groovy",
   "script" : "import io.teknek.feed.*...",
   "theClass" : "GTry",
   "properties" : {
       "number.of.partitions" : 5,
       "number.of.rows" : 1000
    },
    "name" : "GTry"
 },
 "rootOperator" : {
    "spec" : "groovyclosure",
    "script" : "{ tuple, collector ->  collector.emit(tuple) ; println(tuple) }",
    "theClass" : "groovy_identity",
    "name" : "groovy_identity",
    "parameters" : { },
    "children" : [ ]
  },
  "offsetStorageDesc" : null,
  "disabled" : false,
  "maxWorkers" : 4,
  "tupleRetry" : 1,
  "offsetCommitInterval" : 1
}

Now your plan is ready to run. Typing save creates the zookeeper entry.

plan> save

If you are running an instance of TeknekDeamon (which you should be if you are running the stream-stack) it will notice the changes to zookeeper and attempt to start the plan up.

{x=970}
DEBUG 20:50:46,999 No children operators for this operator. Tuple not being passed on {x=971}
{x=971}
DEBUG 20:50:46,999 No children operators for this operator. Tuple not being passed on {x=972}
{x=972}
DEBUG 20:50:46,999 No children operators for this operator. Tuple not being passed on {x=973}
{x=973}
DEBUG 20:50:46,999 No children operators for this operator. Tuple not being passed on {x=974}
{x=974}
DEBUG 20:50:46,999 No children operators for this operator. Tuple not being passed on {x=975}
{x=975}
DEBUG 20:50:47,000 No children operators for this operator. Tuple not being passed on {x=976}

That's it! Your first teknek application! Cool, but it just used static data! Take a look at Reading data from Kafka to see how Teknek can process streams of data from Kafka.