Data Exporter Client

This project is the Java reference implementation of a Data Exporter Client.

The Flow Forwarder Docker Container on the Flow Collector listens to Flow events from the Engine via ZeroMQ. It sends stitched, de-duplicated flow records over a secure web socket to registered clients.

The flow records are sent once per minute as a java.io.ByteBuffer containing Google Protocol Buffer representations of de-duplicated, stitched netflows.

Dependencies

• Java 8 JDK
• Maven 3.3.9
• bash
• openssl

Java Cryptography Extension (JCE)

Download the JCE.

Unzip the downloaded zip.

Copy local_policy.jar and US_export_policy.jar to the $JAVA_HOME/jre/lib/security directory.

(Note: these jars will be already there so you have to overwrite them).

Setup Instructions

Build the Data Exporter Application

mvn clean package

The Flow Collector requires clients to connect over a secure web socket connection. This means you need to:

1. Create a Certificate Authority
2. Create Data Exporter keys.
3. Prepare a Certificate Signing Request for the Data Exporter Keys.
4. Sign the Data Exporter public key by the Certificate Authority.
5. Create a Trust Store with the Flow Collector's certificate for the Data Exporter.
6. Provision the Flow Collector with the Certificate Authority's certificate.

The bin/generate-certs-and-keystore script accomplishes steps 1-5. All created certs appear in the ./certs directory.

$ bin/generate-certs-and-keystore flow-collector-ip-or-hostname

Import certs/data-exporter-certificate-authority.crt into the Flow Collector via the Configuration -> Certificate Authority Certificates menu.

The script generates these files:

• data-exporter-certificate-authority.key – the certificate authority key

• data-exporter-certificate-authority.crt – the certificate authority certificate imported into Stealthwatch.

• data-exporter.key – the key of the data exporter application

• data-exporter.csr – the certificate signing request used to ask the certificate authority for a certificate

• data-exporter.crt – the data exporter certificate signed by the certificate authority

• data-exporter.pkcs12 – the data exporter’s key and certificate stored in pkcs12 format

• {flow-collector-alias}.crt – the flow collector's certificate

• data-exporter-truststore.pkcs12 – the trust store holding the flow collector’s certificate.

The data-exporter.pkcs12 and data-exporter-truststore.pkcs12 are used by log-flows to setup the secure web socket connection.

Stealthwatch prior to 7.0

Import

certs/data-exporter-certificate-authority.crt

into the Flow Collector via the Configuration -> Certificate Authority Certificates menu.

Stealthwatch 7.0 and after

Import

certs/data-exporter-certificate-authority.crt
certs/data-exporter.crt 

into the Flow Collector via the Central Management.

Running the application

The bin/log-flows script runs the application using the certificates created above. It requires the Flow Collector host name or IP address:

bin/log-flows 10.0.37.14

Once the Data Exporter is running the ./log directory is created, and two log files are created: * system.log holds operational and error messages. * messages.log holds all flow record messages from the flow collectors(s).

Both the system and message log output appear in the console. To change this behavior, modify the ./src/main/resources/log4j.properties file.

Protocol Buffers

The protocol buffers used for reading flows are stored in src/main/resources/protos. These get compiled into java classes during the package life cycle via com.google.protobuf.protoc for the local operating system..

The code generated by the protocol buffer compiler is under the target directory.

Flow Collector Notes

The Flow Collector has a Docker Container Service named "Data Exporter", that is is responsible for for exporting flows. Use the Flow Collector admin UI to start or stop the container.

The actual container name is flow-forwarder. Its logs are found in /lancope/var/logs/containers/docker/flow-forwarder.log.

The flow-forwarder waits for websocket connection attempts on wss://<hostname-or-ip>/flowforwarder/websocket or ws://<hostname-or-ip>:8092/websocket.

NOTE: port 8092 is closed by default on the flow collector - only connections originating from the Flow Collector itself are supported.