Installation

Requirements

1. Java 7
2. HBase 0.98

Quickstart

Download and install the required HBase version. You can find HBase distributions in this page.

Clone the Omid repository and build the TSO package:

$ git clone [email protected]:yahoo/omid.git
$ cd omid
$ mvn clean install assembly:single

This will generate a binary package containing all dependencies for the TSO in tso-server/target/tso-server--bin.tar.gz.

Alternatively you can download the version of the tar.gz package from the [release repository](TODO -> add link).

Extract the TSO package on your machine:

$ tar zxvf tso-server-<VERSION>-bin.tar.gz
$ cd tso-server-<VERSION>

Ensure that the setting for hbase.zookeeper.quorum in conf/hbase-site.xml points to your zookeeper instance, and create the commit and timestamp tables:

$ bin/omid.sh create-hbase-commit-table -numSplits 16
$ bin/omid.sh create-hbase-timestamp-table

Finally, start the TSO server:

$ bin/omid.sh tso

By default the tso listens on port 54758.

HBase Client usage

Create a new Java application and add the right version of the hbase-client jar as a dependency. For example in a Maven-based app add the following dependency in the pom.xml file:

<dependency>
   <groupId>com.yahoo.omid</groupId>
   <artifactId>hbase-client</artifactId>
   <version>${hbase_client.version}</version>
</dependency>

The client interfaces for Omid are TTable and TransactionManager (These interfaces will likely change slightly in future.):

1. The TransactionManager is used for creating transactional contexts, that is, transactions. A builder is provided in the HBaseTransactionManager class in order to get the TransactionManager interface.
2. TTable is used for putting, getting and scanning entries in a HBase table. TTable's interface is similar to the standard HTableInterface, and only requires passing the transactional context as a first parameter in the transactional aware methods (e.g. put(Transaction tx, Put put))

Below is provided a sample application accessing data transactionally. To run the application, make sure core-site.xml and hbase-site.xml for your HBase cluster are present in your CLASSPATH. You will need to set tso.host and tso.port appropriately. Also, you will need to create a hbase table "EXAMPLE_TABLE", with column family "EXAMPLE_CF", and with TTL disabled and maxVersions set to Integer.MAX_VALUE. This example assumes non-secure communication with HBase. If your HBase cluster is secured with Kerberos, you will need to use the UserGroupInformation API to log in securely.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;
import com.yahoo.omid.transaction.HBaseTransactionManager;
import com.yahoo.omid.transaction.TTable;
import com.yahoo.omid.transaction.Transaction;
import com.yahoo.omid.transaction.TransactionManager;


public class Example {
    public static void main(String[] args) throws Exception {
        Configuration conf = HBaseConfiguration.create();

        TransactionManager tm = HBaseTransactionManager.newBuilder()
                                                       .withConfiguration(conf)
                                                       .build();

        TTable tt = new TTable(conf, "EXAMPLE_TABLE");
        byte[] exampleRow1 = Bytes.toBytes("EXAMPLE_ROW1");
        byte[] exampleRow2 = Bytes.toBytes("EXAMPLE_ROW2");
        byte[] family = Bytes.toBytes("EXAMPLE_CF");
        byte[] qualifier = Bytes.toBytes("foo");
        byte[] dataValue1 = Bytes.toBytes("val1");
        byte[] dataValue2 = Bytes.toBytes("val2");

        Transaction tx = tm.begin();
        Put row1 = new Put(exampleRow1);
        row1.add(family, qualifier, dataValue1);
        tt.put(tx, row1);
        Put row2 = new Put(exampleRow2);
        row2.add(family, qualifier, dataValue2);
        tt.put(tx, row2);
        tm.commit(tx);

        tt.close();
        tm.close();
    }
}

You can find a detailed explanation of the interfaces in the Basic Examples section.

Omid Compactor Coprocessor

Omid includes a jar with an HBase coprocessor for performing data cleanup that operates during compactions, both minor and major. Specifically, it does the following:

• Cleans up garbage data from aborted transactions
• Purges deleted cells. Omid deletes work by placing a special tombstone marker in cells. The compactor detects these and actually purges data when it is safe to do so (i.e. when there are no committable transactions that may read the data).
• 'Heals' committed cells for which the writer failed to write shadow cells.

To deploy the coprocessor, the coprocessor jar must be placed in a location (typically on HDFS) that is accessible by HBase region servers. The coprocessor may then be enabled on a transactional table by the following steps:

1. Add a co-processor specification to the table via a "co-processor attribute". The co-processor spec may (and usually will) also include the name of the Omid commit table.
2. Add an "OMID_ENABLED => true" flag to any column families which the co-processor should work on.
3. Disable and re-enable the table.

Sample coprocessor enabled table:

'omid_enabled_table', {TABLE_ATTRIBUTES =>
   {coprocessor$1 => 'hdfs:///hbase/co_processor/omid/omid2-hbase-coprocessor-2.2.11.jar|
                      com.yahoo.omid.transaction.OmidCompactor|1001|
                      omid.committable.tablename=ns:OMID_COMMIT_TABLE},
   {NAME => 'n', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
    COMPRESSION => 'LZO', VERSIONS => '2147483647', TTL => 'FOREVER', MIN_VERSIONS => '0',
    KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'true', BLOCKCACHE => 'true'},
   {NAME => 's', DATA_BLOCK_ENCODING => 'PREFIX', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
    COMPRESSION => 'LZO', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0',
    KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true',
    METADATA => {'OMID_ENABLED' => 'true'}}