New filter plugin: logstash-filter-lrucache

[sw-jung]

I want contribute my new plugin logstash-filter-lrucache to logstash-plugins.

This plugin provides caching using the LRU(Least Recently Used) algorithm.

It based on lru_redux.

Example of usage

filter {
  # Find `blacklisted_at` value for particular `host` in cache.
  lrucache {
    namespace => "blacklisted_at"
    action => "get"
    key => "%{host}"
    target => "blacklisted_at"
  }

  if ![blacklisted_at] {
    # If no value is found in cache, Query to elasticsearch.
    elasticsearch {
      query => "host:%{host}"
      index => "blacklist"
      fields => { "@timestamp" => "blacklisted_at" }
    }

    if [blacklisted_at] {
      # Store query result into cache.
      lrucache {
        namespace => "blacklisted_at"
        action => "set"
        key => "%{host}"
        value => "%{blacklisted_at}"
      }
    }
  }

  if [blacklisted_at] {
    # Drop incoming log from blacklisted host.
    drop {}
  }
}

Links

• github: https://github.com/sw-jung/logstash-filter-lrucache
• rubygems: https://rubygems.org/gems/logstash-filter-lrucache

[sw-jung]

This is a plugin created because I can't wait for the logstash-elasticsearch-filter's cache support. ( See this issue)

But I don't like the structure of this plugin.

At first I wanted to create a plugin that looks like this:

filter {
  memoize {
    key => "%{cache_key}"
    fields => ["fields_for_result_expected"]
    filter => elasticsearch {
      # ...
    }
  }
}

But I failed. because grammer parsers always parse nested plugins as codecs!

{ "filter" => plugin("codec", "elasticsearch", ...) }

Do anyone have a solution to this problem?

[sw-jung]

I'm sorry, but I've found a solution to the above problem so I'll re-contribute after implementation new. It expect looks like this:

filter {
  memoize {
    key => "%{cache_key}"
    fields => ["fields_for_result_expected"]
    filter_name => "elasticsearch"
    filter_body => { ... }
  }
}