Centralizing Writes

[cosmicac]

Hi all,

First of all, a big thank you to all of you for all of your hard work maintaining Sequel, we’ve been using it at @affinity and it’s been awesome.

However, I’m writing in today in hopes of receiving some guidance on a bit of an unusual challenge we’re facing with Sequel.

Setup

We’re using Sequel 5.51.0, with Postgres 13.4.

Problem

We’re looking for a way to force writes on a set of table(s) to always go through a certain codepath. Basically, the way our app architecture is setup right now, we’d like to guarantee that every write to a set of table(s) performs another set of arbitrary side effects. Historically, we’ve used model hooks to try to enforce side effects in this manner, but as stated in the documentation, there are lots of ways to bypass this (via Datasets, etc.). At this point, we're looking for something with stronger guarantees than model hooks.

An Idea

Something we’re playing around with is implementing a plugin that overwrites all of the methods that perform writes, and forces them to use another class that we’ve written instead which will perform both the write (using Sequel normally) and any desired side effects. For example, the pseudocode would look something like this (please note, this isn't a working example, many details are left out):

module WriteCentralizationPlugin
  module ClassMethods
    def create(values = Sequel::OPTS, &block)
      if called_from_within_custom_writer
        # If this was called from within the custom writer object, just use Sequel normally.
        super(values, &block)
      else
        # Perform a write through a custom writer object, which uses Sequel normally under the hood.
        # It will also trigger the desired side effects.
        CustomWriter.write
      end
    end

    # Other class methods that write go here, following a similar pattern.

  end

  module InstanceMethods
    def save(opts = Sequel::OPTS)
      if called_from_within_custom_writer
        # If this was called from within the custom writer object, just use Sequel normally.
        super(opts)
      else
        # Perform a write through a custom writer object, which uses Sequel normally under the hood.
        # It will also trigger the desired side effects.
        CustomWriter.write
      end
    end

    # Other instance methods that write go here, following a similar pattern.
  end

  module DatasetMethods
    def multi_insert(hashes, opts = Sequel::OPTS)
      if called_from_within_custom_writer
        # If this was called from within the custom writer object, just use Sequel normally.
        super(hashes, **opts)
      else
        # Perform a write through a custom writer object, which uses Sequel normally under the hood.
        # It will also trigger the desired side effects.
        CustomWriter.write
      end
    end

    # Other dataset methods that write go here, following a similar pattern.
  end
end

A problem with the solution above is that some of these Sequel methods call other methods - for example the create ClassMethod calls the the save InstanceMethod, etc. For these "internal" Sequel calls, we'd like to preserve the original behavior (i.e. just call super). Detecting these "internal" Sequel calls without modifying a ton of Sequel code is proving to be a little unwieldy and challenging.

So, the question is - what is the best way, using Sequel, to guarantee that all writes to a set of table(s) go through a single codepath?

Has this been done before? Is there anything existing in the Sequel ecosystem that tries to tackle this problem? Appreciate any and all thoughts!

[jeremyevans]

Sequel doesn't have anything built in that will do what you want. It sounds like what you want is probably best handled by a database trigger, assuming that whatever CustomWriter.write can be handled completely inside the database. If not, you could try hooking into the execute_ddl, execute_dui, execute_insert, returning_fetch_rows dataset methods. Most code paths that modify the dataset are probably going through one those 4 methods. However, I cannot guarantee that all modifying code paths go through them, and it certainly won't catch things like: DB.run('DELETE FROM table WHERE ...'). If you want to be 100% sure, use a database trigger.

[cosmicac]

Thanks for the advice, much appreciated. Unfortunately, some of our desired side effects cannot be handled completely in the database, so database triggers won't suffice.

For those four methods you mentioned, would there be any way of telling what the overall operation/top-level-method is short of parsing the call stack or parsing the SQL?

For example, in execute_dui, how would one tell if it's being called in a multi_insert, or a delete?

[jeremyevans]

I wouldn't recommend parsing the call stack or the SQL. If you need to know what the top-level method is being called, it's probably best to override all of the top-level methods to store that information, similar to the example code you provided initially.