PROPOSAL: add a fallthrough! method for continuing to check later rules

[jneen]

Usage:

class MyLexer < Rouge::RegexLexer
  state :root do
    rule %r/\w+\b/ do |m|
      # aborts the current block and continues as if the regex for this rule did not match
      fallthrough! unless SOME_SET.include?(m[0].downcase)
      token Name::Builtin
    end

    rule %r/\w+\b/ do
      # will end up here if the previous rule called #fallthrough!
      # ...
    end
  end
end

[jneen]

The purpose of this is to allow for more easily removing massive regexes as in e.g. the Ruby lexer. This way, runtime code can more simply test set inclusion while maintaining the same semantics as the giant regex.