Plugin system for extending cucumber-js

[davidjgoss]

🤔 What's the problem you're trying to solve?

If you want to augment the core functionality of Cucumber - for something that's not a formatter - you have two choices:

1. Push it into the core (i.e. what has been done for publishing to the reports service)
2. Wrap Cucumber, influencing its inputs and acting on its outputs (i.e. what has been started for TCP-based ordering)

Neither of these options are great for the average user with a good idea, and both of these specific use cases would be better served by an official plugin mechanism so you can hook into Cucumber's test run lifecycle, change things, and react to things.

✨ What's your proposed solution?

TBA!

At a basic level we need two things:

A way to write a plugin

The bare minimum might be something like:

export default {
  async init() {}
  onMessage(envelope) {}
}

We'll also likely need:

• A way to provide configuration for/to the plugin
• A way for the plugin to output to stderr (e.g. the publish banner)
• "Lifecycle hooks" so plugins can provide a function which can influence core logic - so for the TCP case something like async inferOrder(pickles: Pickle[]): Promise<Pickle[]>

A way to reference a plugin

Something like:

npx cucumber-js --plugin @cucumber/publish

---

I think a good start would be to create some plumbing for plugins internally, and reimplement the publish functionality in terms of that. I'm happy to give that a go.

[aurelien-reeves]

I like the idea, and I agree with your analysis and choices 👌

[mattwynne]

I think an approach where we try to identify and refactor existing/internal code towards using an API boundary that we could stabilise and make public sounds great. Happy to pair with you on this @davidjgoss if you want to bounce some ideas around.

Not necessarily solving the same problem but worth adding for context:

When @tooky and I re-wrote the guts of cucumber-ruby, we ended up with a "plugin" mechanism based on the chain-of-responsibility design pattern. Express's request handlers are another example of the same pattern. Each plugin (or "filter" as we called them) receives the stream of test cases (like a pickle, but with extra info about how to invoke it) and is given a receiver (the next filter in the chain) to whom it should pass on whichever test cases it chooses. It can filter out test cases by choosing not to pass them on, and it can also enrich or otherwise mutate test cases before passing them on.

A lot of code fell into place when we adopted this model, allowing us to assemble most of Cucumber's behavour from lots of small pieces, e.g:

• filtering test cases by name
• filtering and sorting test cases by location
• glueing step definition functions to the test case's steps
• randomizing test cases
• initializing the world

I'm not sure if you already have something like this inside cucumber-js, but it might be worth thinking about as part of this. I'm pretty sure it would provide what the TCP extension needs, for example.

[mattwynne]

https://github.com/cucumber/cucumber-electron is another piece worth considering in all this, currently implemented as a wrapper but we agreed (in theory) that it should really be part of this codebase.

[mattwynne]

More prior art is the https://github.com/cucumber/cucumber-ruby-wire/blob/main/lib/cucumber/wire.rb extension for cucumber-ruby which does use an officially-supported (though poorly documented / publicised) plugin API.

[charlierudolph]

Formatters could also be a special form of a plugin. Maybe have someway for the plugin to declare its a formatter and thus to get passed a function to log (which will be either a file stream or stdout)

In general, like the idea. Very curious what exactly will be the scope of what get supported

[davidjgoss]

Formatters could also be a special form of a plugin. Maybe have someway for the plugin to declare its a formatter and thus to get passed a function to log (which will be either a file stream or stdout)

I really like this idea. We could definitely make it easier than the current class-based formatter system. And providing a function to write without direct access to the stream would be an improvement too.

[davidjgoss]

The plumbing for plugins is moving along, but specifically on formatters I've started to build out the new style of formatter and converted a couple of low-stakes existing ones in #2400.

I wanted to cc @badeball and @vitalets here as the aim here is to ultimately have all our built-in formatters and related utilities available in library form, and therefore usable in any Cucumber implementation without dragging in cucumber-js itself as a dependency. Thoughts welcome on the direction here!

[vitalets]

@davidjgoss thanks for notifying on that. I like this new approach.
One question is about library form of formatters. For example, html formatter has a package @cucumber/html-formatter and in your pr there is a wrapper inside Cucumber export default { type: 'formatter', ... }. So in the long-term is it expected that all formatters will have a library and a wrapper? Or formatter libraries will eventually embed that wrapper and provide FormatterPlugin compatible api? So that in custom Cucumber implementation we can utilize any formatter in the same form like this:

import htmlFormatter from '@cucumber/html-formatter';

messagesStream.on('message', msg => htmlFormatter.write(msg)); // pseudo code

[davidjgoss]

Yep it's as you lay out in your example pseudo-code, I want to shift the formatter implementations left into the library so they each provide an export that can be used in any implementation without trapping any of that in cucumber-js. At the moment html-formatter is quite fragmented with the implementation split across the two codebases but I'm working on fixing that, in fact cucumber/html-formatter#320 is the first small step there to free up the default export.

[badeball]

Thoughts welcome on the direction here!

I like the idea of moving in a library direction - seems like a move to have somewhat of first class support for other implementations such as the Cypress preprocessor.

I'm currently using the JSON formatter, the HTML formatter and the pretty formatter. I'd be happy to test release condidates, as I have a pretty good regression test suite.

[davidjgoss]

I just raised a PR that refactors the publish functionality to be a plugin (only an internal concept for now). So far this is just a plugin that passively listens for messages and does stuff outside of Cucumber off the back of that, but it's a good draft to drive some more discussion.

[davidjgoss]

As I start to experiment with other use cases for this, it occurs to me that a plugin, much like a formatter, can only really operate at the "coordinator" level, since test execution often happens in a worker process (via the parallel runtime). This is kind of similar to Cypress, where a plugin operates within the Node.js process, and anything you want to happen in the browser is done via support code.

I don't think this is a problem per se, or that we should change direction. I think most of the interesting plugin use cases are coordinator-level. But I think long term we'll want to come up with a nice solution for:

• Causing extra support code to be loaded from a plugin (maybe via an extension point to modify the configuration before code is loaded?)
• Communicating with a plugin from support code

[davidjgoss]

I've started tagging issues with plugins-usecase where it's not something that'd be a core feature but we could potentially support with plugins: https://github.com/cucumber/cucumber-js/issues?q=is%3Aopen+is%3Aissue+label%3Aplugins-usecase