runtime plugin extensibility

[cjnosal]

Plugins are currently defined within the starboard repo. This adds overhead when creating plugins for new scanners as PRs must be accepted and new builds published before a new plugin can be used.

An alternative approach is to allow new plugins to be defined at deploy time from external codebases. This would allow new plugins to be used and updated without having to modify the starboard repo directly. Decoupling the plugin implementations from the core Starboard framework could help grow an ecosystem of third party plugins.

A possible reference for implementation is Hashicorp's go-plugin framework that allows a plugin to communicate with a host process over RPC by implementing a go interface (e.g. vulnerabilityreport.Plugin).

An incremental path forward could be to add a new plugin to Starboard that can act as a bridge to future out-of-repo plugins (e.g. a hypothetical pkg/plugins/rpc_vulnerabilityreport and appropriate configuration added to the starboard configuration to enable it). This would allow extending functionality without requiring immediate changes to the existing in-repo plugins, but would allow for them to eventually be moved out to their own repos (e.g. starboard-plugin-trivy).

Would the maintainers be interested in exploring this approach, or open to PR?

[danielpacak]

👋 @xtreme-conor-nosal I do agree that such plugins ecosystem as you described would be very beneficial in long term.

However, up until now we haven't seen many PRs with new plugins so adding another abstraction on top of Go interfaces might introduce unnecessary complexity. Especially that currently defined plugin interfaces (configauditreport.Plugin and vulnerabilityreport.Plugin) are just incubating and are subject to frequent changes.

Moving plugins out to their own repositories implies that Starboard and a plugin have different life cycles, which adds up to the complexity and release management to keep them compatible.

Regarding interprocess communications between [let's call it] Starboard core and plugin logic it has to be considered very carefully. The response from a plugin must be reliable and ideally do not block a reconciliation loop for too long. Otherwise Starboard operator will have hard time catching up with incoming work items. With a [local] Go method call we're less prone to such problems.

Having said all that we'd love to continue the discussion and elaborate the definition of plugins as well as plugins ecosystem but in small steps. For example, before we change any code I'd like to see a high level diagram that shows how different components (Starboard operator pod, pods controlled by scan job, and plugin [pods?]) interact and how does that compare to the current (v0.10) deployment.

Besides do you have any idea for new plugins or new types of plugins that you'd like to contribute near-term? If so we can prioritise such RFE differently. /cc @itaysk

[ncdc]

Hi @danielpacak, thanks for the reply! I have a few comments:

adding another abstraction on top of Go interfaces might introduce unnecessary complexity

If you use something like go-plugin, you continue to use the same Go interfaces you currently have (configauditreport.Plugin, vulnerabilityreport.Plugin). Plugin authors would implement these interfaces "like normal." You would need to provide a little "glue" to take an implementation and run it as a plugin, but it's possible to provide an example repo to minimize how much an individual developer/contributor would have to build from scratch (see https://github.com/vmware-tanzu/velero-plugin-example for an example).

Especially that currently defined plugin interfaces ... are subject to frequent changes

This is definitely a consideration. The RPC mechanism between plugin client and plugin server is gRPC with protobuf, so you would need to build the APIs and types to be forwards/backwards compatible (or be explicit about breaking changes).

Moving plugins out to their own repositories implies that Starboard and a plugin have different life cycles, which adds up to the complexity and release management to keep them compatible.

True, although you could keep the "core" plugins (e.g. Trivy, Aqua, etc.) directly in the main Starboard repo and release them together with Starboard proper.

Regarding interprocess communications between [let's call it] Starboard core and plugin logic it has to be considered very carefully. The response from a plugin must be reliable and ideally do not block a reconciliation loop for too long. Otherwise Starboard operator will have hard time catching up with incoming work items. With a [local] Go method call we're less prone to such problems.

go-plugin makes this fairly straightforward. When working on Velero plugins, we did encounter some issues where a plugin crashed (bug in code) and we had to write a framework for detecting plugin crashes & automatically restarting them, but that was the only issue we encountered.

If you're concerned about a plugin invocation taking longer that it should, you could invoke it in a goroutine with a timeout (although based on experience with go-plugin, I wouldn't expect that to be necessary).

For example, before we change any code I'd like to see a high level diagram that shows how different components (Starboard operator pod, pods controlled by scan job, and plugin [pods?]) interact and how does that compare to the current (v0.10) deployment.

Apologies but I don't have time to make a diagram just now, but with go-plugin, each plugin is a standalone executable binary. It needs to be in a path that is accessible to the main executable, whether that's a CLI (local) or a pod (in cluster). With Velero, we "attach" plugins to the pod as init containers, and have a small entrypoint whose sole task is to copy the plugin binary from the init container's file system into a path that is accessible to both the init container and the Velero container (i.e. a Kubernetes volume that's mounted into both).

I'd be happy to chat about this in more detail if you'd like. I'm also ncdc on the Kubernetes Slack. Thanks!