Allow easy development of plugins locally

[jgbernalp]

Currently Perses plugins use module federation to share dependencies and allow them to be injected into Perses UI (or other clients). In order to make development simple we can use the development server from the plugin side and adjust Perses Backend to run in a "plugin development mode" that will proxy requests of the /plugin endpoint to the plugin development server. As follows:

Remaining work:

• Create a single server to be able to host several plugins at once: currently the dev server runs per plugin: Add a single dev server for all plugins #58
• Adjust Perses backend to include development plugins when listing: /api/plugins: [ENHANCEMENT] Improve the way the plugins are loaded by considering the local path where they are stored perses#2626
• Adjust Perses backend to proxy plugin files request to the plugin dev server: [ENHANCEMENT] Improve the way the plugins are loaded by considering the local path where they are stored perses#2626

Open questions:

• Can we load dynamically cue files on the backend while developing plugins?

[Nexucis]

when you are referring to a dev server are you talking about the one created when running npm run dev ?

Can we load dynamically cue files on the backend while developing plugins?

Not yet but yeah that will be the plan

[jgbernalp]

when you are referring to a dev server are you talking about the one created when running npm run dev ?

Yes from the plugin side

[Nexucis]

During the development of a plugin, it's not that simple to setup the Perses server(with the Cuelang schema and so on). So I was thinking about creating a light Perses.

• It will only contain the dashboard
• No permission setup
• It will be available on perses.dev, so you can use it to see your plugin --> which means to create a dedicated session when you are connecting to it. I have a vague idea about what would mean to create a session. I was thinking about having a dedicated UUID that will be in every URL. Like that you can only load the plugin you are providing to the remote server.
• The CLI will make a bridge between the remote server and the plugin dev server. It will scan the schema available and the URL of the dev server based on rspack config and push this info to the remote server. Like that you won't need to change or adapt the config. You will just have to start the dev server and the CLI and that's it.

What do you all think about this idea ?

[jgbernalp]

Not sure if having everyone pushing to a server in perses.dev would be ideal but it could be an option. For example you might want to test a plugin with a specific Perses version or would like to test locally without access to external systems.

But I like the idea of the CLI taking care of the heavy lifting. You could also start Perses from a Dockerfile, or directly from the code, and have the CLI connect to Perses and patch the available Plugins to proxy to your local plugin dev server. Basically the CLI will do some or all steps from here

I imagine the usage from the plugin(s) in development like:

percli login https://demo.perses.dev # or http://localhost:8080
percli plugin start # or percli plugin start my-plugin-inside-a-folder  or  percli plugin start timeseries prometheus

in which percli plugin start will patch the proxy configuration and start the plugin dev server, it could even detect there is no perses and start one from Docker and login...

[andreasgerstmayr]

Personally I hit refresh a thousand times during local development, and would prefer fast response times from localhost instead of hitting a remote endpoint every time, so I'd prefer making the local dev env as straightforward and fast as possible (including hot reload etc.).

Regarding the current plugin config

plugin:
  dev_environment:
    plugins:
      - name: Table
        disable_schema: false
        url: http://localhost:3005
        absolute_path: /absolute/path/to/plugin/repository/table

maybe we could just specify a path, /absolute/path/to/plugin/repository, and it picks up all plugins from there?

Probably I'm biased, but recently most of my changes involved multiple plugins at the same time (table and statchart, or prometheus and tempo, or #80 touched all plugins), and some plugins have dependencies in @perses-dev/components, which also need to be updated at the same time (e.g. table panel) and might require a link from the plugin to @perses-dev/components in case the types in @perses-dev/components change (adding new options to the table interface, for example).

[sjcobb]

During the development of a plugin, it's not that simple to setup the Perses server(with the Cuelang schema and so on). So I was thinking about creating a light Perses.

This does sound nice, if hot reload is an issue then given Gabriel's comment in Slack "Developing panel plugins in isolation would be possible using storybook at the plugin level" -- another workflow could be to do most of the development in storybook (with hot reload) and then occasionally push up to perses.dev to test with real data on the sandbox dashboard.

in which percli plugin start will patch the proxy configuration and start the plugin dev server, it could even detect there is no perses and start one from Docker and login

However we can make it one easy command sounds great!

Probably I'm biased, but recently most of my changes involved multiple plugins at the same time (table and statchart, or prometheus and tempo, or #80 touched all plugins), and some plugins have dependencies in @perses-dev/components, which also need to be updated at the same time (e.g. table panel) and might require a link from the plugin to @perses-dev/components in case the types in @perses-dev/components change (adding new options to the table interface, for example).

Yeah with the new plugin architecture, I think the base charts should move to live beside the relevant panel plugin--for ex TimeChart and even components like TimeSeriesTooltip. This could also make it easier to not run into circular dependencies / leverage context more (ex: could add TooltipProvider to pass info about the spec down to the lower level tooltip components or if we ever want to add features like series-level data links)

[Nexucis]

I think I will try to implement the CLI command proposed by Gabriel. From my perspective it will reduce the server config to just

plugin:
  enableDev: true