Remote Control API

[kblaschke]

There are so many use cases which require projectMSDL to be controlled by external events, devices or a remote PC, that it makes sense to implement a remote control API. This API should be able to perform any task which can be achieved via the UI, like changing settings, playback control, listing/changing audio devices and editing preset playlists on the fly.

Other inputs for controlling the preset playback, e.g. via MIDI events, can then be implemented via the API instead of adding all the use-case specific functionality directly into the UI. This will also make it possible to reconfigure those external inputs at runtime without visually disturbing projectMSDL, as the UI doesn't have to be opened.

The API should use a protocol which is easy to use either from a website or any custom application. There are three general ways to implement it technically:

• JSON RPC via HTTP/1.1: The most basic API possible, used in many web applications. Uses simple request-response calls from the client to projectMSDL. Ideally using Open API 3 to create the specification.
• Bi-directional communication via Websockets: Allows communication in both directions, but Websockets are not supported by some HTTP client frameworks.
• gRPC via HTTP/2.0: A mix of the above, using HTTP2 streams. By default uses a request-response mechanism, but clients can also subscribe to specific "events" on the server to be notified if something changes.

Currently, the only events which would make server-side events necessary is automatically switching presets (to get the new preset name) and possibly a change of the current audio device. In the first case, this info could simply be polled every second or so if needed. That would make the JSON RPC API the preferred one for a first implementation. The other ones could also be done later if there's need.

Authentication should use the standard OAuth protocol.

For environments which require it, e.g. open networks or even the internet, projectMSDL should also support using HTTPS. In this case, the user has to supply the proper SSL keys and certificates, generated via OpenSSL or a compatible tool.

[revmischa]

The messages could be wrapped in OSC possibly https://en.wikipedia.org/wiki/Open_Sound_Control.
I think WebSockets are a fine idea.
I've never heard anyone say anything good about gRPC.

[kblaschke]

OSC looks rather use-case specific, and wouldn't be easy to use from a website or similar without requiring devs to implement the (binary) protocol. Something easier like JSON might be better suited. I'd rather see the support for those specialized protocols in a "proxy" app (either one for each third-party protocol, or one app which implements multiple of those) which maps OSC, MIDI and other inputs to the projectMSDL JSON RPC protocol. It would surely be possible to put some of those proxy apps in the same repo and build them for any binary releases, or leave it to the users which ones (if any) they want to build if they make their own binaries.

Yeah, Websockets are great, especially given they use just one connection (no authentication for each request) and support server-side events without subscriptions or polling.

gRPC is okay, but has a rather large overhead as it needs HTTP/2 and another protocol library like Protobuf to wrap messages. It's definitely also my least-favorite one.

[nzoschke]

I also have a remote control use case, changing presets in real-time in sync with events coming from DJ gear (via https://github.com/nzoschke/vizlink).

Currently I'm using a stdin/out RPC, HTTP Server-Sent Event (SSE), and a web browser for everything

• Spawn vizlink, get music events via stdout
• Wrap the RPC in a web server and flush a SSE message on music event
• Poll that API in a web browser and change the pattern (via Butterchurn.loadPreset)

I plan to move over to projectMSDL and/or Emscripten someday, so a way to remotely change the preset would be great.

JSON RPC API sounds right to me too as a simple and universal solution.

Perhaps there's an even lower level stdin/stdout interface to consider too.

The lower level RPC can always be wrapped if folks do want websockets or OSC.

Websockets are efficient but can be tricky with browser security context which may disallow requests that are not WSS with proper HTTPS/SSL.

Having using gRPC before I'd advise against it as the protobuf specs, generated server stubs and clients are pretty awful and complicated.

[nzoschke]

The main RPC I'd use is loadPreset(name, transitionDuration) as this would allow me to control everything from the outside effectively building my own playlist and pattern duration logic.

But I can see how RPCs to loading a preset playlist and changing global settings like pattern duration would be helpful for other use cases.

[kblaschke]

But I can see how RPCs to loading a preset playlist and changing global settings like pattern duration would be helpful for other use cases.

Yeah, it will support different modes of operations, e.g. loading a playlist and just let the app do the rest or control each preset remotely, including sending the preset contents via the API. When doing the latter, the transition duration would be set in a different call though, so you'd rather first set the transition duration, then initiate the preset switch. The last set transition duration will be used in this case.

[nzoschke]

Here's a totally awful POC where I update a files contents with the path of the preset to load

diff --git a/src/RenderLoop.cpp b/src/RenderLoop.cpp
index 73e3390..4a49fcd 100644
--- a/src/RenderLoop.cpp
+++ b/src/RenderLoop.cpp
@@ -5,6 +5,10 @@
 #include <Poco/Util/Application.h>
 
 #include <SDL2/SDL.h>
+#include <sys/stat.h>
+#include <iostream>
+#include <fstream>
+
 
 RenderLoop::RenderLoop()
     : _audioCapture(Poco::Util::Application::instance().getSubsystem<AudioCapture>())
@@ -15,6 +19,28 @@ RenderLoop::RenderLoop()
 {
 }
 
+bool fileChanged(const std::string& filename, std::chrono::system_clock::time_point& lastModified, char*& fileContents) {
+    struct stat fileInfo;
+    if (stat(filename.c_str(), &fileInfo) == 0) {
+        std::time_t modifiedTime = fileInfo.st_mtime;
+        if (modifiedTime > std::chrono::system_clock::to_time_t(lastModified)) {
+            lastModified = std::chrono::system_clock::from_time_t(modifiedTime);
+
+            std::ifstream file(filename);
+            if (!file) {
+                std::cerr << "Error: Unable to open file." << std::endl;
+                return false;
+            }
+            std::string content((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
+            fileContents = new char[content.size() + 1];
+            std::strcpy(fileContents, content.c_str());
+
+            return true;
+        }
+    }
+    return false;
+}
+
 void RenderLoop::Run()
 {
     FPSLimiter limiter;
@@ -24,8 +50,17 @@ void RenderLoop::Run()
 
     _projectMWrapper.DisplayInitialPreset();
 
+    const std::string filename = "/tmp/preset.txt";
+    char* fileContents = nullptr;
+    std::chrono::system_clock::time_point lastModified = std::chrono::system_clock::now();
+
     while (!_wantsToQuit)
     {
+        if (fileChanged(filename, lastModified, fileContents)) {
+            projectm_load_preset_file(_projectMHandle, fileContents, false);
+            std::cout << fileContents << std::endl;
+        }
+
         limiter.StartFrame();
         PollEvents();
         CheckViewportSize();
@@ -172,6 +207,10 @@ void RenderLoop::KeyEvent(const SDL_KeyboardEvent& event, bool down)
             }
             break;

[hack-s]

It would be nice, if the same RPC could also be used for the GStreamer plugin, and not just the front-end app. Not sure how much complexity it would add though, if that would introduce extensions that are application specific. Just a thought..