A GPU Accelerated, self-contained, MediaPipe Plugin for TouchDesigner that runs on Mac and PC with no installation. This project currently supports all MediaPipe vision models except Interactive Segmentation and Image Embedding.
To get an idea of what's possibe with the plugin, and a quick tutorial on how to get up and running, check out our introduction video on YouTube
Download the latest release.zip from the Release Section. Open up the MediaPipe TouchDesigner.toe file. All of the components are stored inside the /toxes folder. The main component is MediaPipe.tox. All of the other components are examples of how to load and display the associated model data in TouchDesigner.
On the MediaPipe component once it's loaded you can select your webcam from the drop-down. You can turn on and off the different MediaPipe models as well as preview overlays. There're also sub-menus available for each model to customize them further.
Currently the model is limited to 720p input resolution - as long as that's a resolution your webcam supports, you're good to go.
The plugin consists of a number of components:
This component launches a Chromium web browser to host and run all of the MediaPipe vision tasks. It has a DAT output for each vision task, as well as a TOP output showing the video feed and any overlays.
Use this to process the face detection results
Use this to process the facial landmark detection results
Use this to process the hand landmark and gesture detection results
Gesture recognition task guide
Use this to process the object detection results
Use this to process the pose landmark detection results
Use this to key out segmentation results Image segmentation guide
Note: If you're hoping to get the most accurate web-cam cutout use the MultiCam model in the MediaPipe.tox. There's also a toggle to display only the background cutout which you can enable while on the multiclass model.
Use this to identify what the image might contain Image classification guide
If you are on Windows, you can use Spout to send any TOP from TouchDesigner to MediaPipe by using a Syphon Spout Out TOP and SpoutCam. Take a look at our introduction video on YouTube for how to set it up.
Note there is no installer, it will run from wherever the folder is
- Enter the frame rate and resolution to match your source material in TouchDesigner
- Enter the default TouchDesigner Spout output name of
TDSyphonSpoutOutinto the Starting Sender box - Click register to create the virtual webcam
Feed your desired TOP into it
The media should appear with no frames of delay!
For diagnosing this issue, I recommend downloading the Spout2 files from the same GitHub profile as the SpoutCam Plugin. This adds some settings and diagnostics options when working with Spout. In my case, my laptop graphics proved to be the problem! When running diagnostics, checking for compatibility and looking at the running spout cam process (all possible through the SpoutPanel and SpoutSettings) it told me that the texture sharing was failing, thus resulting in the noise. After reading through the SpoutSettings and their explanations (not the SpoutCamSettings!), I made sure to put all processes involving spout (so sender & receiver) on the same graphics pipeline (by changing the .exe graphic settings in Windows, this is explained in SpoutSettings), since my laptop has both integrated CPU graphics, as well as a GPU.
This actually fixed my issue!
So for anyone that has a laptop, multiple GPUs or any other Spout issues in combination with this TouchDesigner plugin, I recommend downloading the rest of the Spout software and tinkering a bit.
Thanks to @vVattghern for finding this answer
There is no SpoutCam equivalent on Mac, but you can use Syphon to send video to OBS, and then use the OBS Virtual Webcam output to the MediaPipe task. It's not the most elegant solution, but it works.
There are lots of interesting bits of data in the CHOP output of the MediaPipe tox:
How long the MP detector took to run in ms.
How long the overlays and segmentation colors took to draw in ms.
The frame rate of the webcam video source MediaPipe is using.
What ratio of a frame it took to process the video. So 0.1 means 0.1 of a frame was needed for MediaPipe to process. This uses the project's cookRate as a reference
The number of project frames delay MediaPipe introduces. In TouchDesigner 2022.33910 this is at least 3 frames for the web browser component. If you are using Spout or Syphon to send video to a virtual webcam, you can use this as a parameter in a cache TOP to sync your input video source back up with the MediaPipe output TOP.
Tells you if the whole process is able to keep up with the input frame rate.
The MediaPipe detection tasks are very CPU and GPU intensive, so turn off any that you aren't using
If you are on a PC, you can greatly improve all CPU-based render times within TouchDesigner, including the MediaPipe tasks by disabling HyperThreading (Intel CPUs) or Simultaneous Multi-Threading (SMT - AMD CPUs). On my laptop it was a 60-80% improvement.
Enabling/disabling HyperThreading is done in your system BIOS, so look up the instructions on how to do that for your computer. You can re-enable HyperThreading at any point if you want to go back and it is not a risky change to make to your system.
This project loads the different MediaPipe models through a web browser. All of the ML models are downloaded locally and stored inside TouchDesigner's virtual file system including the website so the component can run without an internet connection. The models run using web assembly and the data coming back from the models are piped into TouchDesigner through a local WebSocket server running inside TD. This allows the components to run as standalone .tox files with GPU acceleration on any device with no setup.
As MediaPipe currently only supports GPU acceleration via the Javascript implementation, and this is the only version that does not require installing local libraries, we have implemented this version, by using the following three main components:
- Web (and websocket) server
- Web browser
- JSON decoders
The web server component has an embedded set of web pages that are served to the web browser just like any website would do. It also acts as a websocket server that allows two-way-communication between the web browser and TouchDesigner.
The embedded Chromium support in TouchDesigner allows us to run a full web browser within TouchDesigner. This web browser opens the web pages served by the web server, which allow it to run all of the MediaPipe detection components and render the final video stream. The web browser also sends coordinate data and other detection data back to TouchDesigner via a websocket connection.
We send the data from the MediaPipe instance back to TouchDesigner in JSON format. We then use the additional detection tox files to process this JSON data into useful things that can be used elsewhere in TouchDesigner.
This package uses yarn with vite inside node, which gives you a few options. Firstly, you need to download and install node.js
Install Node.js
Install yarn package manager globally
npm install --global yarn
yarn install
This installs vite and all the other dependencies required. Should only be needed the first time.
There are 2 options for debugging the web page
- Debug the page as-is within TouchDesigner
- Live debug a development page
Once the MediaPipe project has loaded withing TouchDesigner, you can open a regular Chrome browser on your desktop and go to http://localhost:9222 which will open the developer tools console for the Chromium instance embedded within the MediaPipe component. Note that this will have a performance impact as the page is being rendered twice (once in TouchDesigner, once in Chrome), but it gives you access to the console log to see what's going on and maybe debug some issues.
yarn dev
This launches a tiny web server on port 5173 that fires a reload to the browser every time you save a file change. Super useful for debugging the web page.
Suggested use for this:
- Run
yarn devvia the command line, or the console window of VSCode - Load the TouchDesigner project and toggle all the settings you'll want to test with.
- Enter the
MediaPipeCOMP - Disable the
webbrowsercomponent by clicking the X - Copy the
current_urlDAT string - Paste the url into a new Chrome tab
- Edit the url and replace the port number with 5173, so for example replace
localhost:3001withlocalhost:5173
Chrome will now load the page from yarn dev, while still communicating with TouchDesigner via websockets. Note that parameter changes you make in TouchDesigner will not be reflected in the web page unless you repeat the steps to copy the url.
If you press F12 to open the Chrome Developer Tools you can start digging into the console to see error messages.
yarn build
This builds all the web page files and puts them into the _mpdist folder. If this folder exists, the web pages used within the MediaPipe COMP will be served from here.
There is a build_release COMP in the top level at the top of the toe file that handles an automated build process.
To create a release zip file:
- Open the
MediaPipe TouchDesigner.toefile - Navigate to the layout you want to be loaded when someone opens the project for the first time.
- Press
Ctrl Alt Bto trigger a build. The project will lock up for a while during the build process. - The build script will:
- Remove and recreate a
releasefolder. - Perform a
yarn install - Perform a
yarn build - Load all files from
_mpdistinto theMediaPipeCOMP Virtual File System - Remove all external script references on DATs
- Export all tox files to
release/toxes - Save the toe file to
release - Zip the contents of the
releasefolder torelease.zip - Give you a popup showing the number of errors.
- Clicking ok (or anywhere!) causes the toe file to reload from the main folder so you're back where you started.
- If there were any errors in the build, you can turn on the TouchDesigner text console and run the build again to see what the errors are.
yarn version --patch for backward-compatible bug fixes.
yarn version --minor for backward-compatible new features.
yarn version --major for changes that break backward compatibility.
