Streams and Effects

• Introduction
• Streams
• Effects
  • Virtual Background Effect
    • Applying the virtual background effect
  • Noise Reduction Effect
    • Applying the Noise Reduction Effect
• Helper Methods for Effects

Introduction

Meetings SDK 3.0 introduces new APIs designed to facilitate the management of local media streams for audio, video, and screen sharing. This document explains how Streams function with two newly added effects: VirtualBackgroundEffect and NoiseReductionEffect. These effects and corresponding methods are available in the plugin-meetings package. Developers can explore and play around with these effects in the Kitchen sink app here.

Streams

Introduction

The classes that have been introduced are below.

1. LocalCameraStream
2. LocalMicrophoneStream
3. LocalDisplayStream
4. LocalSystemAudioStream

Use these methods to create Streams for various devices.

1. createCameraStream
2. createMicrophoneStream
3. createDisplayStream
4. createDisplayStreamWithAudio

Import Streams into your applications via plugin-meetings package as shown in the code snippet below.

import { 
  LocalCameraStream,
  LocalMicrophoneStream,
  LocalDisplayStream,
  LocalSystemAudioStream,
  createCameraStream,
  createMicrophoneStream,
  createDisplayStream,
  createDisplayStreamWithAudio
 } from '@webex/plugin-meetings';

Create Streams

To create any of the mentioned local streams (camera, microphone, or display), simply invoke the respective method while optionally passing the required parameters as shown below.

const cameraStream = await createCameraStream(cameraConstraints);

const microphoneStream = await createMicrophoneStream(audioConstraints);

const [localShareVideoStream, localShareAudioStream] = await createDisplayStreamWithAudio();

Here are the constraints that the create functions accept.

const cameraConstraints = {
    deviceId?: ConstrainDOMString;
    width?: ConstrainULong;
    height?: ConstrainULong;
    aspectRatio?: ConstrainDouble;
    frameRate?: ConstrainDouble;
    facingMode?: ConstrainDOMString;
};

const audioConstraints = {
    deviceId?: string;
    autoGainControl?: boolean;
    echoCancellation?: boolean;
    noiseSuppression?: boolean;
};

Effects

Meetings 3.0 SDK introduces two media effects:

1. VirtualBackgroundEffect
2. NoiseReductionEffect

Virtual Background Effect

The virtual background can take the form of an image, an MP4 video, or the user's background with applied blur. The blur option offers flexibility with adjustable strength and quality levels, with higher settings demanding greater computational resources.

We have following API to create the effect. Access this API via the meetings object as mentioned below:

await webex.meetings.createVirtualBackgroundEffect(options);

Asynchronous	Yes
Parameters	options Name Description Values Required generator Determines where the model runs (on main thread or background thread) string local worker Optional, Defaults to worker frameRate Determines how many frames are sent to the model per second number 0-60 Optional, Defaults to 30 quality Determines the accuracy of the model (higher accuracy requires additional CPU resources) string LOW MEDIUM HIGH ULTRA Optional, Defaults to LOW mirror Whether the output image should be flipped horizontally boolean Optional, Defaults to false mode Determines the kind of background to render behind the user string BLUR IMAGE VIDEO Optional, Defaults to BLUR blurStrength How strongly the background should be blurred string WEAK MODERATE STRONG STRONGER STRONGEST Required in BLUR mode bgImageUrl Path to the background image to replace the original background string Fully qualified URL Required in IMAGE mode bgVideoUrl Path to the background video to replace the original background string Fully qualified URL (mp4 only) Required in VIDEO mode
Returns	A promise that returns virtual background effect

Applying the Virtual Background Effect

Follow these steps to apply the virtual background effect to your camera stream.

1. Let's create a local camera stream, which we'll later apply the blur background effect to.

   const cameraConstraints = { width: 640, height: 480 };
   
   const cameraStream = await createCameraStream(cameraConstraints);

2. Add the camera stream on your video srcObject in your code

   meetingStreamsLocalVideo.srcObject = cameraStream.outputStream

3. Then, create the virtual background effect using createVirtualBackgroundEffect()

   const effect = await webex.meetings.createVirtualBackgroundEffect();   //we're not passing any optional parameters as default mode is BLUR

4. After creating the blur background effect, we will utilize the addEffect method of the local camera stream to apply the newly created effect to the camera stream.

   const effect = await cameraStream.addEffect("blur-background", effect);

5. Now that we've added the effect to the camera stream, we need to enable the effect to observe it in action on the video.

   await effect.enable();

6. You will now be able to observe the virtual background effect applied to your camera stream.

Noise Reduction Effect

The noise reduction effect is designed to eliminate background noise from an audio stream, ensuring clear audio during calls. To create this effect, you can access the following API through the meetings object, as shown below.

await webex.meetings.createNoiseReductionEffect(options);

Asynchronous	Yes
Parameters	options Name Description Values Required audioContext An optional AudioContext for custom behaviour, for e.g., const audioContext = new AudioContext({sampleRate: 48000}); object No mode Determines whether to run in WORKLET mode or LEGACY mode for older browsers string WORKLET LEGACY No, Defaults to WORKLET
Returns	A promise that returns noise reduction effect

Applying the Noise Reduction Effect

Follow these steps to apply the noise reduction effect on a microphone stream.

const microphoneStream = await createMicrophoneStream();

const effect = await webex.meetings.createNoiseReductionEffect();
               
meetingStreamsLocalAudio.srcObject = microphoneStream.outputStream;

const effect = await microphoneStream.addEffect("noise-reduction", effect);

await effect.enable();

Helper Methods for Effects

Both effects offer generic helper methods for various purposes.

• effect.disable() - disables the effect
• effect.dispose() - tears down the effect
• effect.setEnabled(enable) - single method to enable or disable the effect    // pass true to enable the effect and false for disable
• stream.getEffectByName('effectName') - get the effect added on the stream
• stream.getAllEffects(****) - get all the effects added on the stream
• stream.disposeEffects() - tears down all the effects from the stream