Update UI and add Proactive Video feature to Websocket-demo-app #2273

tpareek18 · 2025-08-21T00:07:56Z

Description

Introducing the following changes to the gemini/multimodal-live-api/websocket-demo-app:

Cleaner and more-organized UI for the Websocket Demo Application
Added functionality to allow for client-side configuration to enable proactive video in a session
Added functionality to allow for client-side control of audio and video sampling rate
Minor fixes for microphone and video toggle functionality

…g in frames.

…dio and video.

Add account for changes to websocket-demo-app

gemini-code-assist

Summary of Changes

Hello @tpareek18, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the websocket-demo-app by revamping its user interface for improved organization and clarity. It introduces a new proactive video feature, allowing users to enable video streaming to the AI model. Furthermore, it provides client-side controls for audio and video sampling rates, enabling fine-tuning of media capture. The changes also include crucial updates to API endpoints and an upgrade to the audio processing pipeline for better performance and reliability, alongside minor fixes to existing media toggle functionalities.

Highlights

Enhanced User Interface (UI) and Experience: The user interface of the websocket-demo-app has undergone a significant redesign, featuring a cleaner, more organized layout with new sections for connection, model behavior, and advanced settings. This improves usability and clarity for configuring the application.
Introduction of Proactive Video Feature: A new 'Proactive Video' feature has been integrated, allowing the client to configure the session to enable proactive video streams to the model. This is controlled via a new UI checkbox and corresponding API parameter.
Client-Side Control for Audio and Video Sampling Rates: Users can now configure the audio and video sampling rates directly from the client-side UI. This provides greater control over the frequency at which audio chunks and video frames are sent to the model, enabling optimization for different use cases or network conditions.
Improved Audio Processing with AudioWorklet: The underlying audio processing mechanism has been upgraded to use AudioWorkletNode and a new input-processor.js file. This modern approach to Web Audio API processing offers better performance and stability for real-time audio capture and conversion.
Expanded API Response Handling: The application now supports a wider range of API response types, including voice activity detection (VAD) signals, session resumption handles, input/output audio transcriptions, and explicit end-of-turn or interruption signals. This enriches the feedback provided by the multimodal AI.
API Endpoint Updates: The API endpoints have been updated to point to autopush environments and utilize internal.LlmBidiService, indicating a shift towards internal testing or a newer version of the API for multimodal interactions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request significantly overhauls the UI of the websocket demo application, making it much more organized and feature-rich. It also adds extensive new functionality, including proactive video, configurable sampling rates, and other advanced settings. My review focuses on improving code quality by removing dead code and development artifacts, fixing some bugs in the UI logic and HTML, and enhancing consistency and debuggability in the JavaScript code.

gemini/multimodal-live-api/websocket-demo-app/frontend/gemini-live-api.js

gemini-code-assist · 2025-08-21T00:10:03Z

gemini/multimodal-live-api/websocket-demo-app/frontend/index.html

+            <md-select-option selected value="puck">
+              <div slot="headline">puck</div>
+            </md-select-option>
+            <md-select-option selected value="charon">
+              <div slot="headline">charon</div>
+            </md-select-option>
+            <md-select-option selected value="kore">
+              <div slot="headline">kore</div>
+            </md-select-option>
+            <md-select-option selected value="fenrir">
+              <div slot="headline">fenrir</div>
+            </md-select-option>
+            <md-select-option selected value="aoede">
+              <div slot="headline">aoede</div>
+            </md-select-option>
+            <md-select-option selected value="orus">
+              <div slot="headline">orus</div>
+            </md-select-option>
+            <md-select-option selected value="zephyr">
+              <div slot="headline">zephyr</div>
+            </md-select-option>
+            <md-select-option selected value="leda">
+              <div slot="headline">leda</div>
+            </md-select-option>


Multiple <md-select-option> elements within this voiceName select component have the selected attribute. For a single-select dropdown, only one option can be the default. This will lead to unpredictable behavior where the browser selects the last option marked as selected. Please ensure only one option is marked as selected to define the intended default value. This issue is also present in other select elements like voiceLocale.

gemini/multimodal-live-api/websocket-demo-app/frontend/script.js

gemini-code-assist · 2025-08-21T00:10:03Z

gemini/multimodal-live-api/websocket-demo-app/backend/main.py

+# !!! Need to change this as well to make the environment switch work.
+# HOST = "us-central1-staging-aiplatform.sandbox.googleapis.com"
+# HOST = "us-central1-aiplatform.googleapis.com"
+# SERVICE_URL = f"wss://{HOST}/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent"


These commented-out variables and the !!! comment suggest temporary development code. This should be cleaned up before merging. If you need to support multiple environments, consider using environment variables or a configuration file for better maintainability.

gemini/multimodal-live-api/websocket-demo-app/frontend/gemini-live-api.js

gemini/multimodal-live-api/websocket-demo-app/frontend/index.html

gemini/multimodal-live-api/websocket-demo-app/frontend/live-media-manager.js

gemini/multimodal-live-api/websocket-demo-app/frontend/script.js

gemini/multimodal-live-api/websocket-demo-app/backend/main.py

holtskinner · 2025-08-21T15:45:04Z

gemini/multimodal-live-api/websocket-demo-app/frontend/gemini-live-api.js

+        // this.serviceUrl = `wss://${this.apiHost}/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent`;
+        this.serviceUrl = `wss://${this.apiHost}/ws/google.cloud.aiplatform.internal.LlmBidiService/BidiGenerateContent`;


Remove commented code or change this to v1beta1 if internal doesn't work externally.

These values can/should also be set in another way

Changed to v1beta1. Will look for a better way to set these values. Thank you!

gemini/multimodal-live-api/websocket-demo-app/frontend/gemini-live-api.js

holtskinner · 2025-08-21T15:46:34Z

gemini/multimodal-live-api/websocket-demo-app/frontend/index.html

+        <div class="form-grid">
+          <md-outlined-select label="Voice Name" id="voiceName">
+            <md-select-option selected value="puck">
+              <div slot="headline">puck</div>
+            </md-select-option>


Can you load these in from the backend so you can loop instead of hardcoding them all here? Or be able to fetch this info from the API itself?

gemini/multimodal-live-api/websocket-demo-app/frontend/index.html

holtskinner · 2025-08-21T15:47:53Z

Assigning to @ZackAkil as he originally created this

tpareek18 and others added 15 commits July 16, 2025 16:31

make basic changes for demo setup

7116d65

proactive video feature addition

5cfb72d

UI Changes for Proactive Video Demo

5711668

Merge branch 'GoogleCloudPlatform:main' into main

d1cb7cb

Fix microphone and video toggle issue. Fix intervals logic for sendin…

c0503da

…g in frames.

Merge branch 'GoogleCloudPlatform:main' into main

27ec72a

Fix microphone on right after connecting. Change sampling rate for au…

59aa62f

…dio and video.

Fix microphone issue when capturing audio chunk recordings.

c8f0828

Removed adding empty user SI. Add user SI only if explicitly added.

f3d987f

add staging environment endpoint as backup.

0eada99

Merge branch 'GoogleCloudPlatform:main' into main

73192f9

Merge branch 'GoogleCloudPlatform:main' into main

847d55b

Merge branch 'GoogleCloudPlatform:main' into main

ec3f37c

Allow to control audio and video sampling rate from UI.

8d5b05c

Update CODEOWNERS

fb4aa31

Add account for changes to websocket-demo-app

tpareek18 requested a review from a team as a code owner August 21, 2025 00:07

gemini-code-assist bot reviewed Aug 21, 2025

View reviewed changes

holtskinner requested changes Aug 21, 2025

View reviewed changes

holtskinner assigned ZackAkil Aug 21, 2025

tpareek18 added 2 commits August 21, 2025 14:00

Fix syntax issues. Change API endpoints for external users.

e1e80f9

reflect Service URL change everywhere in the code.

beaa6f3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update UI and add Proactive Video feature to Websocket-demo-app #2273

Update UI and add Proactive Video feature to Websocket-demo-app #2273

Uh oh!

tpareek18 commented Aug 21, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

gemini-code-assist bot Aug 21, 2025

Uh oh!

Uh oh!

gemini-code-assist bot Aug 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

holtskinner Aug 21, 2025

Uh oh!

tpareek18 Aug 21, 2025

Uh oh!

Uh oh!

holtskinner Aug 21, 2025

Uh oh!

Uh oh!

holtskinner commented Aug 21, 2025

Uh oh!

Uh oh!

		// this.serviceUrl = `wss://${this.apiHost}/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent`;
		this.serviceUrl = `wss://${this.apiHost}/ws/google.cloud.aiplatform.internal.LlmBidiService/BidiGenerateContent`;

Update UI and add Proactive Video feature to Websocket-demo-app #2273

Are you sure you want to change the base?

Update UI and add Proactive Video feature to Websocket-demo-app #2273

Uh oh!

Conversation

tpareek18 commented Aug 21, 2025

Description

Introducing the following changes to the gemini/multimodal-live-api/websocket-demo-app:

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

holtskinner Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

tpareek18 Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

holtskinner Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

holtskinner commented Aug 21, 2025

Uh oh!

Uh oh!