Python: draft initial implementation of Realtime API

[eavanvalkenburg]

Motivation and Context

Implements the OpenAI Realtime API with Semantic Kernel

Description

Implements a separate Service Client class with its own ExecutionSettings, but still based on ChatCompletionClientBase.
Only support streaming operations with additional public methods for sending data to the conversation.
TBD if that is the way to move forward with it.

TODO:

• lots of comments
• tests
• cleanup

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
semantic_kernel/connectors/ai
chat_completion_client_base.py	124	2	98%	395, 405
function_calling_utils.py	52	10	81%	161–186
realtime_client_base.py	22	4	82%	102, 109–110, 114
semantic_kernel/connectors/ai/open_ai/services
open_ai_realtime.py	37	14	62%	35–37, 84–98, 127, 147
semantic_kernel/connectors/ai/open_ai/services/realtime
open_ai_realtime_base.py	181	130	28%	79–117, 127–163, 170–203, 211–384, 393–397, 403, 412, 416, 420
open_ai_realtime_webrtc.py	122	83	32%	61–63, 66–74, 84–129, 134–141, 144–171, 179–188, 192–211
open_ai_realtime_websocket.py	58	29	50%	46–64, 67–73, 83–86, 91–94
utils.py	40	32	20%	38–44, 54, 69–126
semantic_kernel/contents
audio_content.py	25	2	92%	81, 86
binary_content.py	115	16	86%	81, 120, 138–139, 180–184, 192–198
function_call_content.py	107	3	97%	197, 225–226
semantic_kernel/contents/utils
data_uri.py	101	4	96%	44–45, 68, 133
TOTAL	17721	2182	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3049	4 💤	0 ❌	0 🔥	1m 20s ⏱️

[moonbox3]

Should we call out whether the “control” versus “content” distinction is a fundamental part of real-time interaction or just an implementation detail? For example, OpenAI distinguishes control events (input_audio_buffer.committed) from content events (conversation.item.create), while Google appears to treat everything as part of a unified content stream (BidiGenerateContent*).

This distinction might influence our decision in a few ways:

• If the distinction is inherent to real-time systems, separating control from content may result in a cleaner, more flexible design.
• However, if it’s just a specific quirk of OpenAI’s API, enforcing it could complicate support for providers like Google that don’t make the same distinction.
• On the other hand, ignoring OpenAI’s finer-grained controls might limit the ability to fully utilize other features in the future.

I think it would make sense to call this out explicitly in the doc and could provide additional context for why we’re choosing one approach over the other.