VoicePipelineAgent don't send the transcription automatically to client

[cuongpham-1001]

I tried to use VoicePipelineAgent but didn't receive the transcription from client when listening to event RoomEvent.TranscriptionReceived
The livekit document mention "VoicePipelineAgent and MultimodalAgent can generate and deliver transcriptions automatically".
I also try to find the code that implement the transcription forwarding but look like it hasn't implemented yet
Here is my code to setup VoicePipelineAgent

const { logger, voiceSettings } = options
  const initialContext = new llm.ChatContext().append({
    role: llm.ChatRole.SYSTEM,
    text: options.defaultInstruction
  });

  const vad = await silero.VAD.load();
  const agent = new pipeline.VoicePipelineAgent(
    vad,
    new openai.STT(),
    new openai.LLM(),
    new openai.TTS({
      voice: voiceSettings?.voice,
    }),
    {
      chatCtx: initialContext,
      allowInterruptions: true,
      interruptSpeechDuration: 500,
      interruptMinWords: 0,
      minEndpointingDelay: 500,
      transcription: {
        userTranscription: true,
        agentTranscription: true,
        agentTranscriptionSpeech: 1,
        sentenceTokenizer: new tokenize.basic.SentenceTokenizer(),
        wordTokenizer: new tokenize.basic.WordTokenizer(false),
        hyphenateWord: tokenize.basic.hyphenateWord,
      },
      beforeLLMCallback: (_, ctx) => {

        const lastMessage = ctx.messages[ctx.messages.length - 1]
        if (lastMessage) {
          logger.info({
            content: lastMessage.content,
          })
        }
      },
      beforeTTSCallback: async (_, source) => {
        const messageChunks: string[] = []
        if (typeof source === 'string') {
          messageChunks.push(source)
        } else {
          for await (const chunk of source) {
            messageChunks.push(chunk)
          }
        }
        const message = messageChunks.join('')
        logger.info({
          content: message,
        })
        return message
      },
    }

[valdrox]

I left a comment in the docs. I had the same issue. I think I saw a comment somewhere saying they were working on this.

This is what i'm going with until this is added:

  let messageCounter = 0;

  const handleSpeechCommitted = async (text: { content?: string }) => {
    if (text.content != undefined) {
      messageCounter++;
      const textForwarder = new BasicTranscriptionForwarder(
        ctx.room,
        participant.identity,
        'trackSid',
        messageCounter.toString(),
      );
      textForwarder.start();
      textForwarder.pushText(text.content);
      textForwarder.markTextComplete();
      textForwarder.close(false);
    }
  };

  agent.on(pipeline.VPAEvent.USER_SPEECH_COMMITTED, handleSpeechCommitted);
  agent.on(pipeline.VPAEvent.AGENT_SPEECH_COMMITTED, handleSpeechCommitted);

Then catch it in the front-end with the code in the docs.

[nbsp]

our method of sending transcription events is currently very rudimentary; this is something we're working on for a future release

[heymartinadams]

Thanks, @valdrox! Slightly modified to:

let messageCounter = 0

const handleSpeechCommitted = async (msg: ChatMessage) => {
  let content = ''

  if (Array.isArray(msg.content)) {
    content = msg.content
      .map(c => {
        if (typeof c === 'string') return c
        if ('text' in c) return c.text
        return ''
      })
      .join(' ')
  } else if (typeof msg.content === 'string') {
    content = msg.content
  }

  if (content) {
    messageCounter++
    const textForwarder = new BasicTranscriptionForwarder(
      ctx.room,
      participant.identity!,
      'trackSid',
      messageCounter.toString()
    )
    textForwarder.start()
    textForwarder.pushText(content)
    textForwarder.markTextComplete()
    textForwarder.close(false)
  }
}

agent.on(pipeline.VPAEvent.USER_SPEECH_COMMITTED, handleSpeechCommitted)
agent.on(pipeline.VPAEvent.AGENT_SPEECH_COMMITTED, handleSpeechCommitted)

[nbsp]

agent and user transcriptions will be sent on VoicePipelineAgent, with PR #301.