Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement LMNT full-duplex TTS generation #87

Closed
wants to merge 1 commit into from

Conversation

chadbailey59
Copy link
Contributor

LMNT has an interesting approach to 'full-duplex' TTS:

async def main():
  async with Speech() as speech:
    connection = await speech.synthesize_streaming('lily')
    t1 = asyncio.create_task(reader_task(connection))
    t2 = asyncio.create_task(writer_task(connection))
    asyncio.gather([t1, t2])
    connection.close()

This doesn't really work with our FrameProcessor architecture, but it does actually work pretty well with the idea of a custom pipeline. I've started building that out in examples/foundational/02a-async-llm-say-one-thing.py.

I'm currently stuck because I think there's a bug in their implementation. After I get the first chunk of audio back, the websocket connection to LMNT seems to close. I've tried logging just about everywhere I can think of, and I can't find an explanation. I'm going to try and get in touch with the LMNT team to troubleshoot.

(I found out about LMNT through Andy Korman, who I met at our Voice + AI Summit.)

@shaper
Copy link

shaper commented Mar 26, 2024

Hi Chad, this is awesome! I patched this and was able to initially repro the issue you describe, but after the issue manigesting on my first test room and connection, it is now working every time.

Do you have specific repro steps that you find surface the issue each time? This would make debugging easier.

A guess is that somehow the way the pipeline logic is set up for a new room, or an edge case on entering a room, is causing the in/out tasks to immediately conclude. Also open to getting on Slack or a VC to discuss more efficiently whenever works for you, FYI @sayitwithai

@sharvil sharvil mentioned this pull request Aug 17, 2024
@aconchillo
Copy link
Contributor

Closing in favor of #391

@aconchillo aconchillo closed this Aug 28, 2024
@aconchillo aconchillo deleted the cb/lmnt-service branch October 23, 2024 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants