[Page Add][Enhanced Function] Building a Voice-Interactive Chatbot with STT, TTS, and Local LLMs!

[elainedanwu]

We are building a voice-interactive chatbot that leverages cutting-edge technologies such as Speech-to-Text (STT), Text-to-Speech (TTS), and local Large Language Models (LLMs), with a focus on Ollama's local LLM capabilities. This system will enable real-time conversations with users through a privately deployed, Dockerized setup, making it a versatile and secure solution for various applications. We hope the system has the following features:

1. Speech-to-Text (STT) Module:

   • Implement a high-efficiency speech recognition system to convert user voice input into text in real-time.
   • Multilingual support is a significant plus!

2. Text-to-Speech (TTS) Module:

   • Integrate a natural and smooth TTS system that converts chatbot text responses into voice.
   • Prioritize support for multiple languages, especially Chinese and English, with a focus on natural and emotionally responsive outputs.

3. Local LLM (Ollama):

   • Deploy a local LLM using Ollama to understand and generate text responses based on user voice input.
   • Ensure the model operates efficiently with fast response times suitable for real-time voice interactions.
   • Support context retention for understanding the logical flow of continuous conversations.

4. Retrieval-Augmented Generation (RAG) Module:

   • Integrate RAG technology to enhance the accuracy and informativeness of the model's responses.
   • Enable the system to retrieve information from local or preset knowledge bases to enrich conversation content.

5. User Interaction:

   • Implement button-triggered voice input and termination, allowing users to easily control the start and end of conversations.
   • Minimize latency in voice input and output to ensure a smooth interaction experience.

How to Work with Us

We welcome contributions to improve jetson-examples! If you have an example you'd like to share, please submit a pull request. Thank you to all of our contributors! 🙏

If this is your first time joining us, click here to learn how the project works. We follow these steps:

1. Assignments: Leave a comment to let us know you are interested in this project!
2. Submission: Contributors can submit their content via a Pull Request after completing the assignments.
3. Review: Maintainers will merge the submission and record the contributions.

Contributors receive a $300 cash bonus as a token of appreciation for this task.

For any questions or further information, feel free to reach out via the GitHub issues page or contact us at [email protected].

Technical Requirements:

1. Docker Containerization:

   • Package the entire system into one or more Docker containers for merge into jetson-example of one-click deployment.
   • Ensure that the containerized system can run smoothly on the Jetson Orin.

2. Private Deployment:

   • Enable the system to run entirely in a local environment, without relying on external servers or APIs, ensuring user data privacy and security.
   • Support integration with local knowledge bases or databases for customized content generation.

3. Button Trigger:

   • Implement button-triggered functionality, allowing users to control voice input using physical or virtual buttons seamlessly integrated with the STT module.

4. Development Language and Framework:

   • Core modules should be developed in Python, utilizing appropriate open-source libraries for STT, TTS, LLM, and RAG.
   • The user interface (e.g., button control) can be implemented using cross-platform frameworks like Electron or PyQt based on project needs.

5. Testing and Debugging:

   • Conduct multiple rounds of testing to ensure accuracy and fluency in speech recognition, speech synthesis, and LLM responses.
   • Develop comprehensive test cases, including various voice input scenarios, exception handling, and system fault tolerance.

Deliverables:

1. Docker Images and Files:

   • Include all dependencies, configurations, and environment variables required for the system.

2. User Documentation:

   • Provide detailed deployment steps, configuration methods, button usage guides, and solutions for common issues.

3. Source Code and Development Documentation:

   • Include all source code, comments, and detailed development documentation to facilitate maintenance and feature expansion.

4. Test Report:

   • Include results from functional testing, performance testing, and user experience testing.

Project Standards:

• The system must support one-click deployment and stable operation across multiple environments.
• Ensure smooth voice interaction with minimal latency.
• Private deployment must be secure and reliable, with no data leaks.

Reference Links:

• Local RAG based on Jetson with LlamaIndex
• Local Voice Chatbot: Deploy Riva and Llama2 on reComputer
• ChatTTS
• Speech to Text (STT) and Text to Speech (TTS)
• Ollama

[github-actions]

👋 @elainedanwu

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.

[kouroshkarimi]

@elainedanwu
Let's go for it :)