Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: upgrading to 0.0.49 bot does not reply #785

Open
sadimoodi opened this issue Dec 5, 2024 · 8 comments
Open

BUG: upgrading to 0.0.49 bot does not reply #785

sadimoodi opened this issue Dec 5, 2024 · 8 comments

Comments

@sadimoodi
Copy link

The following code that used to work with 0.0.43 does not work now with current version = 0.0.49 as the bot does not reply, rolling back to 0.0.43 solves the problem:

import asyncio, argparse
import aiohttp
import os
import sys
from pipecat.frames.frames import EndFrame
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
    LLMAssistantResponseAggregator, LLMUserResponseAggregator)
from pipecat.services.openai import OpenAILLMService, OpenAITTSService
from pipecat.services.ollama import OLLamaLLMService
from pipecat.services.xtts import XTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer #, VADParams
from pipecat.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from loguru import logger

from dotenv import load_dotenv
load_dotenv(override=True)

logger.remove(0)
logger.add(sys.stderr, level="DEBUG")


async def main(room_url, token):
    
    async with aiohttp.ClientSession() as session:
        logger.info("Just started bot")
        transport = DailyTransport(
            room_url,
            token,
            "AI BOT",
            DailyParams(
                # audio_out_sample_rate=24000,
                audio_out_enabled=True,
                transcription_enabled=True,
                vad_enabled=True,
                vad_analyzer=SileroVADAnalyzer() #(params=VADParams(stop_secs=0.3)),
            ))
        
        llm = OpenAILLMService(
            api_key=os.getenv("OPENAI_API_KEY"),
            model="gpt-4o-mini")
        
        # otts = OpenAITTSService()
        tts = XTTSService(
                aiohttp_session=session,
                voice_id="Brenda Stern", #"Claribel Dervla"
                language="en",
                base_url="http://localhost:8001"
            )


        messages = [
            {
                "role": "system",
                "content": "You are a helpful AI BOT.Your name is Kacy, Keep your answers concise and very short,do not exceed 4 sentences, ask follow up questions if you need to, DO NOT HALLUCINATE, ACCURACY MODE = ON."
            },
        ]

        context = OpenAILLMContext(messages)
        context_aggregator = llm.create_context_aggregator(context)

        pipeline = Pipeline([
            transport.input(),   # Transport user input
            context_aggregator.user(),              # User responses
            llm,                 # LLM
            tts,                 # TTS
            transport.output(),  # Transport bot output
            context_aggregator.assistant(),              # Assistant spoken responses
        ])

        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))

        @transport.event_handler("on_first_participant_joined")
        async def on_first_participant_joined(transport, participant):
            await transport.capture_participant_transcription(participant["id"])
            logger.info("First participant joined")
            messages.append({"role": "system", "content": "Please introduce yourself to the user."})
            await task.queue_frames([LLMMessagesFrame(messages)])

        @transport.event_handler("on_participant_joined")
        async def on_participant_joined(transport, participant):
            await transport.capture_participant_transcription(participant["id"])
            logger.info("participant joined")


        @transport.event_handler("on_participant_left")
        async def on_participant_left(transport, participant, reason):
            await task.queue_frame(EndFrame())
            logger.info("Partcipant left. Exiting.")

        @transport.event_handler("on_call_state_updated")
        async def on_call_state_updated(transport, state):
            logger.info("Call state %s " % state)
            if state == "left":
                await task.queue_frame(EndFrame())
        
        runner = PipelineRunner()
        await runner.run(task)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="RTVI Bot Example")
    parser.add_argument("-u", type=str, help="Room URL")
    parser.add_argument("-t", type=str, help="Token")
    #parser.add_argument("-c", type=str, help="Bot configuration blob")
    p_config = parser.parse_args()

    #bot_config = json.loads(config.c) if config.c else {}
    #logger.warning()
    if p_config.u and p_config.t:
        asyncio.run(main(p_config.u, p_config.t))
    else:
        logger.error("Room URL and Token are required")

when the bot joins the room, i see the following in the logs, the bot does not reply at all.

2024-12-04 21:34:51.673 | INFO     | __main__:main:32 - Just started bot
2024-12-04 21:34:51.673 | INFO     | pipecat.audio.vad.vad_analyzer:set_params:69 - Setting VAD params to: confidence=0.7 start_secs=0.2 stop_secs=0.8 min_volume=0.6
2024-12-04 21:34:51.673 | DEBUG    | pipecat.audio.vad.silero:__init__:114 - Loading Silero VAD model...
2024-12-04 21:34:51.724 | DEBUG    | pipecat.audio.vad.silero:__init__:136 - Loaded Silero VAD
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking PipelineSource#0 -> DailyInputTransport#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking DailyInputTransport#0 -> OpenAIUserContextAggregator#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAIUserContextAggregator#0 -> OpenAILLMService#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAILLMService#0 -> XTTSService#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking XTTSService#0 -> DailyOutputTransport#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking DailyOutputTransport#0 -> OpenAIAssistantContextAggregator#0
2024-12-04 21:34:51.759 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAIAssistantContextAggregator#0 -> PipelineSink#0
2024-12-04 21:34:51.760 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking Source#0 -> Pipeline#0
2024-12-04 21:34:51.760 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking Pipeline#0 -> Sink#0
2024-12-04 21:34:51.760 | DEBUG    | pipecat.pipeline.runner:run:27 - Runner PipelineRunner#0 started running PipelineTask#0
2024-12-04 21:34:54.459 | INFO     | pipecat.transports.services.daily:join:310 - Joining https://sadimoodi.daily.co/W6Huv4tZg7NvJzsupgmy
2024-12-04 21:34:56.330 | INFO     | __main__:on_call_state_updated:122 - Call state joining 
2024-12-04 21:34:57.783 | INFO     | pipecat.transports.services.daily:on_participant_joined:575 - Participant joined 9fe68fc1-ca04-4912-a39d-cc6a0c94f54f
2024-12-04 21:34:57.784 | INFO     | __main__:on_first_participant_joined:105 - First participant joined
2024-12-04 21:34:57.794 | INFO     | __main__:on_participant_joined:112 - participant joined
2024-12-04 21:34:58.023 | INFO     | __main__:on_call_state_updated:122 - Call state joined 
2024-12-04 21:34:59.390 | INFO     | pipecat.transports.services.daily:join:329 - Joined https://sadimoodi.daily.co/W6Huv4tZg7NvJzsupgmy
2024-12-04 21:34:59.390 | INFO     | pipecat.transports.services.daily:_start_transcription:345 - Enabling transcription with settings language='en' tier=None model='nova-2-general' profanity_filter=True redact=False endpointing=True punctuate=True includeRawResponse=True extra={'interim_results': True}
2024-12-04 21:35:00.541 | DEBUG    | pipecat.transports.services.daily:on_transcription_started:593 - Transcription started: {'transcriptId': '0703e74e-c268-4c26-ae24-da3b69b5708c', 'model': 'nova-2-general', 'language': 'en', 'startedBy': 'ebb0daaa-a45d-47cd-86b3-e4c81ede3f8e'}
2024-12-04 21:35:00.542 | DEBUG    | pipecat.services.openai:_stream_chat_completions:174 - Generating chat: [{"role": "system", "content": "You are a helpful psychologist.Your name is Kacy, you are able to understand and analyze human emotions, Keep your answers concise and very short,do not exceed 4 sentences, ask follow up questions if you need to, DO NOT HALLUCINATE, ACCURACY MODE = ON.", "name": "system"}, {"role": "system", "content": "Please introduce yourself to the user.", "name": "system"}]
@aconchillo
Copy link
Contributor

@sadimoodi I believe this might have been due to OpenAI being down for some period of time yesterday. Could you try again?

@sadimoodi
Copy link
Author

@aconchillo i tried again several times, i even tested a lot during the last few days, but no success, the server still shows this in the log :

2024-12-05 21:15:47.053 | INFO     | __main__:main:32 - Just started bot
2024-12-05 21:15:47.053 | INFO     | pipecat.audio.vad.vad_analyzer:set_params:69 - Setting VAD params to: confidence=0.7 start_secs=0.2 stop_secs=0.8 min_volume=0.6
2024-12-05 21:15:47.053 | DEBUG    | pipecat.audio.vad.silero:__init__:114 - Loading Silero VAD model...
2024-12-05 21:15:47.106 | DEBUG    | pipecat.audio.vad.silero:__init__:136 - Loaded Silero VAD
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking PipelineSource#0 -> DailyInputTransport#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking DailyInputTransport#0 -> OpenAIUserContextAggregator#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAIUserContextAggregator#0 -> OpenAILLMService#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAILLMService#0 -> XTTSService#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking XTTSService#0 -> DailyOutputTransport#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking DailyOutputTransport#0 -> OpenAIAssistantContextAggregator#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking OpenAIAssistantContextAggregator#0 -> PipelineSink#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking Source#0 -> Pipeline#0
2024-12-05 21:15:47.139 | DEBUG    | pipecat.processors.frame_processor:link:143 - Linking Pipeline#0 -> Sink#0
2024-12-05 21:15:47.140 | DEBUG    | pipecat.pipeline.runner:run:27 - Runner PipelineRunner#0 started running PipelineTask#0
2024-12-05 21:15:49.736 | INFO     | pipecat.transports.services.daily:join:310 - Joining https://sadimoodi.daily.co/zOmXEuZkt7sT0u9i3y89
2024-12-05 21:15:51.657 | INFO     | __main__:on_call_state_updated:122 - Call state joining 
2024-12-05 21:15:53.163 | INFO     | pipecat.transports.services.daily:on_participant_joined:575 - Participant joined 9ed8580c-554b-46ef-9757-e9c0a5a1aa95
2024-12-05 21:15:53.171 | INFO     | __main__:on_first_participant_joined:105 - First participant joined
2024-12-05 21:15:53.171 | INFO     | __main__:on_participant_joined:112 - participant joined
2024-12-05 21:15:53.413 | INFO     | __main__:on_call_state_updated:122 - Call state joined 
2024-12-05 21:15:55.120 | INFO     | pipecat.transports.services.daily:join:329 - Joined https://sadimoodi.daily.co/zOmXEuZkt7sT0u9i3y89
2024-12-05 21:15:55.121 | INFO     | pipecat.transports.services.daily:_start_transcription:345 - Enabling transcription with settings language='en' tier=None model='nova-2-general' profanity_filter=True redact=False endpointing=True punctuate=True includeRawResponse=True extra={'interim_results': True}
2024-12-05 21:15:56.324 | DEBUG    | pipecat.transports.services.daily:on_transcription_started:593 - Transcription started: {'model': 'nova-2-general', 'startedBy': 'e9984ce0-cfae-43c2-9192-5c99dc5221d8', 'transcriptId': '54a9b188-2efb-4928-8829-a20bf43e936a', 'language': 'en'}
2024-12-05 21:15:56.325 | DEBUG    | pipecat.services.openai:_stream_chat_completions:174 - Generating chat: [{"role": "system", "content": "You are a helpful psychologist.Your name is Kacy, you are able to understand and analyze human emotions, Keep your answers concise and very short,do not exceed 4 sentences, ask follow up questions if you need to, DO NOT HALLUCINATE, ACCURACY MODE = ON.", "name": "system"}, {"role": "system", "content": "Please introduce yourself to the user.", "name": "system"}]

@aconchillo
Copy link
Contributor

I've tried the example we have for XTTS (https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07i-interruptible-xtts.py) and I had no issues. This is from main but I don't think anything has changed that would affect this. If you go back to Pipecat 0.0.43 it works?

@sadimoodi
Copy link
Author

Yes, exactly the same code runs smoothly on V 0.0.43

@aconchillo
Copy link
Contributor

Yes, exactly the same code runs smoothly on V 0.0.43

Would it be too much to ask you to try from the main branch?

@sadimoodi
Copy link
Author

U mean install from the main branch? I don't see any manual installation instructions, only using pop install.
Else, can u clarify?

@aconchillo
Copy link
Contributor

U mean install from the main branch? I don't see any manual installation instructions, only using pop install. Else, can u clarify?

Yes, you can follow https://github.com/pipecat-ai/pipecat?tab=readme-ov-file#hacking-on-the-framework-itself

Let me know if you need help.

@sadimoodi
Copy link
Author

@aconchillo thanks for your suggestion, its now working with this version "0.0.51.dev6"
Do u have any idea why would it not work with a pip install?
I dont fee any difference in voice quality between V 0.0.43 and the current version, are there any other advantages to upgrade to this version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aconchillo @sadimoodi