-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding a Streaming demo as well as the Regular one (#15)
Showing
21 changed files
with
865 additions
and
342 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,22 +1,22 @@ | ||
<!DOCTYPE html> | ||
<html lang="en"> | ||
<head> | ||
<meta charset="utf-8" content="initial-scale=1, width=device-width" name="viewport"/> | ||
<meta content="width=device-width, initial-scale=1" name="viewport"/> | ||
<meta content="The main webpage of Jackie Gleason" name="description"> | ||
<meta content="Software Engineer" name="keywords"> | ||
<meta content="Jackie Gleason" name="author"> | ||
<title>THE Jackie Gleason</title> | ||
<link href="/favicon.ico" rel="icon"/> | ||
<meta content="width=device-width, initial-scale=1" name="viewport"/> | ||
<meta content="#000000" name="theme-color"/> | ||
<meta content="Web site created using create-react-app" name="description"/> | ||
<link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" rel="stylesheet"/> | ||
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"/> | ||
</head> | ||
<body> | ||
<noscript>You need to enable JavaScript to run this app.</noscript> | ||
<div id="root"></div> | ||
<script src="/src/index.jsx" type="module"></script> | ||
</body> | ||
<head> | ||
<meta charset="utf-8" content="initial-scale=1, width=device-width" name="viewport"/> | ||
<meta content="width=device-width, initial-scale=1" name="viewport"/> | ||
<meta content="The main webpage of Jackie Gleason" name="description"> | ||
<meta content="Software Engineer" name="keywords"> | ||
<meta content="Jackie Gleason" name="author"> | ||
<title>THE Jackie Gleason</title> | ||
<link href="/favicon.ico" rel="icon"/> | ||
<meta content="width=device-width, initial-scale=1" name="viewport"/> | ||
<meta content="#000000" name="theme-color"/> | ||
<meta content="Web site created using create-react-app" name="description"/> | ||
<link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" rel="stylesheet"/> | ||
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"/> | ||
</head> | ||
<body> | ||
<noscript>You need to enable JavaScript to run this app.</noscript> | ||
<div id="root"></div> | ||
<script src="/src/index.jsx" type="module"></script> | ||
</body> | ||
</html> |
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,125 +1,112 @@ | ||
import {useState} from 'react'; | ||
import {v4 as uuidv4} from 'uuid'; | ||
|
||
export const useChat = () => { | ||
const [messages, setMessages] = useState([]); | ||
const [isLoading, setIsLoading] = useState(false); | ||
const [audioElements, setAudioElements] = useState({}); | ||
|
||
const getEndpoint = (mode) => { | ||
switch (mode) { | ||
case 'openai-chat': | ||
return '/openai'; | ||
case 'openai-image': | ||
return '/openai/image'; | ||
case 'anthropic': | ||
return '/anthropic/anthropic'; | ||
default: | ||
return '/openai'; | ||
const endpoints = { | ||
'openai-chat': '/openai', | ||
'openai-image': '/openai/image', | ||
'anthropic': '/anthropic' | ||
}; | ||
return endpoints[mode] || '/openai'; | ||
}; | ||
|
||
const generateAudioForMessage = async (messageId, text, send) => { | ||
try { | ||
const audioResponse = await fetch(`/openai/audio?message=${encodeURIComponent(text)}`, { | ||
method: 'GET' | ||
}); | ||
if (!audioResponse.ok) throw new Error('Audio stream response was not ok'); | ||
|
||
await send({ | ||
type: 'PLAYBACK', | ||
audioResponse, | ||
responseId: messageId | ||
}); | ||
} catch (error) { | ||
console.error('Audio generation error:', error); | ||
} | ||
}; | ||
|
||
const sendMessage = async (message, mode) => { | ||
const sendMessage = async (message, mode, send) => { | ||
if (!message.trim()) return; | ||
|
||
const messageId = Date.now().toString(); | ||
const userMessage = {type: 'user', content: message, timestamp: new Date(), mode}; | ||
setMessages(msgs => [...msgs, userMessage]); | ||
setIsLoading(true); | ||
const messageId = uuidv4(); | ||
await send({ | ||
type: 'ASK', | ||
message, | ||
speaker: "user", | ||
responder: "ai", | ||
responseId: messageId, | ||
mode | ||
}); | ||
|
||
try { | ||
// First get the text response | ||
const endpoint = getEndpoint(mode); | ||
const textResponse = await fetch( | ||
`${endpoint}?message=${encodeURIComponent(message)}`, | ||
{ | ||
method: 'GET', | ||
headers: { | ||
'Content-Type': 'application/json', | ||
}, | ||
} | ||
); | ||
|
||
const textResponse = await fetch(`${endpoint}?message=${encodeURIComponent(message)}`, { | ||
method: 'GET', | ||
headers: {'Content-Type': 'application/json'} | ||
}); | ||
const responseText = await textResponse.text(); | ||
|
||
// Create and set up audio element | ||
const audio = new Audio(); | ||
await send({type: 'STREAM', chunk: responseText, responseId: messageId}); | ||
await send({type: 'COMPLETE'}); | ||
|
||
// Now use that response text for the audio stream | ||
const audioResponse = await fetch( | ||
`/openai/stream?message=${encodeURIComponent(responseText)}`, | ||
{method: 'GET'} | ||
); | ||
await generateAudioForMessage(messageId, responseText, send); | ||
return true; | ||
|
||
if (!audioResponse.ok) { | ||
throw new Error('Audio stream response was not ok'); | ||
} | ||
} catch (error) { | ||
console.error('Error:', error); | ||
await send({type: 'STREAM_ERROR', error: 'Failed to get response', responseId: messageId}); | ||
return false; | ||
} finally { | ||
setIsLoading(false); | ||
} | ||
}; | ||
|
||
const mediaSource = new MediaSource(); | ||
audio.src = URL.createObjectURL(mediaSource); | ||
|
||
mediaSource.addEventListener('sourceopen', async () => { | ||
const sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg'); | ||
const reader = audioResponse.body.getReader(); | ||
const chunks = []; | ||
|
||
while (true) { | ||
const {done, value} = await reader.read(); | ||
if (done) break; | ||
chunks.push(value); | ||
} | ||
|
||
for (const chunk of chunks) { | ||
await new Promise((resolve) => { | ||
if (!sourceBuffer.updating) { | ||
sourceBuffer.appendBuffer(chunk); | ||
sourceBuffer.addEventListener('updateend', resolve, {once: true}); | ||
} else { | ||
sourceBuffer.addEventListener('updateend', () => { | ||
sourceBuffer.appendBuffer(chunk); | ||
sourceBuffer.addEventListener('updateend', resolve, {once: true}); | ||
}, {once: true}); | ||
} | ||
}); | ||
} | ||
|
||
mediaSource.endOfStream(); | ||
}); | ||
const streamMessage = async (message, send) => { | ||
if (!message.trim()) return; | ||
|
||
const messageId = uuidv4(); | ||
let completeMessage = ''; | ||
|
||
// Store the audio element | ||
setAudioElements(prev => ({ | ||
...prev, | ||
[messageId]: audio | ||
})); | ||
|
||
// Add AI message | ||
setMessages(msgs => [...msgs, { | ||
type: 'ai', | ||
content: responseText, | ||
timestamp: new Date(), | ||
mode, | ||
messageId, | ||
hasAudio: true | ||
}]); | ||
await send({type: 'ASK', message, speaker: "user", responder: "ai", responseId: messageId}); | ||
|
||
try { | ||
const response = await fetch(`/openai/stream?message=${encodeURIComponent(message)}`, { | ||
method: 'GET', | ||
headers: {'Content-Type': 'application/json'} | ||
}); | ||
const reader = response.body.getReader(); | ||
const decoder = new TextDecoder("utf-8"); | ||
|
||
while (true) { | ||
const {done, value} = await reader.read(); | ||
if (done) break; | ||
const chunk = decoder.decode(value, {stream: true}); | ||
completeMessage += chunk; | ||
await send({ | ||
type: 'STREAM', | ||
chunk, | ||
responseId: messageId | ||
}); | ||
} | ||
|
||
await send({type: 'COMPLETE'}); | ||
if(message.mode !== 'openai-image'){ | ||
await generateAudioForMessage(messageId, completeMessage, send); | ||
} | ||
return true; | ||
|
||
} catch (error) { | ||
console.error('Audio Error:', error); | ||
setMessages(msgs => [...msgs, { | ||
type: 'error', | ||
content: 'Error: Failed to get response', | ||
timestamp: new Date(), | ||
mode | ||
}]); | ||
await send({type: 'STREAM_ERROR'}); | ||
return false; | ||
} finally { | ||
setIsLoading(false); | ||
} | ||
}; | ||
|
||
return { | ||
messages, | ||
isLoading, | ||
sendMessage, | ||
audioElements | ||
}; | ||
return {isLoading, sendMessage, streamMessage}; | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,225 @@ | ||
import {assign, createMachine, fromPromise} from "xstate"; | ||
import {v4 as uuidv4} from 'uuid'; | ||
|
||
const handleAudioPlayback = fromPromise(async ({ input, context }) => { | ||
const audio = new Audio(); | ||
const mediaSource = new MediaSource(); | ||
|
||
// Create object URL and set as audio source | ||
audio.src = URL.createObjectURL(mediaSource); | ||
|
||
return new Promise((resolve, reject) => { | ||
mediaSource.addEventListener('sourceopen', async () => { | ||
try { | ||
console.log("Media source opened"); | ||
// Get the audio stream | ||
const response = input.audioResponse; | ||
const reader = response.body.getReader(); | ||
|
||
// Try to determine content type from response headers | ||
const contentType = response.headers.get('content-type'); | ||
let sourceBuffer; | ||
|
||
// Fallback MIME types if content-type header is missing or unsupported | ||
const mimeTypes = [ | ||
contentType, | ||
'audio/mpeg', | ||
'audio/mp4', | ||
'audio/aac', | ||
'audio/webm', | ||
'audio/webm; codecs=opus' | ||
].filter(Boolean); // Remove null/undefined entries | ||
|
||
// Try each MIME type until we find one that works | ||
for (const mimeType of mimeTypes) { | ||
if (MediaSource.isTypeSupported(mimeType)) { | ||
try { | ||
sourceBuffer = mediaSource.addSourceBuffer(mimeType); | ||
break; | ||
} catch (e) { | ||
console.warn(`Failed to create source buffer for ${mimeType}:`, e); | ||
} | ||
} | ||
} | ||
|
||
if (!sourceBuffer) { | ||
throw new Error('No supported audio format found'); | ||
} | ||
|
||
// Function to safely append buffer | ||
const appendBuffer = async (chunk) => { | ||
return new Promise((resolveAppend) => { | ||
if (!sourceBuffer.updating) { | ||
sourceBuffer.appendBuffer(chunk); | ||
sourceBuffer.addEventListener('updateend', resolveAppend, { once: true }); | ||
} else { | ||
sourceBuffer.addEventListener('updateend', () => { | ||
sourceBuffer.appendBuffer(chunk); | ||
sourceBuffer.addEventListener('updateend', resolveAppend, { once: true }); | ||
}, { once: true }); | ||
} | ||
}); | ||
}; | ||
|
||
// Process chunks as they arrive | ||
while (true) { | ||
const { done, value } = await reader.read(); | ||
if (done) break; | ||
|
||
try { | ||
await appendBuffer(value); | ||
} catch (e) { | ||
console.error('Error appending buffer:', e); | ||
// If we hit a quota exceeded error, try to remove some data | ||
if (e.name === 'QuotaExceededError') { | ||
const removeAmount = value.length; | ||
await new Promise(resolveRemove => { | ||
sourceBuffer.remove(0, removeAmount / sourceBuffer.timestampOffset); | ||
sourceBuffer.addEventListener('updateend', resolveRemove, { once: true }); | ||
}); | ||
await appendBuffer(value); | ||
} else { | ||
throw e; | ||
} | ||
} | ||
} | ||
|
||
// All chunks processed, end the stream | ||
if (mediaSource.readyState === 'open') { | ||
mediaSource.endOfStream(); | ||
} | ||
|
||
// Set up audio element event handlers | ||
audio.addEventListener('canplay', () => { | ||
resolve(audio); | ||
}, { once: true }); | ||
|
||
audio.addEventListener('error', (e) => { | ||
reject(new Error('Audio element error: ' + e.error)); | ||
}, { once: true }); | ||
|
||
} catch (error) { | ||
reject(error); | ||
} | ||
}, { once: true }); | ||
|
||
mediaSource.addEventListener('sourceclosed', () => { | ||
reject(new Error('MediaSource was closed')); | ||
}, { once: true }); | ||
|
||
mediaSource.addEventListener('sourceerror', (error) => { | ||
reject(new Error('MediaSource error: ' + error)); | ||
}, { once: true }); | ||
}); | ||
}); | ||
|
||
const askQuestion = fromPromise(async ({input}) => { | ||
try { | ||
console.log(`Adding message ${input.message} by ${input.speaker}`); | ||
return { | ||
...input.messages, | ||
[uuidv4()]: { | ||
type: input.speaker, | ||
content: input.message, | ||
timestamp: new Date() | ||
}, | ||
[input.responseId]: { | ||
type: input.responder, | ||
content: "", | ||
mode: input.mode, | ||
timestamp: new Date() | ||
} | ||
}; | ||
} catch (error) { | ||
console.error('Error in askQuestion:', error); | ||
throw error; | ||
} | ||
}); | ||
|
||
export const simpleMachine = createMachine({ | ||
initial: 'idle', | ||
context: { | ||
messages: {}, | ||
isLoading: false, | ||
audioElements: {}, | ||
errorMessage: "" | ||
}, | ||
states: { | ||
idle: { | ||
on: { | ||
ASK: 'ask', | ||
PLAYBACK: { | ||
target: 'playback', | ||
} | ||
} | ||
}, | ||
playback: { | ||
invoke: { | ||
src: handleAudioPlayback, | ||
input: ({event}) => ({ | ||
responseId: event.responseId, | ||
audioResponse: event.audioResponse | ||
}), | ||
onDone: { | ||
target: 'idle', | ||
actions: assign(({event, context}) => { | ||
context.audio = event.output; | ||
}) | ||
}, | ||
onError: { | ||
target: 'idle', | ||
actions: assign({ | ||
errorMessage: ({event}) => event.data | ||
}) | ||
} | ||
} | ||
}, | ||
ask: { | ||
invoke: { | ||
src: askQuestion, | ||
input: ({event, context}) => ({ | ||
messages: context.messages, | ||
message: event.message, | ||
responseId: event.responseId, | ||
speaker: event.speaker, | ||
responder: event.responder, | ||
mode: event.mode | ||
}), | ||
onDone: { | ||
actions: assign(({event, context}) => { | ||
context.messages = event.output | ||
}) | ||
}, | ||
onError: { | ||
target: 'idle', | ||
actions: assign(({event, context}) => { | ||
context.errorMessage = event.data | ||
}) | ||
} | ||
}, | ||
on: { | ||
STREAM: { | ||
actions: assign(({context, event}) => { | ||
const currentValue = context.messages[event.responseId]; | ||
context.messages[event.responseId] = { | ||
...currentValue, | ||
content: currentValue.content + event.chunk | ||
} | ||
}) | ||
}, | ||
STREAM_ERROR: { | ||
target: 'idle', | ||
actions: assign({ | ||
// TODO: Add error message to the responseId | ||
errorMessage: ({event}) => { | ||
return event.error | ||
} | ||
}) | ||
}, | ||
COMPLETE: { | ||
target: 'idle' | ||
} | ||
} | ||
} | ||
} | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
// StateProvider.jsx | ||
import React, {createContext, useContext, useEffect} from 'react'; | ||
import {useMachine} from '@xstate/react'; | ||
import {simpleMachine} from "./StateMachine.js"; | ||
|
||
const StateContext = createContext(); | ||
|
||
export const StateProvider = ({children}) => { | ||
const [state, send] = useMachine(simpleMachine); | ||
|
||
useEffect(() => { | ||
send( | ||
{type: 'FETCH'} | ||
) | ||
}, []); | ||
|
||
return ( | ||
<StateContext.Provider value={{state, send}}> | ||
{children} | ||
</StateContext.Provider> | ||
); | ||
}; | ||
|
||
export const useStateContext = () => { | ||
return useContext(StateContext); | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters