Skip to content

Mic button #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AnkushMalaker merged 7 commits into from
Sep 9, 2025
Merged

Mic button #85

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6c4eb93
Add Foreground Service to keep WebSocket alive
Sep 6, 2025
90f610c
Update package-lock for added Notifee dependency
Sep 6, 2025
8f36458
test build
AnkushMalaker Sep 8, 2025
3b4ef54
working!
AnkushMalaker Sep 9, 2025
a1552ad
visualization fixed
AnkushMalaker Sep 9, 2025
3d83714
update docs
AnkushMalaker Sep 9, 2025
4032654
follow coderabbit :pray::skin-tone-4:
AnkushMalaker Sep 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@
2. Configure your API keys and service settings manually
3. Start with `docker compose up --build -d`

**Mobile App + Phone Audio (Latest Feature):**
1. **Setup Backend**: Follow Advanced Backend setup above
2. **Install Mobile App**: Go to `app/` directory and run `npm install && npm start`
3. **Configure Mobile App**: Point to your backend IP in app settings
4. **Enable Phone Audio**: Tap "Stream Phone Audio" in app for direct microphone streaming
5. **Grant Permissions**: Allow microphone access when prompted
6. **Start Streaming**: Speak into phone for real-time processing with live audio visualization

**Documentation:** See `CLAUDE.md` and `Docs/init-system.md` for detailed setup guide

## Overview
Expand Down
78 changes: 76 additions & 2 deletions app/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ React Native mobile application for connecting OMI devices and streaming audio t
## Features

- **OMI Device Integration**: Connect via Bluetooth and stream audio
- **Phone Audio Streaming**: Use phone's microphone directly (NEW)
- **Cross-Platform**: iOS and Android support using React Native
- **Real-time Audio Streaming**: OPUS audio transmission to backend services
- **WebSocket Communication**: Efficient real-time data transfer
- **Backend Selection**: Configure connection to any compatible backend
- **Live Audio Visualization**: Real-time audio level meters

## Quick Start

Expand Down Expand Up @@ -115,7 +117,7 @@ The app connects to any backend that accepts OPUS audio streams:
- Good for testing and development
- WebSocket endpoint: `/ws`

2. **Advanced Backend** (`backends/advanced-backend/`)
2. **Advanced Backend** (`backends/advanced/`)
- Full transcription and memory features
- Real-time processing with speaker recognition
- WebSocket endpoint: `/ws_pcm`
Expand Down Expand Up @@ -149,6 +151,60 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
- Public: `wss://[your-domain]/ws_pcm`
5. **Save configuration**

## Phone Audio Streaming (NEW)

### Overview
Stream audio directly from your phone's microphone to Friend-Lite backend, bypassing Bluetooth devices. This feature provides a direct audio input method for users who want to use their phone as the audio source.

### Features
- **Direct Microphone Access**: Use phone's built-in microphone
- **Real-time Audio Streaming**: Live audio processing with visualization
- **Seamless Integration**: Switch between Bluetooth and phone audio modes
- **Cross-Platform**: Works on both iOS and Android
- **Live Audio Meters**: Visual feedback showing audio levels in real-time

### Setup & Usage

#### Enable Phone Audio Streaming
1. **Open Friend-Lite app**
2. **Configure Backend Connection** (see Backend Configuration section)
3. **Grant Microphone Permissions** when prompted
4. **Tap "Stream Phone Audio" button** in main interface
5. **Start speaking** - audio streams in real-time to backend

#### Requirements
- **iOS**: iOS 13+ with microphone permissions
- **Android**: Android API 21+ with microphone permissions
- **Network**: Stable connection to Friend-Lite backend
- **Backend**: Advanced backend running with `/ws_pcm` endpoint

Comment on lines +175 to +180
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify minimum OS/API levels stated.

React Native/Expo baselines may be higher than iOS 13 / API 21 with RN 0.79 + Expo SDK 53.

🌐 Web query:

What minimum iOS version and Android API level are supported by Expo SDK 53 with React Native 0.79?

💡 Result:

Expo SDK 53 (React Native 0.79) requires iOS 15.1+ and Android 7.0+ (minSdkVersion API level 24); compile/target SDK = 35. [1][2]

Sources:
[1] Expo docs — Supported OS / SDK table.
[2] Expo SDK 53 changelog.

Update minimum OS/API levels in README
Change iOS requirement to 15.1+ and Android minSdkVersion to API level 24 (Android 7.0+) in app/README.md (lines 175–180) to align with Expo SDK 53 / React Native 0.79.

🤖 Prompt for AI Agents 
In app/README.md around lines 175 to 180, the platform minimums are outdated;
update the iOS requirement to "iOS 15.1+" and change the Android requirement to
"Android API 24+ (Android 7.0+)" or "minSdkVersion 24" where the README
references Android API level, ensuring the text reflects Expo SDK 53 / React
Native 0.79 compatibility and keeps the existing notes about microphone
permissions and backend/network requirements unchanged.

#### Switching Audio Sources
- **Mutual Exclusion**: Cannot use Bluetooth and phone audio simultaneously
- **Automatic Detection**: App disables conflicting options when one is active
- **Visual Feedback**: Clear indicators show active audio source

### Troubleshooting Phone Audio

#### Audio Not Streaming
- **Check Permissions**: Ensure microphone access granted
- **Verify Backend URL**: Confirm `ws://[ip]:8000/ws_pcm` format
- **Network Connection**: Test backend connectivity
- **Authentication**: Verify JWT token is valid

#### Poor Audio Quality
- **Check Signal Strength**: Ensure stable network connection
- **Reduce Background Noise**: Use in quiet environment
- **Restart Recording**: Stop and restart phone audio streaming

#### Permission Issues
- **iOS**: Settings > Privacy & Security > Microphone > Friend-Lite
- **Android**: Settings > Apps > Friend-Lite > Permissions > Microphone

#### No Audio Level Visualization
- **Restart App**: Close and reopen the application
- **Check Audio Input**: Ensure microphone is working in other apps
- **Backend Logs**: Verify backend is receiving audio data

## User Workflow

### Device Connection
Expand All @@ -163,6 +219,7 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm

### Audio Streaming

#### Option 1: Bluetooth Audio (OMI Device)
1. **Configure backend connection** (see Configuration Steps above)
2. **Test connection**:
- Tap "Test Connection" in settings
Expand All @@ -172,6 +229,16 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
- Speak into OMI device
- Audio streams to backend in real-time

#### Option 2: Phone Audio Streaming
1. **Configure backend connection** (see Configuration Steps above)
2. **Enable phone audio**:
- Tap "Stream Phone Audio" button
- Grant microphone permissions when prompted
3. **Start speaking**:
- Speak directly into phone microphone
- Watch real-time audio level visualization
- Audio streams to backend automatically

### Monitoring

1. **Check connection status** in app header
Expand All @@ -197,6 +264,13 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
- Test with simple backend first
- Monitor backend logs for connection attempts

**Phone Audio Streaming Issues:**
- Grant microphone permissions in device settings
- Ensure stable network connection to backend
- Restart phone audio streaming if no data flowing
- Check backend logs for audio data reception
- Verify JWT authentication token is valid

**Build Errors:**
- Clear Expo cache: `npx expo start --clear`
- Clean prebuild: `npx expo prebuild --clean`
Expand Down Expand Up @@ -290,5 +364,5 @@ BluetoothService.onAudioData = (audioBuffer) => {

- **[Backend Setup](../backends/)**: Choose and configure backend services
- **[Quick Start Guide](../quickstart.md)**: Complete system setup
- **[Advanced Backend](../backends/advanced-backend/)**: Full-featured backend option
- **[Advanced Backend](../backends/advanced/)**: Full-featured backend option
- **[Simple Backend](../backends/simple/)**: Basic backend for testing
43 changes: 40 additions & 3 deletions app/app.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@
],
"ios": {
"supportsTablet": true,
"bundleIdentifier": "com.cupbearer5517.friendlite"
"bundleIdentifier": "com.cupbearer5517.friendlite",
"infoPlist": {
"NSMicrophoneUsageDescription": "Friend-Lite needs access to your microphone to stream audio to the backend for processing."
}
Comment on lines +21 to +23
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix iOS build failure: add ITSAppUsesNonExemptEncryption.

The pipeline error indicates this key is missing. Set it explicitly (typically false unless you ship custom encryption).

Apply this diff:

       "infoPlist": {
         "NSMicrophoneUsageDescription": "Friend-Lite needs access to your microphone to stream audio to the backend for processing."
+        ,"ITSAppUsesNonExemptEncryption": false
       }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"infoPlist": {
"NSMicrophoneUsageDescription": "Friend-Lite needs access to your microphone to stream audio to the backend for processing."
}
"infoPlist": {
"NSMicrophoneUsageDescription": "Friend-Lite needs access to your microphone to stream audio to the backend for processing."
,"ITSAppUsesNonExemptEncryption": false
}
🤖 Prompt for AI Agents 
In app/app.json around lines 21 to 23, add the ITSAppUsesNonExemptEncryption key
to the infoPlist object (set to false unless your app ships custom encryption)
to satisfy iOS build pipeline requirements; update the JSON by inserting
"ITSAppUsesNonExemptEncryption": false alongside the existing
NSMicrophoneUsageDescription entry so the Info.plist includes this explicit key.

},
"android": {
"adaptiveIcon": {
Expand All @@ -29,12 +32,29 @@
"android.permission.BLUETOOTH",
"android.permission.BLUETOOTH_ADMIN",
"android.permission.BLUETOOTH_CONNECT",
"android.permission.ACCESS_NETWORK_STATE"
"android.permission.ACCESS_NETWORK_STATE",
"android.permission.FOREGROUND_SERVICE",
"android.permission.FOREGROUND_SERVICE_DATA_SYNC",
"android.permission.POST_NOTIFICATIONS",
"android.permission.RECORD_AUDIO"
],
"usesCleartextTraffic": true
},
"newArchEnabled": true,
"plugins": [
[
"@siteed/expo-audio-studio",
{
"enablePhoneStateHandling": true,
"enableNotifications": true,
"enableBackgroundAudio": true,
"enableDeviceDetection": true,
"iosBackgroundModes": { "useProcessing": true },
"iosConfig": {
"microphoneUsageDescription": "We use the mic for live audio streaming"
}
}
],
[
"react-native-ble-plx",
{
Expand All @@ -45,11 +65,28 @@
"bluetoothAlwaysPermission": "This app uses Bluetooth to connect to and interact with nearby BLE devices."
}
],
[
"./plugins/with-ws-fgs",
{
"android": {
"permissions": [
"FOREGROUND_SERVICE",
"FOREGROUND_SERVICE_DATA_SYNC",
"POST_NOTIFICATIONS"
]
}
}
],
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Declare microphone FGS type in the custom plugin too.

Ensure the manifest gets a service with foregroundServiceType="microphone" and that the permission is listed.

Apply this diff:

       [
         "./plugins/with-ws-fgs",
         {
           "android": {
             "permissions": [
               "FOREGROUND_SERVICE",
               "FOREGROUND_SERVICE_DATA_SYNC",
-              "POST_NOTIFICATIONS"
+              "POST_NOTIFICATIONS",
+              "FOREGROUND_SERVICE_MICROPHONE"
             ]
           }
         }
       ],

Then verify your plugin injects android:foregroundServiceType="microphone" on the service. If not, we should extend it.

🏁 Script executed:

#!/bin/bash
# Check plugin and generated AndroidManifest for microphone FGS type
rg -n "with-ws-fgs|foregroundServiceType.*microphone|FOREGROUND_SERVICE_MICROPHONE" -S -C2

Length of output: 743

🏁 Script executed:

#!/bin/bash
# Display the custom plugin code to verify if it injects the service with foregroundServiceType="microphone"
sed -n '1,200p' app/plugins/with-ws-fgs.js

Length of output: 1910

Include the microphone flag and permission in your plugin config

In app/app.json, update the plugin entry to pass microphone: true and add the FOREGROUND_SERVICE_MICROPHONE permission:

       [
         "./plugins/with-ws-fgs",
         {
+          "microphone": true,
           "android": {
             "permissions": [
               "FOREGROUND_SERVICE",
               "FOREGROUND_SERVICE_DATA_SYNC",
-              "POST_NOTIFICATIONS"
+              "POST_NOTIFICATIONS",
+              "FOREGROUND_SERVICE_MICROPHONE"
             ]
           }
         }
       ],

This ensures your custom plugin will inject android:foregroundServiceType="dataSync|microphone" and declare the microphone FGS permission.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"./plugins/with-ws-fgs",
{
"android": {
"permissions": [
"FOREGROUND_SERVICE",
"FOREGROUND_SERVICE_DATA_SYNC",
"POST_NOTIFICATIONS"
]
}
}
],
"./plugins/with-ws-fgs",
{
"microphone": true,
"android": {
"permissions": [
"FOREGROUND_SERVICE",
"FOREGROUND_SERVICE_DATA_SYNC",
"POST_NOTIFICATIONS",
"FOREGROUND_SERVICE_MICROPHONE"
]
}
}
],
🤖 Prompt for AI Agents 
In app/app.json around lines 69 to 79, the plugin config for
"./plugins/with-ws-fgs" is missing the microphone flag and corresponding
permission; update the plugin object to include "microphone": true and add
"FOREGROUND_SERVICE_MICROPHONE" to the android.permissions array so the plugin
will inject android:foregroundServiceType="dataSync|microphone" and declare the
microphone FGS permission.

[
"expo-build-properties",
{
"android": {
"usesCleartextTraffic": true
"usesCleartextTraffic": true,
"compileSdkVersion": 35,
"targetSdkVersion": 35,
"extraMavenRepos": [
"../../node_modules/@notifee/react-native/android/libs"
]
}
}
]
Expand Down
Loading
Loading