Feature request: audio stream

[crxporter]

If I understand correctly, some cameras have two-way audio. This plugin should work for audio coming from a camera playing with the RTSP stream.

What about if I have a camera with a speaker? It should be section 9.102 of the homekit spec describing audio from a microphone to a speaker. I don't actually currently have a camera with two-way audio but I might add one later.

It would be fun to talk to my wife and kids over the homekit camera!

---

Are others interested in this? Is this something that could be added sometime? Does it already work if we set up the right services?

Please don't rush anything, just wondering if it's part of hap node js.

[Shaquu]

About two way audio.
It looks like guys from hap-nodes has to solve it firstly.
I think someone mentioned here that it is possible to do with ffmpeg but can be choppy. Though not too many mentions of working two way audio.

About one way audio.
It is said in the discussion in the link above that one way audio should work good enough.
"You should use AAC-ELD" ~someone said.
So I would start with configuring ffmpeg to play audio and video from camera in VLC or any video player. Then we can move to HomeKit and see if configuration is ease or not. If not then I have to order some cheap camera supporting audio to test it.

[crxporter]

It seems like testing could be done by sending the audio stream buffer (from HomeKit) to a file then seeing if that audio file could play.

Honestly I don't even know if the camera I'm looking at (unifi g3 micro) can play audio from anything but the official Unifi system.

I currently only have cameras with one-way audio (camera to homekit) and it is working properly.

[crxporter]

Some interesting discussion about this here

Way above my understanding but seems like it’s at least possible.

[crxporter]

Just for fun tonight I added a linked "Speaker" service to one of my 3 cameras. Interestingly it added a microphone icon in the Home app to all 3 of the cameras - even though they are each on their own bridge. Also interesting, when I enabled/disabled the little microphone icon it would trigger the audio stream from the camera microphone (same as the speaker icon would normally).

I'd guess it all comes down with properly configuring the audio streams. Also seems like nobody else has spoken up asking for this so it's not something to spend time on at this point.

Maybe if we ever get to version 2.0 😉

[crxporter]

Closing - links are available to watch for future users/developers (future versions of ourselves?).

[crxporter]

I just scored a beta release unifi video doorbell. It's got two-way audio features. It should arrive in a week or so. I'll be working with everyone I can find to figure out the two way audio on it - I'm re-opening this issue to document my findings.

@Shaquu - any chance we'll get a beta release with the new hap-nodejs video features sometime soon? I'm not actually sure what changed there but I'd like to work with the latest and greatest...

[burnbrigther]

I hope you get some progress with this. 2-way audio with security cameras and homebridge has been a problem for a few years now.

[Shaquu]

@Shaquu - any chance we'll get a beta release with the new hap-nodejs video features sometime soon?

Yes!

Edit: npm i [email protected]

[crxporter]

Awesome, I'll be setting up a dev environment for this - my doorbell should arrive tomorrow.

I'm also looking for help from this project, there are a couple of links there for a ring plugin that supports 2-way audio.

The question will be how it works with my particular camera... I don't know where or how to send the audio, that's another place things may get difficult.

[Shaquu]

I wanted to say that I will buy UniFi 2-way audio device to support you directly and then I have opened their site and saw that prices... Hah :)

I will try to contact them anyway.
Also do you have an access to UniFi protect?
If you want we can try to cooperate on that online to figure out how to plug it into nrchkb.

[crxporter]

Yeah, it's not a cheap system! I went all in early 2019 when my second "cheap" router in 6 months burned out. Then I've been collecting to the point where I now have way too much of their stuff. Oops.

I currently have a dream machine pro and 3 g3 flex cameras. They're all working just great in nrchkb. The dream machine runs UniFi protect. I'm polling the API every second to detect motion, it took me a little bit but it's been running for weeks without issues.

Here's a screenshot of a single camera data that I get back every second. I marked in red where I hope the doorbell might have helpful information for 2-way audio... We'll see tomorrow!

[crxporter]

Oh also for reference here's my current functional camera setup - still image, video, and audio all working.

[Shaquu]

There might be an option to export a rtsp url from UniFi Protext? What is the full name for the new Doorbell device?

Also I will have to write some code to support new camera features available in hap-nodejs
Code we use right now is marked as deprecated.

[crxporter]

The doorbell device is currently in their "early access" program. It's not generally available at this point, still in their beta testing program. With that, there's very little out there about the device. It's the "UniFi Protect G4 Doorbell" - honestly the best information will likely be around reddit.

I'll of course have much more information once I get it set up at my house, then I can really start fiddling.

The RTSP URL is available in unifi protect. The unknown is where (and how) the ACC audio from Home app will need to be sent.

Here's a quick screenshot of the doorbell product page:

[crxporter]

Alright, the camera came today. The camera settings that work with my other cameras also work for the bell - no surprise there.

I will need to change the aspect ratio as the bell is a 4:3 aspect ratio, no problem I can handle that.

I need to watch the timestamp in lastRing and change it to a programmable switch event (for homekit notifications). Again, something I can easily handle, it's the same setup as the camera motion detection.

I went ahead and added a linked Speaker service on the bell and the little microphone appears to enable 2-way audio. And that's where I start needing help.

I don't see anything different coming through the API when I'm using the official app to run 2-way audio. I don't see any different ports open up on the unifi protect controller or the camera when I run 2-way audio. I was hoping to see something in "talkback settings" but the port 7004 is not open anywhere.

I'm not sure where to go next. If anyone has ideas I'll try them out. Interesting information from the API might be "talkback settings" - my non speaker cameras return different information here.

If I can do some packet sniffing to watch what the bell is up to when I'm using the official unifi app to do 2-way audio maybe that would provide some useful information. I don't know how to do that though, someone could point me to some instructions?

I think that's enough for today. It's a cool little device, I'm hopeful that 2-way audio will get working both with NRCHKB and with my hardware - eventually.

[Shaquu]

Surely we will have to add support for different camera setups here. One for talk-back port.
I will need some time to implement this since coding with ffmpeg is messy.
Also we will start using this ffmpeg-for-homebridge as default (fallback will be our ffmpeg installed manually).

Also solution is here:
https://github.com/dgreif/ring/blob/master/homebridge/camera-source.ts
Sadly author decided to use terrible comma operator for chain operation so code is even worse to read.

Regarding the port. I think it is the port on camera (or some UniFi system) to receive audio.

[crxporter]

I’m planning to set up Wireshark today or tomorrow and watch what changes to/from the camera when I use the Unifi app. I’m hoping to make a discovery... I’m hoping to see a UDP encoded AAC stream...

[Shaquu]

@crxporter when you are ready you can write on Discord for live support (https://discord.gg/H9CWzXv)
I might be able to help you some how with Wireshark or other topics.

[Shaquu]

I believe that audio support depends on device you are using.
As soon Plugin System is here I will work on audio stream for RTSP and 2 way audio for UniFi cameras. The rest devices to be implemented in future