how to replace a remote audio stream and send back a new stream to the client?

[JohnMarsden24]

I'm trying to manipulate the audio samples that come from the client and return back the altered stream from the server. My thought would be: create a new local stream from RTCAudioSource/createTrack() and a sink from RTCAudioSink using the original audio stream, get the samples from the original stream using the sink, manipulate them, push them into the new stream using .onData and then replace the original stream with my new stream. Using the audio-video-loopback as a start point I've got this:

`"use strict";

const { RTCAudioSink } = require("wrtc").nonstandard;
const { RTCAudioSource } = require("wrtc").nonstandard;

function beforeOffer(peerConnection) {
const audioTransceiver = peerConnection.addTransceiver("audio");
const videoTransceiver = peerConnection.addTransceiver("video");

const { track } = audioTransceiver.receiver;

const source = new RTCAudioSource();
const newTrack = source.createTrack();
const sink = new RTCAudioSink(track);

const sampleRate = 48000;
const samples = new Int16Array(sampleRate / 100); // 10 ms of 16-bit mono audio

const dataObj = {
samples,
sampleRate,
};

const interval = setInterval(() => {
// Update audioData in some way before sending.
source.onData(dataObj);
audioTransceiver.sender.replaceTrack(newTrack);
});

sink.ondata = (data) => {
// Do something with the received audio samples.
const newArr = data.samples.map((el) => el * 2);
dataObj[samples] = newArr;
};

return Promise.all([
audioTransceiver.sender.replaceTrack(track),
videoTransceiver.sender.replaceTrack(videoTransceiver.receiver.track),
]);
}

module.exports = { beforeOffer };`

Now this doesn't work, also tried returning the new stream in the replaceTrack return promise and I get no audio. I would prefer to just work with the original stream and edit that but I can't see any way of updating it without having to create a new track and using the onData method.

Any ideas?

[markandrus]

You are calling replaceTrack inside setInterval. I don't think you should do that. You should replaceTrack once.

Also, rather than setInterval, it would probably be better to call the source's onData from within the sink's ondata. But please understand this is not a great way to do audio… it's just a hack.

When you say "get no audio", do you mean server-side or browser-side?