I want all new APIs to be supported in Web Workers (within reason)

[aarongustafson]

For performance reasons, it's good to have as much code running in Web Workers instead of the main thread. There are APIs like OffscreenCanvas that make it possible to do things like advanced WebGL rendering off the main thread. However many APIs are still missing in Workers, such as input events, Web Audio, video, media streams, WebRTC and so on. Each missing API is another hurdle to moving logic to a Web Worker.

More worryingly, new APIs are still being added that are not available in Web Workers. This means there are more hurdles still being added all the time, so the problem is largely getting worse, not better. Some examples of this are: async clipboard; Web Share; Web Bluetooth; Media Session; Native File System; getDisplayMedia; visual viewport; and so on. These are all added on top of the existing notable omissions (e.g. media streams, Web Audio etc).

To try to make sure the problem improves instead of gets worse, I'd like to see a commitment to support all new APIs in Web Workers wherever possible.

---

https://webwewant.fyi/wants/76/

[bradisbell]

+1, for sure! The one that bites me the most often is an OfflineAudioContext. If I want to do audio analysis on a file, it would be great to be able to decode the file in the worker. Currently, I have to decode it on the main thread, then pass the whole buffer over to the worker, which kills most of the benefit.

[guest271314]

The video frames version is for Firefox, which has OffscreenCanvas issues, at Chrome and Chromium we can use a module Worker and transferable streams to perform the initial and entirery of the media processing in the Worker https://plnkr.co/edit/gCjYSt.

[bradisbell]

@guest271314 Your suggestions don't apply in my case. I need to be able to decode using all the codecs available on the system, beyond just PCM in WAV. Implementing these codecs in JavaScript is slow, and in some cases there are legal consequences. Therefore, I need to rely on the browser to provide the access.

[guest271314]

The title of this issue is APIs to be supported in Web Workers, not decoding using all the codecs available on the system. Which specific codecs are you referring to? Opus https://github.com/AnthumChris/opus-stream-decoder? AAC? H264? AV1? OP does not mention decoding at all.

[guest271314]

@bradisbell The question I have is what exactly you are trying to do. You can decode any codec in a Worker yourself, on your own machine. FileSystemHandle, ReadableStream, WritableStream are transferable on Chrome and Chromium. Web Audio API deals with PCM, thus I decoded to PCM to be able to use AudioWorklet to stream the output. The input could be any code that ffmpeg or any other native application that is doing the decoding supports, for example, we capture raw PCM then encode to Opus in WebM container just to use MediaSource at the browser (note, MediaSource is now supported in Worker w3c/media-source#192).

[guest271314]

This is what I am referring to re PCM => opusenc => ffmpeg => MediaSource https://github.com/guest271314/captureSystemAudio#stream-file-being-written-at-local-filesystem-to-mediasource-capture-as-mediastream-record-with-mediarecorder-in-real-time. You can configure that approach however you want.

Also, there is a distinction between whether we can make stuff work in the browser on our own and creating specifying and implementing an API.

I have not observed any evidence that supporting fetch() in AudioWorkletGlobalScope will impact performance - that is, the ability to fetch and post data in AudioWorkletGlobalScope, however, the maintainers of that API still do not even want to test that proposition, WebAudio/web-audio-api-v2#73, https://bugs.chromium.org/p/chromium/issues/detail?id=114424. So, we have to test other novel or workaround approaches. WebTransport provides a means to do whatever you want locally and get the result in the browser. For example, Web Speech API does not provide any means to capture the output of synthesis, so we get the output of espeak-ng directly https://github.com/guest271314/webtransport/blob/main/webTransportEspeakNg.js. We can also stream in "real-time" from the local native application or shell script, performing whatever encoding or decoding we want, growing one or more WebAssembly.Memory objects to store the PCM data https://github.com/guest271314/webtransport/blob/main/webTransportAudioWorkletWebAssemblyMemoryGrow.js to accomodate only the memory we need for that particular stream. We can use that code instead of the dedicated JavaScript that get the bufer in one read, so that we read directly from STDOUT.

[guest271314]

@bradisbell See

• https://github.com/webrtc/samples/blob/gh-pages/src/content/insertable-streams/endtoend-encryption/js/worker.js
• https://github.com/yuki-uchida/webcodecs_webtransport