Feedback, features propositions, more questions (time, duration, seeking, audio samples, etc)

[Catsvilles]

Hi, I know, I know, last time I was after node.js version but it was before I realized that actually WebAssembly does everything I would ever need, and we can render and play audio server-side in node.js+sox same as in browser, WASM should be truly pronounced as AWESOME! :) Last few days I spent creating and live coding music in the browser and I got few ideas and feature requests, help offers:

1. Tracking current time while playing the song, calculating full song duration, seeking:
   Now, for the first one I know there are already few attempts to track time with logCurrentSongTime() in pattern_tools.js, but unfortunately I could not hack it to work good. I found another way that works good - tracking currentTime of global.audioworkletnode.context. Now, in my experience developers use setInterval() function a lot, when making music players with JS, etc but I'm sure there are better ways to dynamically update and log current time while playing. I would dig more into logCurrentSongTime() function and check how Tone.js does things, they have Transport class with .scheduleRepeat ( ) function, from what I know they use time precise clock of WebAudio, not JS one. More about this: here
   In any case, one way or another getting current time while playing should be fairly easy to implement, but getting full duration and seeking, this is where I'm limited in ideas, I just believe that it would be a cool feature and good user experience to get full song duration after we evaluate the code, then we could have a simple horizontal slider which scrolls while song is playing, and we can update the current position. This would be a cool feature, so we don't have to wait one part playing through when we actually wanna heart next patters. What do you think? With your blessings and guidance I'm ready to get on this one and submit PR as soon as I hack it together. So, you could just give me your thoughts and theory behind it, I will do the coding if something. :)

2. Audio Samples. Okay, this one is straight feature request, I remember you mentioned something about this in our previous issue discussion, I'm sure you are already planning those in any case, just wanna mention that it would be cool to have this freedom of adding custom audio samples, for cool kicks, drums, atmos, percussion, etc. And also would be cool if this would work same as whole AssemblyScript synth, both in browser and node.js+sox for quick audio rendering, so I guess we would have to implement custom AudioBuffer/AudioBuffersource in typescript, without using browser's AudioContext? Once we have AudioBuffersource it should be fairly easy to implement something like Sampler , allowing the user to play multiple samples with predefined pitches. I'm ready to help with this one too, let me know what you think!

3. Now, a bit of a feedback of a user who tried to create some music in the browser... Mixes... oh, I fairly know JS, never touched TypeScript before, and overall doing DSP stuff, making instruments so you can later code patterns with them, oh, this is quite exhausting for unprepared user :D I guess the current system holds quite high entry threshold for newcomers. Maybe it would be possible to implement some kind of presets of ready instruments/effects like other live coding environments do (read Sonic Pi)? I know, I'm probably over my head here, sorry, just sharing my thoughts. :)

4. I decided to forget about other ways of live coding music, like on server with node.js, and stick with AssemblyScript implementation you proposed but still I'm thinking that it would be cool to have some freedom of using other synthesizers with current sequencer and interface. For example, there is WASM version of Yoshimi I believe we could use it too, for composing in browser and rendering the audio on server with sox. Just, from what I understand the current sequencer is coming after your 4klang experiences, would it works with other synths, I mean, would it be possible overall? What do you think?

Thanks for reading, sorry, for that much of text, I hope I'm not being too much annoying here! One more thing, you should finally name your project, so people can be like:

insertnamehere is Awesome!!! Please continue developing and improving it!

[petersalomonsen]

Thanks for nice feedback :-)

1. The best source of time is here: https://github.com/petersalomonsen/javascriptmusic/blob/master/wasmaudioworklet/synth1/assembly/index.ts#L46
   The "tick" is used by the player to look up which pattern to play and the position in the pattern.

The sequencer timing happens inside the WebAssembly synth based on actual samples rendered,
unlike my experiments with nodejs, where I use setTimeout for scheduling when to play notes and
send midi signals to the midi synth.

If you want to have a go at this, you'd have to expose the tick through the AudioWorkletProcessor.
Send a signal through the message port to the AudioWorkletNode and then it can be used from a frontend JS api.

2. Audio samples is on my todo list. I'd like to record vocals and acoustic instruments. My plan / idea
   is to have uncompressed sample data in the WebAssembly memory. And so it will be easy for a "Sampler"
   instrument to play back sample data and it could be used in combination with other effects in the synth
   such as reverb, and even combine with synthesized instruments.

So this would have to be done in AssemblyScript, and it would have to be interfaces for filling memory with sample
data. Probably also some functions in JavaScript to decode/decompress audio formats when importing samples,
and also functions to store sample data when not in use ( e.g. to indexedDB ). I'm also thinking of this
in combination with my plan to implement a GIT client using the wasm-git project,
where there's also a virtual filesystem backed by indexedDB.

3. Yeah I want to simplify the instrument creation part. Possibly by making a simple macros that
   translates easily to assemblyscript sources, close to what 4klang does. Here's a script for a
   4klang instrument: https://github.com/petersalomonsen/javascriptmusic/blob/master/4klang/instruments/BA_SawBass.inc

And you can see more examples of how to generate sounds here:

https://www.youtube.com/watch?v=wP__g_9FT4M
https://www.youtube.com/watch?v=1nEcbAgRPtc&t=29s (Amiga klang)

Also yes I want more presets. I have some instruments already in the sources, but I think it would be easier
with 4klang style macros, and it would be even possible to import instruments from 4klang.

4. Thanks for the link to the wasm port of Yoshimi. I did use Yoshimi (ZynAddSubFX) for my nodejs
   experiments. I think actually the 4klang sequencer is not fit for use with Yoshimi, it's better
   with a midi sequencer, so I'd like to port my nodejs midi sequencer stuff to the web. Will definately have a go
   at this!

Naming... yes ... not sure what to call it yet.. Maybe something in the direction of w-awesome, like pronouncing WASM :)

[Catsvilles]

@petersalomonsen thank you for the reply! :)

Yeah I want to simplify the instrument creation part. Possibly by making a simple macros that
translates easily to assemblyscript source

My 5 cents here is since the project is targeting JS devs, I guess it make sense to make it in json format, so users can easily, write, save .json files/patches, should be a good UX :)

At the moment I cannot decide what I would like more to start working on, 1. time, duration, scroll or 4. Adding and making music with Yoshimi, asap :) Will probably work on this and that, and we'll see where it will take me.

so I'd like to port my nodejs midi sequencer stuff to the web. Will definately have a go at this!

Do you have any thoughts on the timeline, when you would be working on this? Also, I guess makes sense to use something like https://github.com/jazz-soft/JZZ as they claim to support both node.js and browser?

[petersalomonsen]

You just put Yoshimi on top of my list :) I had to test it yesterday. Managed to build it, and will start looking into integrating it. Those sounds are just amazing!

[Catsvilles]

@petersalomonsen Hah! I'm in the same mood, playing with it right now, these pads sound amazing, perfect for my darkish ambient things, don't understand why it's not much more popular! It's not a problem to play it in browser with JZZ midi library, but I cannot make it work and output audio in node.js, following your example. I understand now, that I could just use the provided glue code and abstract away AudioWorklet somehow, to pipe it to SOX, but meeh, cannot wrap my head around it, never worked with .wasm in node.js before, need your help with this! Really hope you will show some examples on how to pipe it to SOX! :)

[Catsvilles]

Interestingly, I also was able to play the AudioWorklet browser version directly from node.js env by sending midi directly to browser. JZZ library does provide what it promises, it works same both in browser and node.js! Just, from what I noticed overall these days, there is still big performance difference, when I play audio from node.js piped to SOX, and AudioWorklet, SOX loads my CPU at 20%-25% rate while AudioWorklet version is at 45%-50%.

[petersalomonsen]

I'm looking into the pure browser based approach. Managed to control Yoshimi from javascript here (click the play song button )

https://petersalomonsen.github.io/yoshimi/wam/dist/

this is a song I wrote in the nodejs setup, now translated to web. so I think this will work fine with the live coding environment. I think for the web it's probably better to export wav directly from the web page instead of using sox.

[petersalomonsen]

yoshimi support is here: #8

[Catsvilles]

Cool!! Was playing around it, sounds very good, but it's kinda hard to grasp how the sequencer works, for example, in provided song example, I could not find a way to hold the pad notes longer than it is now, how could I hold the note for 5 or 15, etc, seconds? I tried to adjust steps per beat but if the number is bigger the notes play actually even faster, I even set the number to negative -4 and only then I got my long, ambient pads, too bad they seem not to release and play infinitely :D

• Would it be possible in the future to use WAM synths and AssemblyScript synthesizer together in the same project? This would give possibility to have and choose from variety of instruments + external audio samples (once AssemblyScript synth will have those)? And also, this way will be easier to have same interface for tracking the currentTime, seeking, etc. I understand that AssemblyScript currently have totally different sequencer but making all the synths with one common sequencer gives much more freedom and options?

• Does song compile actually works in Yoshimi mode? I just get the message in console but the .wasm file is not downloading. Well, in any case I suppose this comes later.

[petersalomonsen]

regarding note lengths consider this example:

// SONGMODE=YOSHIMI
setBPM(80);

const lead = createTrack(
                  5, // midi channel
                  2 // default duration in steps per beat ( lower is longer ) 
            );

await lead.steps(
  4, // track resolution in steps per beat
  [
    d5(1/8),,,, // first parameter of a note is the note duration
    f5(1/4),,,,
    a5(1),,,,
    f5,,,,
]);
loopHere();

• Yes, I plan to be able to combine the assemblyscript synth and wasm synth, but the way I see it currently is that they will be separate AudioWorklets mixed using the Web Audio Context.
• There's no downloading of a WASM in Yosimi mode, and I'm not sure if it will be since it also would be a quite big WASM. Also it's based on emscripten which does not run in the browser. Instead I'm planning an export-wav function using OfflineAudioContext. Export WASM is more a feature of the AssemblyScript synth, and there it's also easier to make compact and small binaries.

[petersalomonsen]

demo video: https://youtu.be/HH92wXnP4WU

[Catsvilles]

@petersalomonsen Hey, it's working very good, it's possible already to compose whole songs! Thanks for your previous reply and explaining things! I was following the whole progress, looks like what's left now is to implement .wav rendering and adding audio samples, or merging Yoshimi with AssemblyScript version for those.

Meanwhile, I was working on Time, Duration, Seeking and have a bit of success with the first two in AssemblyScript mode. but I guess I'll wait when you will finalize the whole version so I can proceed with those. I noticed that logCurrentSongTime() actually calculates the whole duration of the current song, so I guess something like this is needed for OfflineAudioContext in Yoshimi version. Still have no idea how to proceed with Seeking but I'm really excited to get on it, it's my first experience in participating/building such musical software so I will be happy to get it done.

• A little bit hard to understand and grasp on what we can do in .xml file, I tried to read more about it in Yoshimi documentation but still not fully clear if we can edit/control whole Yoshimi synth or just separate instruments/presets? I guess you don't have to spend your time now and answer this here, I just thought it would be cool to mention this or/and link to sources where users can learn more about it, in the documentation/read.md files in the future.

[petersalomonsen]

The xml file is for controlling the whole Yoshimi synth. If you load multiple instruments, you'll see that they all appear in the xml file.

For OfflineAudioContext in the Yoshimi context there's no worry about the timing, as the sequencer player is embedded into the audioworklet. So it's just to start it and it will trigger the midi events from within the audioworklet processor.

[petersalomonsen]

a little demo:

https://petersalomonsen.github.io/javascriptmusic/wasmaudioworklet/?gist=29d0b6f8e8d3cf3267ae4b7b4ffa49bf

[petersalomonsen]

export audio to wav, work in progress:

#15

already made an export that is sent to spotify:

https://twitter.com/salomonsen_p/status/1261404943484772353?s=20

[petersalomonsen]

Export wav implemented now.