Improve Performance

[spessasus]

somehow improve performance of the synthesizer

Either:
Port the worklet system to wasm/emscripten to WASM (c++ or maybe rust) to improve performance, then figure out how to keep all the available communication with the JS code

or improve the current JavaScript implementation.

When the website is loaded and before a MIDI file is selected or something is changed, the RAM usage, on my PC, is already at 470 MB (but likely Firefox itself uses two third here independent of the website). Replacing algorithms here can lower the amount of RAM quite a bit. I assume, the soundfont loader is not optimized, maybe loads the complete soundfont into RAM even though only small parts are required for a given MIDI file later.

Replacing js core code by wasm will most likely reduce the amount of RAM by half.
But it is not clear whether the runtime will be faster. Benchmarks indicate 2 times faster to 2 times slower, depending on the task.

My guess is, that changes to the pure js code can already achieve the wanted performance improvements, but first, it has to be clear what are the morst resource-hungry task w.r.t. RAM usage and CPU runtime. My guess without looking into the code: soundfont loader and soundfont renderer, gui renderer (especially the notes and effects at high FPS), calculating and rendering the wave effects.

Nonetheless, so far I have not seen the performance to be problematic on my laptop. It could play all MIDIs I have tested and RAM usage was also still OK.

[spessasus]

Unfortunately RAM is a big problem, because of a few reasons:

1. The entire soundfont HAS to be stored in the RAM. Unlike native synths (like fluidsynth) spessasynth, being in the browser, cannot operate on the file on the disk and load specific elements, but has to fetch the file as a whole in order to read it.
2. Forced 32-bit float rendering: WebAudio uses Float32 as the bit depth of choice. It's a great and incredibly high quality choice, but given that soundfonts use 16-bit audio, spessasynth has to resample it to 32-bit and keep it in its cache. Note that during the MIDI loading only samples that the said file uses are actually cached, so at least there's that
3. Transferring the data between contexts: Since AudioWorklets operate in a different thread than the main thread, the entire soundfont binary has to be copied there. This is fine for smaller soundfonts, but if we're talking about 4GB, it can freeze my system and take a good two minutes to copy it. Note that this only works on Firefox because Chrome has a hard-coded 4GB memory limit for a tab, so maybe its Firefox's fault for taking so long to copy it. The parser itself seems to be quite fast, as after the said 4GB soundfont has been succesfully copied to the audio thread, the parser has it ready in about a second or so.

1. is unfortunate, there are ways to still solve this by splitting the soundfont into different chunks and each has an own webadress. Load only those which are needed. But for small soundfonts this will be overkill and for others it will add much complexity.
2. This sounds like difficult to improve.
3. This sounds like it is possible to pass through the threads by allocating it to some static adress and only pass the adress. But maybe there is a good reason that this does not work.

Maybe it is possible to supply a "bad quality" default soundfont which uses high compression but has a lower filesize.

If it's mainly about RAM, then WASM can indeed give a factor 2 improvement according to the linked benchmarks. Have you tried WASM and were there any problems with it? I could port the code to Rust if the c++ code has issues.

[spessasus]

Have you tried WASM and were there any problems with it?

I did try it once. There's the web-assembly-system brach if you want to see that, though it was created before v3. This was before the entire synth was in the worklet thread and the main thread contained the soundfont parser. The worklet received ready-to-use data (playback rate, tuning, volume, etc.) along with the samples through the message port. Now that the worklet contains the entire synth + sequencer, it would be a complete rewrite.

I could port the code to Rust if the c++ code has issues.

Since I'd have to start over, if I will ever get to porting it to wasm, I think using memory safe rust from the start would be better, unless c++ is way faster.

I noticed the following website performance issue that comes from algorithmic problems: When I play a piano midi with 4 voices and no other effects beside piano keys, the notes are not falling, just the keys are pressed, then on my notebook the cpu usage is 35 %.
When I go to "settings" -> and turn off "show", the midi continues to play, but the keyboard is now hidden. The cpu usage falls to 25 % now.

This shows that a simple piano rendering (no falling notes) uses 10 % additional cpu.

This seems like an algorithmic problem, since only up to 4 keys are updated per 0.5 seconds. Such an update could be done with less than 1 % cpu per second, I guess. It seems, the keyboard is redrawn too often. Maybe related to this is the FPS count in the upper right corner. For me it says ~60 fps. At best, for this scenario we would have less than 10 fps and get the same rendering. This would lower the cpu usage for this task by factor 6.

I think, such improvements can already achieve the wanted performance without wasm code.

Update:

It seems toogling the "midi keyboard" -> "show" button on and off makes at least 5 % cpu difference even when no notes are falling and when I add an immediate return; in the pressNote function (with the result that no notes are rendered at all).
Since the toogle has no visual effect under this scenario, it becomes clear that the additional 5 % cpu comes not from rendering the notes. Some background task is turned on after toogling the button that uses high cpu but has no visual effect.

[spessasus]

That's strange. The keyboard is not rendered at all in js, it's just a html element with keys being div elements with styling.

Edit: though this issue is related to the underlying audio engine, not the frontend. I want it to be able to match fluidsynth at least and be able to play back a MIDI file on a phone with the screen off. Currently, locking the screen on my phone causes massive stutters on both chrome and Firefox.

[spessasus]

I've made some perfomance improvements (hopefully) with filter coefficients getting cached and all trigonometric functions (sin/cos) getting cached, such as the pan table.

However, it seems that fixing spessasus/SpessaSynth#124 will require filter smoothing, which will be problematic for perfomance as a lot more coefficient recalculations will be required...

Here is one idea to better find out out, how much each component influences the total cpu und memory usage:

• add support for completely turning off audio rendering in the website (all other should stay the same like piano rendering etc). When turning it back on, there should be a clear difference in cpu usage.
• using this, we can find out if the difference is 2 % or 15 % or so. If it is only 2 %, we know that wasm will not help.
• additionally, it could be a good idea to look at caching parts of the audio rendering or the complete audio rendering: how much difference in cpu/memory does it make to load an imported WAV file instead of realtime rendering?

[spessasus]

Hi TS,
Thank you for your feedback and suggestions again. You're very helpful for this project :-)

Regarding turning off audio, I can't really do that, as literally every component of the UI relies on the synthesizer existing. But when it doesn't have voices to render, it shouldn't use any CPU. That being said, I forgot the if statement that checks for 0 voices and quits immediately. So it goes through every channel, doing nothing. Still, that shouldn't use CPU, should it?

About disabling other parts: if you comment these lines out:
https://github.com/spessasus/SpessaSynth/blob/ec96049a412d57083e5a2632a71f497519393b07/src/website/js/manager/manager.js#L211-L452
it should disable all the UI, so only the audio engine would be running.

About rendering a WAV file instead of real-time: unfortunately this is not an option. If I wanted to change, like an instrument or a controller, the entire thing would have to be rendered again, which is incredibly expensive with large soundfonts (offline audioContext has to have the entire binary copied to its own thread, so up to 4GB of memory getting copied every time you make a change.

PS: What CPU are you testing this on?