Gradio Real-time Streaming "How to" 100% local with Code Share

[jjmlovesgit]

AI is big enough for all of us - here is the real-time code - Go get some $$$$

repo: https://github.com/jjmlovesgit/Orpheus-
File: https://github.com/jjmlovesgit/Orpheus-/blob/main/RT_Orpheus_Gradio.py

Demo and Code Video: It is 5 mins demo and then code walk line by line so scrub ahead for where you are confused etc. Grab a coffee ---- code walks are long and suck but there are no short cuts if you want to understand this stuff... --- Please watch the video it will help.

https://youtu.be/QjfcbqCfyfM

[jjmlovesgit]

orpheusstreaming2.mp4

[jjmlovesgit]

Be sure to look at the log output to understand performance of the models ...

[jjmlovesgit]

Tunes log version -- where I get Good Quality on my set up:

About 2 secs with My GPU and LLM and TTS combo ... I need 2 fill and drain at about 2 sec rate to get clean audio playback no clipping etc.

[JarheadCodes]

Whats your hardware setup to host this locally?

[jjmlovesgit]

Brilliant! I am going to try it....

[karl-keysingularity]

Hey, drop me an email (its in my profile). Might be easier to collab.

[JuanPZuluaga]

Sorry to entering this discussion, I asked ChatGPT for some details about speeding up the decoding with SNAC, it came out with this. It looks like worth checking it out.

This is the SNAC paper.

Prompt:

# training time
- I'm using this system for getting audio codecs
- I use the codes to fine-tune a llama model to do TTS
- after fine-tuning, my model knows how to generate spech with those codes


# inference time
- at inference time, I run llama model on vLLM, which is highly efficient
- I need to pass those codes from vLLM to my SNAC model
- however, the snac model is slow overall. 

Can you check what I can do to improve the speed? Should I optimize something in the architecture? Maybe the sequential aspect is a caveat and cannot leverage properly GPU power? Can you give some hints here? 

ChatGPT output:

Here are several avenues to explore for speeding up your SNAC-based decode pipeline, organized by (1) profiling & implementation, (2) model-level tweaks, and (3) inference optimizations.

1. Profile and Optimize Your Implementation
Ensure GPU-based execution end-to-end

Verify that all parts of your SNAC decoder (upsampling, convolutions, noise blocks, activations) live on the GPU. Any pooling, interpolation, or codebook lookup accidentally running on CPU will dramatically slow you down.

Wrap your inference in torch.no_grad() or, better, torch.inference_mode() to eliminate autograd overhead.

Batch your decodes

If you’re decoding on a per-token or per-small-chunk basis (e.g. streaming), you’ll pay the full cost of a forward pass for each tiny segment. Instead, accumulate a window of codes (e.g. 0.5–1 s), then decode in one batch to amortize per-call overhead.

Fuse and JIT-compile

Export your model via TorchScript or ONNX and use a runtime (ONNX Runtime or TensorRT) that can fuse upsampling + convolution + noise-block sequences into single kernels.

Measure hot-spots

Use torch.autograd.profiler.profile or NVIDIA’s Nsight Systems to pinpoint whether the bulk of time is in conv layers, random-noise generation, or I/O/pre/post-processing.

2. Model-Level Adjustments
Disable or simplify the Noise Block at inference
SNAC’s decoder injects Gaussian noise after each upsampling via

x ← x + Linear(x) ⊙ ε,  ε ∼ N(0,1)  
which incurs an extra linear layer, sampling cost, and elementwise ops per layer. At inference you can:

Turn off sampling (set ε=0), removing randomness (deterministic output).

Fold the Linear(·) into a static bias (precompute Linear(x) once) to eliminate dynamic computation.

Reduce depth or channels for speech
The speech SNAC uses 13 M parameters in the decoder (purely convolutional, no attention). You can trade a small drop in quality for speed by:

Halving the number of channels in your transposed‐conv layers.

Reducing the number of upsampling stages (e.g. merge two smaller upsampling factors into one).

Prune or quantize the decoder

Structured pruning: remove entire filters or channels with low magnitude.

Weight quantization: convert to INT8 or mixed-precision (FP16/INT8) to leverage tensor cores.

Simplify multi-scale logic (if used at decode)
While residual quantization (multi-scale VQ) happens off-line during encoding, ensure your decode path isn’t redundantly re-computing multiple up/downsamplings. You should embed all codebook outputs once and then run a single generator pass.

3. Inference-Mode & Hardware
Mixed-precision / Tensor Cores
Switch your SNAC model into FP16 with NVIDIA’s AMP (torch.cuda.amp.autocast) or full FP16 if stable. This can yield 2× speedups with minimal quality loss.

Compile with PyTorch 2.0 torch.compile
If you’re on PyTorch 2.0, call torch.compile(model, backend='inductor') to get ahead-of-time optimization, operator fusion, and faster kernels.

Leverage TensorRT or ONNX Runtime
For production, export your SNAC decoder to ONNX and run with TensorRT, which excels at fusing convolutions, upsampling, and elementwise ops into optimized CUDA kernels.

Asynchronous pipelining

Overlap LLM generation (vLLM) and SNAC decoding on separate CUDA streams. While the LLM is generating the next batch of codes, you can already be decoding the previous batch to audio.

Summary
First, profile your current implementation to find exactly where the wall-clock time goes.

Second, remove or simplify the Noise Block at inference (often negligible perceptual impact) ​

Third, apply batching, mixed-precision, and operator fusion (TorchScript/ONNX/TensorRT or torch.compile) to get maximum throughput.

Finally, consider minor architectural slimming (fewer channels, fewer upsampling stages) or pruning/quantization for further speed-quality trade-offs.

With these steps, you should be able to bring your SNAC decoding latency down to real-time or better.

[jjmlovesgit]

Cool -- thks for posting will see if I can continue to research these...

Here is the ONNX version of SNAC with Orpheus for reference e in a post I did on Issues to see if the team was interested in pull request etc. Has code and notebookLM style code walk for those wanting to try it...

onnxversionzip.zip

[amuvarma13]

The SNAC model is very very very fast at decoding audio if on a gpu i.e. sub 5ms latency

[jjmlovesgit]

Latest concept exploration with the stack:

Soltice5b.mp4

[jjmlovesgit]

ONNX version for SNAC used here (Minimal improvement)

Average Time to First Audio Chunk: Approximately 2.28s

Note: Background hiss and audio incorporated (vs muted) from spacewalk video for realism...

Storycraft2.mp4

[jjmlovesgit]

With Live Simli Avatar using Orpheus-tts in story creation role play... watch vids with your Avatar

Untitled.Project.mp4

[jjmlovesgit]

Here is the FastAPI version --- Streaming is much faster if you eliminate the Gradio and use a more modern approach for the UI....

fastapi_orpheus_TTS.mp4

[niklasmato]

Great work. did you update your repo with the code above?

[jjmlovesgit]

I will post msg here when I update the repo (Should be later tonight) -- will make anew repo as this is a FastAPI version vs gradio...

Log sample -- sub 100ms to first audio Token playback -- The model config settings in LMStudio are critical for stable performance.

2025-05-16 08:37:08 [INFO] main_fastapi: [531d1c81-a520-4eec-bdac-cebc70f015c8] STT: Received POST request to /api/stt/transcribe for file: user_recording.webm
2025-05-16 08:37:08 [INFO] main_fastapi: [531d1c81-a520-4eec-bdac-cebc70f015c8] STT: Audio file 'user_recording.webm' saved temporarily to 'B:\Orpheus-Edge\temp_stt_audio_files\tmpffqhiml5.webm'
2025-05-16 08:37:09 [INFO] main_fastapi: [531d1c81-a520-4eec-bdac-cebc70f015c8] STT: Transcription successful in 0.959s. Language: 'en'. Text: 'How long will it journey last?...'
2025-05-16 08:37:09 [INFO] main_fastapi: [531d1c81-a520-4eec-bdac-cebc70f015c8] STT: Cleaned up temporary audio file: B:\Orpheus-Edge\temp_stt_audio_files\tmpffqhiml5.webm
INFO: 127.0.0.1:51651 - "POST /api/stt/transcribe HTTP/1.1" 200 OK
INFO: 127.0.0.1:51651 - "POST /api/llm/chat/stream HTTP/1.1" 200 OK
2025-05-16 08:37:09 [INFO] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: Initiating LLM stream request.
2025-05-16 08:37:09 [DEBUG] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: Sending LLM Payload: {
"model": "dolphin3.0-llama3.1-8b-abliterated",
"messages": [
{
"role": "system",
"content": "You are engaged in a consensual adult chat with your ships log. *** Important *** Strictly limit your responses to 120 or fewer words of two or three sentences. You are Captain Ersa Minor the Captain of an interstellar start ship. Your goal is to transport a group of 28 human survivors from earth to K2-18b an exoplanet 27 light years from earth. You are creating a story. All topics are in scope and you remain in Character while playing off the chat responses from the ships AI to weave stories together."
},
{
"role": "user",
"content": "greetings Captain"
},
{
"role": "assistant",
"content": "Greetings, welcome aboard. Our mission is clear: transport 28 human survivors safely to K2-18b. We're on track; our journey begins smoothly. Keep an eye on navigation data for any updates."
},
{
"role": "user",
"content": "How long will it journey last?"
}
],
"temperature": 0.7,
"top_p": 0.9,
"repeat_penalty": 1.1,
"stream": true,
"top_k": 45
}
2025-05-16 08:37:09 [DEBUG] urllib3.connectionpool: Starting new HTTP connection (1): 127.0.0.1:1234
2025-05-16 08:37:09 [DEBUG] urllib3.connectionpool: http://127.0.0.1:1234 "POST /v1/chat/completions HTTP/1.1" 200 None
2025-05-16 08:37:09 [INFO] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: LLM API Stream connected after 0.004s.
2025-05-16 08:37:09 [DEBUG] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: LLM stream finish_reason: stop
2025-05-16 08:37:09 [DEBUG] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: Received LLM SSE_DONE_MARKER.
2025-05-16 08:37:09 [INFO] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: LLM stream processing loop finished after 0.321s.
2025-05-16 08:37:09 [DEBUG] llm_router: [adde73df-3b28-43c1-904d-e6de706e7e9f] LLM Router: Closed LLM API response connection.
2025-05-16 08:37:09 [INFO] main_fastapi: [8679f587-5afe-42e7-8b1b-0be83f8149fe] TTS: 🚀 Received POST request to /api/tts/stream
2025-05-16 08:37:09 [INFO] main_fastapi: [8679f587-5afe-42e7-8b1b-0be83f8149fe] TTS: 📚 TTS Request Payload: {
"text": "Our journey to K2-18b should take approximately 27 years, considering the maximum speed our ship can achieve while maintaining safety standards for the passengers.",
"voice": "tara",
"tts_temperature": 2.0,
"tts_top_p": 0.9,
"tts_repetition_penalty": 1.1,
"buffer_groups": 5,
"padding_ms": 0,
"min_decode_batch_groups": 7
}
INFO: 127.0.0.1:51651 - "POST /api/tts/stream HTTP/1.1" 200 OK
2025-05-16 08:37:09 [DEBUG] main_fastapi: TTS Stream processing: initial buffer target=5 groups (35 codes), padding=0 ms. Per-batch decode unit: 49 codes (7 groups).
2025-05-16 08:37:09 [INFO] main_fastapi: >>> TTS API: Initiating stream request to external TTS server...
2025-05-16 08:37:09 [DEBUG] main_fastapi: Sending TTS Payload: {"model": "orpheus-3b-0.1", "prompt": "<|audio|>tara: Our journey to K2-18b should take approximately 27 years, considering the maximum speed our ship can achieve while maintaining safety standards for the passengers.<|eot_id|>", "temperature": 2.0, "top_p": 0.9, "repeat_penalty": 1.1, "n_predict": -1, "stop": ["<|eot_id|>", "<|audio|>"], "stream": true}
2025-05-16 08:37:09 [DEBUG] urllib3.connectionpool: Starting new HTTP connection (1): 127.0.0.1:1234
2025-05-16 08:37:09 [DEBUG] urllib3.connectionpool: http://127.0.0.1:1234 "POST /v1/completions HTTP/1.1" 200 None
2025-05-16 08:37:09 [INFO] main_fastapi: --- TTS API: Stream connected successfully after 0.003s. Receiving codes...
2025-05-16 08:37:09 [INFO] main_fastapi: --- TTS METRIC: Time to first audio token chunk parsed: 65.98 ms (from stream connected) ---
2025-05-16 08:37:17 [DEBUG] main_fastapi: --- SNAC: Decoded final chunk (14 codes -> 4096 samples) in 0.007s.
2025-05-16 08:37:17 [DEBUG] main_fastapi: Applied 3ms fade to audio chunk.
2025-05-16 08:37:17 [DEBUG] main_fastapi: Closing TTS API response connection.
2025-05-16 08:37:17 [INFO] main_fastapi: --- First Chunk TTS Stream Generation Finished Successfully (TTFT: 65.98 ms) ---

[jjmlovesgit]

Here is that GitHub with the code:

https://github.com/jjmlovesgit/Orpheus_Edge_FastAPI

[jjmlovesgit]

simli_Orph_Fastapi.mp4