ValueError: attempt to get argmin of an empty sequence in Diarization!

[behroozazarkhalili]

@kadirnar
I have received the following error when running the diarization pipeline:

ValueError: attempt to get argmin of an empty sequence

I ran the diarization example exactly. The error seems related to the code below in the diarization pipeline.

 # align the diarizer timestamps and the ASR timestamps
for segment in new_segments:
    # get the diarizer end timestamp
    end_time = segment["segment"]["end"]
    # find the ASR end timestamp that is closest to the diarizer's end timestamp and cut the transcript to here
    upto_idx = np.argmin(np.abs(end_timestamps - end_time))

The pipeline is also too slow, taking 10 minutes for a 5-minute audio file. Can you provide a detailed example of setting hqq for diarization?

[kadirnar]

What code are you using? Which GPU?

[behroozazarkhalili]

What code are you using? Which GPU?

Hi,

from whisperplus.pipelines.whisper_diarize import ASRDiarizationPipeline
from whisperplus import download_youtube_to_mp3, format_speech_to_dialogue

audio_path = download_youtube_to_mp3("https://www.youtube.com/watch?v=6sUwRiIncKU")

device = "cuda"  # cpu or mps
pipeline = ASRDiarizationPipeline.from_pretrained(
    asr_model="openai/whisper-large-v3",
    diarizer_model="pyannote/speaker-diarization-3.1",
    use_auth_token=False,
    chunk_length_s=30,
    device=device,
)

output_text = pipeline(audio_path, num_speakers=2, min_speaker=1, max_speaker=2)
dialogue = format_speech_to_dialogue(output_text)
print(dialogue)

RTX 3090; however, it is not related to the GPU. It seems that the returned result is empty in some places.

[behroozazarkhalili]

@kadirnar, any update on this?

[kadirnar]

I will test it today.

[KolosDan]

@kadirnar any news? got the same problem on rtx4090 in runpod

[behroozazarkhalili]

@kadirnar, Could you please kindly update us regarding this issue as it makes it impossible to use this great package?

[kadirnar]

I am sorry for my late answer. I am so busy.

@KolosDan @behroozazarkhalili

Have you done all the installations correctly? Have you used hf_token?

[Terisback]

Have you done all the installations correctly?

If README installation section is correct, we good.

@kadirnar generally long inputs and long silence parts trigger this error, found example to trigger this error - https://youtu.be/SbZ5ONmdwgM?si=0QcdTMcamqyngqzU

For context: I download the input (an example video URL with -f 140), convert it to MP3 using ffmpeg, and then send it to the worker. Then it does processing for 20 min and fails.

Error content

{"error_type": "<class 'ValueError'>", "error_message": "attempt to get argmin of an empty sequence", "error_traceback": "Traceback (most recent call last):\
  File \"/usr/local/lib/python3.10/dist-packages/runpod/serverless/modules/rp_job.py\", line 134, in run_job\
    handler_return = handler(job)\
  File \"/handler.py\", line 22, in handler\
    diarization = pipeline(audio_path)\
  File \"/usr/local/lib/python3.10/dist-packages/whisperplus/pipelines/whisper_diarize.py\", line 171, in __call__\
    upto_idx = np.argmin(np.abs(end_timestamps - end_time))\
  File \"/usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py\", line 1325, in argmin\
    return _wrapfunc(a, 'argmin', axis=axis, out=out, **kwds)\
  File \"/usr/local/lib/python3.10/dist-packages/numpy/core/fromnumeric.py\", line 59, in _wrapfunc\
    return bound(*args, **kwds)\
ValueError: attempt to get argmin of an empty sequence\
", "hostname": "xxx", "worker_id": "xxx", "runpod_version": "1.6.2"}

Source code of runpod worker

import runpod
from runpod.serverless.utils import rp_download
from whisperplus.pipelines.whisper_diarize import ASRDiarizationPipeline
from whisperplus import download_youtube_to_mp3, format_speech_to_dialogue
import os


def handler(job):
    job_input = job["input"]
    file = rp_download.file(job_input["url"])

    audio_path = file['file_path']

    pipeline = ASRDiarizationPipeline.from_pretrained(
        asr_model="openai/whisper-large-v3",
        diarizer_model="pyannote/speaker-diarization-3.1",
        use_auth_token=os.getenv('HF_TOKEN'),
        chunk_length_s=30,
        device="cuda",
    )

    diarization = pipeline(audio_path)
    return diarization

if __name__ == "__main__":
    runpod.serverless.start({"handler": handler})

[behroozazarkhalili]

@kadirnar
Yes, It is not related to the installation, though. I highlighted the root cause of the error in the code I already submitted.

[kadirnar]

Thank you for the detailed explanation. I'll try again this evening. I install Runpod gpu for each trial. So it's a little late. Today I will solve this problem. Thank you for your interest.

[theandrepo]

Hi.
I just wanted to add that I also ended in the same issue.

In my case I have tested with both :
device = "cpu"
and
device = "mps"

The code I used is 100% the one of the example in the repo for Speaker Diarization.
All the installation, requirements, Models, HF token and pyannote permissions are fine.

I cloned the repo just minutes ago.

[kadirnar]

What code are you using? Which GPU?

Hi,

from whisperplus.pipelines.whisper_diarize import ASRDiarizationPipeline
from whisperplus import download_youtube_to_mp3, format_speech_to_dialogue

audio_path = download_youtube_to_mp3("https://www.youtube.com/watch?v=6sUwRiIncKU")

device = "cuda"  # cpu or mps
pipeline = ASRDiarizationPipeline.from_pretrained(
    asr_model="openai/whisper-large-v3",
    diarizer_model="pyannote/speaker-diarization-3.1",
    use_auth_token=False,
    chunk_length_s=30,
    device=device,
)

output_text = pipeline(audio_path, num_speakers=2, min_speaker=1, max_speaker=2)
dialogue = format_speech_to_dialogue(output_text)
print(dialogue)

RTX 3090; however, it is not related to the GPU. It seems that the returned result is empty in some places.

I tested it with A40 on the Runpod platform and it works.

[Terisback]

@kadirnar generally long inputs and long silence parts trigger this error, found example to trigger this error - https://youtu.be/SbZ5ONmdwgM?si=0QcdTMcamqyngqzU

could you test it with longer example?