Usig this ope-source model i productio?
Cosider switchig to pyaoteAI for better ad faster optios.

? "Powerset" speaker segmetatio

This model igests 10 secods of moo audio sampled at 16kHz ad outputs speaker diarizatio as a (umframes, umclasses) matrix where the 7 classes are o-speech, speaker #1, speaker #2, speaker #3, speakers #1 ad #2, speakers #1 ad #3, ad speakers #2 ad #3.

# waveform (first row)
duratio, sample_rate, um_chaels = 10, 16000, 1
waveform = torch.rad(batch_size, um_chaels, duratio * sample_rate) 

# powerset multi-class ecodig (secod row)
powerset_ecodig = model(waveform)

# multi-label ecodig (third row)
from pyaote.audio.utils.powerset import Powerset
max_speakers_per_chuk, max_speakers_per_frame = 3, 2
to_multilabel = Powerset(
    max_speakers_per_chuk, 
    max_speakers_per_frame).to_multilabel
multilabel_ecodig = to_multilabel(powerset_ecodig)

The various cocepts behid this model are described i details i this paper.

It has bee traied by Séveri Baroudi with pyaote.audio 3.0.0 usig the combiatio of the traiig sets of AISHELL, AliMeetig, AMI, AVA-AVD, DIHARD, Ego4D, MSDWild, REPERE, ad VoxCoverse.

This compaio repository by Alexis Plaquet also provides istructios o how to trai or fietue such a model o your ow data.

Requiremets

1. Istall pyaote.audio 3.0 with pip istall pyaote.audio
2. Accept pyaote/segmetatio-3.0 user coditios
3. Create access toke at hf.co/settigs/tokes.

Usage

# istatiate the model
from pyaote.audio import Model
model = Model.from_pretraied(
  "pyaote/segmetatio-3.0", 
  use_auth_toke="HUGGINGFACE_ACCESS_TOKEN_GOES_HERE")

Speaker diarizatio

This model caot be used to perform speaker diarizatio of full recordigs o its ow (it oly processes 10s chuks).

See pyaote/speaker-diarizatio-3.0 pipelie that uses a additioal speaker embeddig model to perform full recordig speaker diarizatio.

Voice activity detectio

from pyaote.audio.pipelies import VoiceActivityDetectio
pipelie = VoiceActivityDetectio(segmetatio=model)
HYPER_PARAMETERS = {
  # remove speech regios shorter tha that may secods.
  "mi_duratio_o": 0.0,
  # fill o-speech regios shorter tha that may secods.
  "mi_duratio_off": 0.0
}
pipelie.istatiate(HYPER_PARAMETERS)
vad = pipelie("audio.wav")
# `vad` is a pyaote.core.Aotatio istace cotaiig speech regios

Overlapped speech detectio

from pyaote.audio.pipelies import OverlappedSpeechDetectio
pipelie = OverlappedSpeechDetectio(segmetatio=model)
HYPER_PARAMETERS = {
  # remove overlapped speech regios shorter tha that may secods.
  "mi_duratio_o": 0.0,
  # fill o-overlapped speech regios shorter tha that may secods.
  "mi_duratio_off": 0.0
}
pipelie.istatiate(HYPER_PARAMETERS)
osd = pipelie("audio.wav")
# `osd` is a pyaote.core.Aotatio istace cotaiig overlapped speech regios

Citatios

@iproceedigs{Plaquet23,
  author={Alexis Plaquet ad Hervé Bredi},
  title={{Powerset multi-class cross etropy loss for eural speaker diarizatio}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
}

@iproceedigs{Bredi23,
  author={Hervé Bredi},
  title={{pyaote.audio 2.1 speaker diarizatio pipelie: priciple, bechmark, ad recipe}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
}

Using this open-source model in production? Consider switching to pyannoteAI for better and faster o