MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::Yantra::SileroVadModel Class Reference

Silero VAD at 16 kHz as a stream: samples in, one speech probability per chunk out. More...

#include <SileroVadModel.hpp>

Collaboration diagram for MayaFlux::Yantra::SileroVadModel:

Classes

struct  Impl
struct  ProbabilityBuffer
 Where the probabilities of the latest run are on the device. More...

Public Member Functions

std::vector< float > flush ()
 Run the partial chunk, padded with zeros to a whole one.
ProbabilityBuffer hidden_buffer ()
 Where the recurrent output of the latest run is on the device.
void keep_hidden (bool keep)
 Keep the recurrent output of every chunk process and flush run, for take_hidden.
SileroVadModel & operator= (const SileroVadModel &)=delete
SileroVadModel & operator= (SileroVadModel &&)=delete
size_t pending () const noexcept
 Samples received but not yet part of a completed chunk.
ProbabilityBuffer probability_buffer ()
 The device buffer of the probabilities.
std::vector< float > process (std::span< const float > samples)
 Append samples and run every chunk they complete.
size_t record (std::span< const float > samples)
 Run every chunk the samples complete, without waiting for the result.
void reset ()
 Forget the recurrent state, the context and any pending samples.
size_t resident_count () const noexcept
 Probabilities the latest run left in probability_buffer.
 SileroVadModel (const Kakshya::TensorSet &weights)
 Upload the weights and prepare the GPU resources.
 SileroVadModel (const Kakshya::TensorSet &weights, std::shared_ptr< GpuDispatchRecorder > recording)
 As above, appending to a recording the caller shares with other work.
 SileroVadModel (const SileroVadModel &)=delete
 SileroVadModel (SileroVadModel &&)=delete
std::vector< float > take_hidden ()
 The recurrent output kept since the last call, hidden_width values per chunk, chunk after chunk.
 ~SileroVadModel ()

Static Public Member Functions

static std::string diagnose (const Kakshya::TensorSet &weights)
 Check that a tensor set holds the weights the network needs.
static StreamFraming framing () noexcept
 The framing of this network.

Static Public Attributes

static constexpr uint32_t chunk_samples = 512
static constexpr uint32_t context_samples = 64
static constexpr uint32_t hidden_width = 128
 Width of the recurrent state the network holds for each chunk.
static constexpr uint32_t sample_rate = 16000
 Sample rate in Hz the network was trained and runs at.

Private Attributes

std::unique_ptr< Impl > m_impl

Detailed Description

Silero VAD at 16 kHz as a stream: samples in, one speech probability per chunk out.

Built from the 15 tensors of silero_vad_16k.safetensors. Each chunk is 512 samples. The 64 samples before a chunk are prepended as context and the frame is reflect padded by 64 on the right. A 258 channel strided convolution is the transform, its two channel halves give a magnitude, four convolutions with ReLU encode it, an LSTM cell carries the state from chunk to chunk, and a ReLU, a 1x1 convolution and a sigmoid give the probability.

Everything after the upload runs on the GPU in one submission per call. The pad, transform and encoder do not depend on earlier chunks, so a call over many chunks runs them as one batch and only the recurrence is sequential. The recurrent state, the 64 sample context and a partial chunk are kept between calls, so the model continues a stream. It runs on native float and agrees with a double reference only to float precision. The result of a chunk is available only after the chunk is complete, 32 ms of audio plus the compute time, and the model must not run on the audio thread.

Definition at line 50 of file SileroVadModel.hpp.


The documentation for this class was generated from the following files: