|
MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
|
Silero VAD at 16 kHz as a stream: samples in, one speech probability per chunk out. More...
#include <SileroVadModel.hpp>
Classes | |
| struct | Impl |
| struct | ProbabilityBuffer |
| Where the probabilities of the latest run are on the device. More... | |
Public Member Functions | |
| std::vector< float > | flush () |
| Run the partial chunk, padded with zeros to a whole one. | |
| ProbabilityBuffer | hidden_buffer () |
| Where the recurrent output of the latest run is on the device. | |
| void | keep_hidden (bool keep) |
| Keep the recurrent output of every chunk process and flush run, for take_hidden. | |
| SileroVadModel & | operator= (const SileroVadModel &)=delete |
| SileroVadModel & | operator= (SileroVadModel &&)=delete |
| size_t | pending () const noexcept |
| Samples received but not yet part of a completed chunk. | |
| ProbabilityBuffer | probability_buffer () |
| The device buffer of the probabilities. | |
| std::vector< float > | process (std::span< const float > samples) |
| Append samples and run every chunk they complete. | |
| size_t | record (std::span< const float > samples) |
| Run every chunk the samples complete, without waiting for the result. | |
| void | reset () |
| Forget the recurrent state, the context and any pending samples. | |
| size_t | resident_count () const noexcept |
| Probabilities the latest run left in probability_buffer. | |
| SileroVadModel (const Kakshya::TensorSet &weights) | |
| Upload the weights and prepare the GPU resources. | |
| SileroVadModel (const Kakshya::TensorSet &weights, std::shared_ptr< GpuDispatchRecorder > recording) | |
| As above, appending to a recording the caller shares with other work. | |
| SileroVadModel (const SileroVadModel &)=delete | |
| SileroVadModel (SileroVadModel &&)=delete | |
| std::vector< float > | take_hidden () |
| The recurrent output kept since the last call, hidden_width values per chunk, chunk after chunk. | |
| ~SileroVadModel () | |
Static Public Member Functions | |
| static std::string | diagnose (const Kakshya::TensorSet &weights) |
| Check that a tensor set holds the weights the network needs. | |
| static StreamFraming | framing () noexcept |
| The framing of this network. | |
Static Public Attributes | |
| static constexpr uint32_t | chunk_samples = 512 |
| static constexpr uint32_t | context_samples = 64 |
| static constexpr uint32_t | hidden_width = 128 |
| Width of the recurrent state the network holds for each chunk. | |
| static constexpr uint32_t | sample_rate = 16000 |
| Sample rate in Hz the network was trained and runs at. | |
Private Attributes | |
| std::unique_ptr< Impl > | m_impl |
Silero VAD at 16 kHz as a stream: samples in, one speech probability per chunk out.
Built from the 15 tensors of silero_vad_16k.safetensors. Each chunk is 512 samples. The 64 samples before a chunk are prepended as context and the frame is reflect padded by 64 on the right. A 258 channel strided convolution is the transform, its two channel halves give a magnitude, four convolutions with ReLU encode it, an LSTM cell carries the state from chunk to chunk, and a ReLU, a 1x1 convolution and a sigmoid give the probability.
Everything after the upload runs on the GPU in one submission per call. The pad, transform and encoder do not depend on earlier chunks, so a call over many chunks runs them as one batch and only the recurrence is sequential. The recurrent state, the 64 sample context and a partial chunk are kept between calls, so the model continues a stream. It runs on native float and agrees with a double reference only to float precision. The result of a chunk is available only after the chunk is complete, 32 ms of audio plus the compute time, and the model must not run on the audio thread.
Definition at line 50 of file SileroVadModel.hpp.