MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
SileroVadModel.hpp
Go to the documentation of this file.
1#pragma once
2
4
5/**
6 * @file SileroVadModel.hpp
7 * @brief The Silero voice activity network run on the GPU, one probability per 512 sample chunk.
8 */
9
10namespace MayaFlux::Yantra {
11
12class GpuDispatchCore;
14
15/**
16 * @struct StreamFraming
17 * @brief How a stream classifier cuts its input: sample rate, chunk length, the context carried into each chunk, and the recurrent state width.
18 */
20 /** Sample rate in Hz the network runs at. */
21 uint32_t sample_rate;
22 /** Samples in one chunk, one output per chunk. */
23 uint32_t chunk_samples;
24 /** Samples before a chunk that are carried in as context. */
26 /** Width of the recurrent state kept for each chunk. */
27 uint32_t hidden_width;
28};
29
30/**
31 * @class SileroVadModel
32 * @brief Silero VAD at 16 kHz as a stream: samples in, one speech probability per chunk out.
33 *
34 * Built from the 15 tensors of silero_vad_16k.safetensors. Each chunk is 512
35 * samples. The 64 samples before a chunk are prepended as context and the
36 * frame is reflect padded by 64 on the right. A 258 channel strided
37 * convolution is the transform, its two channel halves give a magnitude, four
38 * convolutions with ReLU encode it, an LSTM cell carries the state from chunk
39 * to chunk, and a ReLU, a 1x1 convolution and a sigmoid give the probability.
40 *
41 * Everything after the upload runs on the GPU in one submission per call. The
42 * pad, transform and encoder do not depend on earlier chunks, so a call over
43 * many chunks runs them as one batch and only the recurrence is sequential.
44 * The recurrent state, the 64 sample context and a partial chunk are kept
45 * between calls, so the model continues a stream. It runs on native float and
46 * agrees with a double reference only to float precision. The result of a
47 * chunk is available only after the chunk is complete, 32 ms of audio plus
48 * the compute time, and the model must not run on the audio thread.
49 */
50class MAYAFLUX_API SileroVadModel {
51public:
52 static constexpr uint32_t chunk_samples = 512;
53 static constexpr uint32_t context_samples = 64;
54 /** Sample rate in Hz the network was trained and runs at. */
55 static constexpr uint32_t sample_rate = 16000;
56 /** Width of the recurrent state the network holds for each chunk. */
57 static constexpr uint32_t hidden_width = 128;
58
59 /** @brief The framing of this network. VadAnalyzer reads its window, hop and rate from it. */
60 [[nodiscard]] static StreamFraming framing() noexcept;
61
62 /**
63 * @brief Check that a tensor set holds the weights the network needs.
64 * @param weights Candidate weights.
65 * @return Empty when usable, otherwise the first missing tensor or shape mismatch.
66 */
67 [[nodiscard]] static std::string diagnose(const Kakshya::TensorSet& weights);
68
69 /**
70 * @brief Upload the weights and prepare the GPU resources.
71 * @param weights A set that diagnose accepts. Other element types than float32 are converted.
72 * @throws std::invalid_argument When diagnose rejects the set, or a tensor cannot become float32. Raised before any device is touched.
73 * @throws std::runtime_error When no device is available.
74 */
75 explicit SileroVadModel(const Kakshya::TensorSet& weights);
76
77 /**
78 * @brief As above, appending to a recording the caller shares with other work.
79 *
80 * With a shared recording the network and a consumer such as the interval
81 * extractor go to the device in one submission. The recording must be
82 * compute isolated.
83 *
84 * @param weights A set that diagnose accepts.
85 * @param recording Recording the model records into. A null recording gives the model its own.
86 */
87 SileroVadModel(const Kakshya::TensorSet& weights, std::shared_ptr<GpuDispatchRecorder> recording);
89
91 SileroVadModel& operator=(const SileroVadModel&) = delete;
93 SileroVadModel& operator=(SileroVadModel&&) = delete;
94
95 /**
96 * @brief Append samples and run every chunk they complete.
97 * @param samples Mono samples at 16 kHz, continuing the previous call.
98 * @return One probability per completed chunk, in order. Empty while a chunk is incomplete.
99 */
100 [[nodiscard]] std::vector<float> process(std::span<const float> samples);
101
102 /**
103 * @brief Where the probabilities of the latest run are on the device.
104 *
105 * The buffer holds float32 probabilities, one per chunk, and is overwritten
106 * by the next run. A consumer on the same recording imports it with
107 * import_shared. It follows the model when the buffer grows.
108 */
110 const GpuDispatchCore* context { nullptr };
111 uint32_t set { 0 };
112 size_t binding { 0 };
113 };
114
115 /** @brief The device buffer of the probabilities. It exists from this call on. */
116 [[nodiscard]] ProbabilityBuffer probability_buffer();
117
118 /**
119 * @brief Run every chunk the samples complete, without waiting for the result.
120 *
121 * The probabilities stay on the device, in probability_buffer, and are
122 * recorded into the model's recording; nothing is submitted. Submit the
123 * recording, for instance by recording a consumer into it and submitting
124 * that, then use resident_count for how many probabilities there are. At
125 * most 4096 chunks run per call and the rest stay pending, so a consumer
126 * must read the buffer before the next call.
127 *
128 * @param samples Mono samples at 16 kHz, continuing the previous call.
129 * @return The number of chunks recorded.
130 */
131 [[nodiscard]] size_t record(std::span<const float> samples);
132
133 /** @brief Probabilities the latest run left in probability_buffer. */
134 [[nodiscard]] size_t resident_count() const noexcept;
135
136 /**
137 * @brief Where the recurrent output of the latest run is on the device.
138 *
139 * The buffer holds hidden_width float32 values per chunk, chunk after
140 * chunk: what the network knows about the stream at each chunk, from
141 * which its last layer derives the probability. It was trained only to
142 * tell speech from other sound, so nothing else is claimed of it. It is
143 * overwritten by the next run and follows the model when it grows.
144 */
145 [[nodiscard]] ProbabilityBuffer hidden_buffer();
146
147 /**
148 * @brief Keep the recurrent output of every chunk process and flush run, for take_hidden.
149 * @param keep Off by default. Turning it off discards what was kept.
150 */
151 void keep_hidden(bool keep);
152
153 /**
154 * @brief The recurrent output kept since the last call, hidden_width values per chunk, chunk after chunk.
155 * @return Empty unless keep_hidden is on. Chunks run by record are not kept.
156 */
157 [[nodiscard]] std::vector<float> take_hidden();
158
159 /**
160 * @brief Run the partial chunk, padded with zeros to a whole one.
161 * @return One probability, or nothing when no samples are pending.
162 */
163 [[nodiscard]] std::vector<float> flush();
164
165 /**
166 * @brief Forget the recurrent state, the context and any pending samples.
167 */
168 void reset();
169
170 /** @brief Samples received but not yet part of a completed chunk. */
171 [[nodiscard]] size_t pending() const noexcept;
172
173private:
174 struct Impl;
175 std::unique_ptr<Impl> m_impl;
176};
177
178} // namespace MayaFlux::Yantra
Non-template base that owns all type-independent GPU dispatch logic.
Shared command buffer for GPU dispatch work.
static constexpr uint32_t sample_rate
Sample rate in Hz the network was trained and runs at.
static std::string diagnose(const Kakshya::TensorSet &weights)
Check that a tensor set holds the weights the network needs.
static constexpr uint32_t hidden_width
Width of the recurrent state the network holds for each chunk.
static constexpr uint32_t context_samples
SileroVadModel(const Kakshya::TensorSet &weights)
Upload the weights and prepare the GPU resources.
std::vector< float > process(std::span< const float > samples)
Append samples and run every chunk they complete.
static StreamFraming framing() noexcept
The framing of this network.
static constexpr uint32_t chunk_samples
std::pair< std::shared_ptr< Kriya::BufferPipeline >, std::shared_ptr< Buffers::AudioBuffer > > record(const std::shared_ptr< Kakshya::DynamicSoundStream > &stream, uint32_t input_channel, double seconds, std::optional< uint32_t > monitor, uint32_t channel)
Record a live input channel into a stream for a number of seconds.
Definition Rigs.cpp:262
Where the probabilities of the latest run are on the device.
uint32_t chunk_samples
Samples in one chunk, one output per chunk.
uint32_t context_samples
Samples before a chunk that are carried in as context.
uint32_t sample_rate
Sample rate in Hz the network runs at.
uint32_t hidden_width
Width of the recurrent state kept for each chunk.
How a stream classifier cuts its input: sample rate, chunk length, the context carried into each chun...