MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
HubertPhonemeModel.hpp
Go to the documentation of this file.
1#pragma once
2
4
5/**
6 * @file HubertPhonemeModel.hpp
7 * @brief A HuBERT base network with a phoneme head run on the GPU, one phoneme distribution per 20 ms frame.
8 */
9
10namespace MayaFlux::Yantra {
11
12class GpuDispatchCore;
14
15/**
16 * @struct PhonemeVocabulary
17 * @brief The classes of a phoneme network: their names, how many are phonemes, the blank class, and the vowels.
18 *
19 * The views refer to static tables and stay valid for the program's lifetime.
20 */
22 /** Name of each class, as HubertPhonemeModel::label gives it. */
23 std::span<const std::string_view> labels;
24 /** Classes 0 to phonemes - 1 are phonemes. */
25 uint32_t phonemes;
26 /** Class the network emits between phonemes. */
27 uint32_t blank;
28 /** Classes that are vowel phonemes, as HubertPhonemeModel::is_vowel gives them. */
29 std::span<const uint32_t> vowels;
30};
31
32/**
33 * @class HubertPhonemeModel
34 * @brief HuBERT base fine tuned for English phonemes at 16 kHz: an utterance in, a distribution over phonemes per frame out.
35 *
36 * Built from the 212 tensors of hubert_base_phoneme_en.safetensors. Seven
37 * convolutions turn the waveform into one 512 wide feature every 320 samples,
38 * each seeing 400 samples. A convolution over 128 frames adds position, twelve
39 * attention layers let every frame read every other frame of the utterance,
40 * and a final layer gives 43 scores per frame: 39 ARPABET phonemes, an unknown
41 * class, the blank class and two unused markers.
42 *
43 * The network was trained to emit each phoneme once, on one frame, and the
44 * blank class on the frames between, so the distribution says which phoneme is
45 * recognised near a frame, not where it starts or ends. The 768 wide state
46 * before the final layer is kept as well: frames that sound alike in context
47 * lie close together in it.
48 *
49 * One call is one utterance. Every frame depends on the whole call, the input
50 * is brought to zero mean and unit variance over the call, and nothing is
51 * carried to the next call. An utterance is at most max_samples long. It runs
52 * on native float and agrees with a double reference only to float precision.
53 * Trained on read English speech; it must not run on the audio thread.
54 */
55class MAYAFLUX_API HubertPhonemeModel {
56public:
57 /** Sample rate in Hz the network was trained and runs at. */
58 static constexpr uint32_t sample_rate = 16000;
59 /** Samples between the starts of successive frames. */
60 static constexpr uint32_t frame_hop = 320;
61 /** Samples one frame's features are computed from. */
62 static constexpr uint32_t frame_span = 400;
63 /** Scores per frame. */
64 static constexpr uint32_t classes = 43;
65 /** Classes 0 to phonemes - 1 are phonemes. */
66 static constexpr uint32_t phonemes = 39;
67 /** Class the network emits between phonemes. */
68 static constexpr uint32_t blank_class = 40;
69 /** Width of the state before the final layer. */
70 static constexpr uint32_t hidden_width = 768;
71 /** Longest utterance one call accepts, 30 s. */
72 static constexpr size_t max_samples = 480000;
73
74 /**
75 * @brief Check that a tensor set holds the weights the network needs.
76 * @param weights Candidate weights.
77 * @return Empty when usable, otherwise the first missing tensor or shape mismatch.
78 */
79 [[nodiscard]] static std::string diagnose(const Kakshya::TensorSet& weights);
80
81 /**
82 * @brief Name of a class: the ARPABET symbol of a phoneme, or a bracketed name for the others.
83 * @param index Class, 0 to classes - 1. Anything else gives an empty name.
84 */
85 [[nodiscard]] static std::string_view label(uint32_t index) noexcept;
86
87 /** @brief Whether a class is a vowel phoneme. Diphthongs and the r coloured vowel count. */
88 [[nodiscard]] static bool is_vowel(uint32_t index) noexcept;
89
90 /** @brief The vocabulary of this network: the names, the phoneme count, the blank class and the vowels. */
91 [[nodiscard]] static PhonemeVocabulary vocabulary() noexcept;
92
93 /** @brief Frames an utterance of this many samples gives, 0 below frame_span samples. */
94 [[nodiscard]] static uint32_t frame_count(size_t samples) noexcept;
95
96 /**
97 * @brief Upload the weights and prepare the GPU resources.
98 * @param weights A set that diagnose accepts. Other element types than float32 are converted.
99 * @throws std::invalid_argument When diagnose rejects the set, raised before any device is touched, or a tensor cannot become float32.
100 * @throws std::runtime_error When no device is available.
101 */
102 explicit HubertPhonemeModel(const Kakshya::TensorSet& weights);
103
104 /**
105 * @brief As above, appending to a recording the caller shares with other work.
106 * @param weights A set that diagnose accepts.
107 * @param recording Recording the model records into, compute isolated. A null recording gives the model its own.
108 */
109 HubertPhonemeModel(const Kakshya::TensorSet& weights, std::shared_ptr<GpuDispatchRecorder> recording);
111
113 HubertPhonemeModel& operator=(const HubertPhonemeModel&) = delete;
116
117 /**
118 * @brief Run one utterance.
119 * @param samples Mono samples at 16 kHz.
120 * @return The distribution of every frame in turn, frame_count(samples.size()) * classes values, each frame summing to 1. Empty below frame_span samples.
121 * @throws std::invalid_argument When the utterance is longer than max_samples.
122 */
123 [[nodiscard]] std::vector<float> process(std::span<const float> samples);
124
125 /**
126 * @brief The state before the final layer for every frame of the latest utterance.
127 * @return resident_frames() * hidden_width values, frame after frame.
128 */
129 [[nodiscard]] std::vector<float> hidden_states();
130
131 /** @brief Frames the latest utterance left on the device. */
132 [[nodiscard]] size_t resident_frames() const noexcept;
133
134 /**
135 * @brief Where a result of the latest utterance is on the device.
136 *
137 * The buffer holds float32 values, one row of resident_frames() values
138 * per class or per state channel, and is overwritten by the next call. A
139 * consumer on the same recording imports it with import_shared. It follows
140 * the model when the buffer grows.
141 */
143 const GpuDispatchCore* context { nullptr };
144 uint32_t set { 0 };
145 size_t binding { 0 };
146 };
147
148 /** @brief The device buffer of the distributions, classes rows of frames. It exists from this call on. */
149 [[nodiscard]] DeviceBuffer posterior_buffer();
150
151 /** @brief The device buffer of the states, hidden_width rows of frames. It exists from this call on. */
152 [[nodiscard]] DeviceBuffer hidden_buffer();
153
154private:
155 struct Impl;
156 std::unique_ptr<Impl> m_impl;
157};
158
159} // namespace MayaFlux::Yantra
Non-template base that owns all type-independent GPU dispatch logic.
Shared command buffer for GPU dispatch work.
static constexpr uint32_t frame_hop
Samples between the starts of successive frames.
static constexpr uint32_t phonemes
Classes 0 to phonemes - 1 are phonemes.
HubertPhonemeModel(const Kakshya::TensorSet &weights)
Upload the weights and prepare the GPU resources.
static constexpr uint32_t classes
Scores per frame.
static bool is_vowel(uint32_t index) noexcept
Whether a class is a vowel phoneme.
static std::string diagnose(const Kakshya::TensorSet &weights)
Check that a tensor set holds the weights the network needs.
static constexpr uint32_t blank_class
Class the network emits between phonemes.
static constexpr uint32_t sample_rate
Sample rate in Hz the network was trained and runs at.
static PhonemeVocabulary vocabulary() noexcept
The vocabulary of this network: the names, the phoneme count, the blank class and the vowels.
static std::string_view label(uint32_t index) noexcept
Name of a class: the ARPABET symbol of a phoneme, or a bracketed name for the others.
std::vector< float > process(std::span< const float > samples)
Run one utterance.
static constexpr uint32_t frame_span
Samples one frame's features are computed from.
size_t resident_frames() const noexcept
Frames the latest utterance left on the device.
static constexpr uint32_t hidden_width
Width of the state before the final layer.
static constexpr size_t max_samples
Longest utterance one call accepts, 30 s.
std::vector< float > hidden_states()
The state before the final layer for every frame of the latest utterance.
static uint32_t frame_count(size_t samples) noexcept
Frames an utterance of this many samples gives, 0 below frame_span samples.
Named tensors that belong together, such as a model's weights or I/O.
Definition Tensor.hpp:167
Where a result of the latest utterance is on the device.
uint32_t blank
Class the network emits between phonemes.
std::span< const uint32_t > vowels
Classes that are vowel phonemes, as HubertPhonemeModel::is_vowel gives them.
uint32_t phonemes
Classes 0 to phonemes - 1 are phonemes.
std::span< const std::string_view > labels
Name of each class, as HubertPhonemeModel::label gives it.
The classes of a phoneme network: their names, how many are phonemes, the blank class,...