|
MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
|
HuBERT base fine tuned for English phonemes at 16 kHz: an utterance in, a distribution over phonemes per frame out. More...
#include <HubertPhonemeModel.hpp>
Classes | |
| struct | DeviceBuffer |
| Where a result of the latest utterance is on the device. More... | |
| struct | Impl |
Public Member Functions | |
| DeviceBuffer | hidden_buffer () |
| The device buffer of the states, hidden_width rows of frames. | |
| std::vector< float > | hidden_states () |
| The state before the final layer for every frame of the latest utterance. | |
| HubertPhonemeModel (const HubertPhonemeModel &)=delete | |
| HubertPhonemeModel (const Kakshya::TensorSet &weights) | |
| Upload the weights and prepare the GPU resources. | |
| HubertPhonemeModel (const Kakshya::TensorSet &weights, std::shared_ptr< GpuDispatchRecorder > recording) | |
| As above, appending to a recording the caller shares with other work. | |
| HubertPhonemeModel (HubertPhonemeModel &&)=delete | |
| HubertPhonemeModel & | operator= (const HubertPhonemeModel &)=delete |
| HubertPhonemeModel & | operator= (HubertPhonemeModel &&)=delete |
| DeviceBuffer | posterior_buffer () |
| The device buffer of the distributions, classes rows of frames. | |
| std::vector< float > | process (std::span< const float > samples) |
| Run one utterance. | |
| size_t | resident_frames () const noexcept |
| Frames the latest utterance left on the device. | |
| ~HubertPhonemeModel () | |
Static Public Member Functions | |
| static std::string | diagnose (const Kakshya::TensorSet &weights) |
| Check that a tensor set holds the weights the network needs. | |
| static uint32_t | frame_count (size_t samples) noexcept |
| Frames an utterance of this many samples gives, 0 below frame_span samples. | |
| static bool | is_vowel (uint32_t index) noexcept |
| Whether a class is a vowel phoneme. | |
| static std::string_view | label (uint32_t index) noexcept |
| Name of a class: the ARPABET symbol of a phoneme, or a bracketed name for the others. | |
| static PhonemeVocabulary | vocabulary () noexcept |
| The vocabulary of this network: the names, the phoneme count, the blank class and the vowels. | |
Static Public Attributes | |
| static constexpr uint32_t | blank_class = 40 |
| Class the network emits between phonemes. | |
| static constexpr uint32_t | classes = 43 |
| Scores per frame. | |
| static constexpr uint32_t | frame_hop = 320 |
| Samples between the starts of successive frames. | |
| static constexpr uint32_t | frame_span = 400 |
| Samples one frame's features are computed from. | |
| static constexpr uint32_t | hidden_width = 768 |
| Width of the state before the final layer. | |
| static constexpr size_t | max_samples = 480000 |
| Longest utterance one call accepts, 30 s. | |
| static constexpr uint32_t | phonemes = 39 |
| Classes 0 to phonemes - 1 are phonemes. | |
| static constexpr uint32_t | sample_rate = 16000 |
| Sample rate in Hz the network was trained and runs at. | |
Private Attributes | |
| std::unique_ptr< Impl > | m_impl |
HuBERT base fine tuned for English phonemes at 16 kHz: an utterance in, a distribution over phonemes per frame out.
Built from the 212 tensors of hubert_base_phoneme_en.safetensors. Seven convolutions turn the waveform into one 512 wide feature every 320 samples, each seeing 400 samples. A convolution over 128 frames adds position, twelve attention layers let every frame read every other frame of the utterance, and a final layer gives 43 scores per frame: 39 ARPABET phonemes, an unknown class, the blank class and two unused markers.
The network was trained to emit each phoneme once, on one frame, and the blank class on the frames between, so the distribution says which phoneme is recognised near a frame, not where it starts or ends. The 768 wide state before the final layer is kept as well: frames that sound alike in context lie close together in it.
One call is one utterance. Every frame depends on the whole call, the input is brought to zero mean and unit variance over the call, and nothing is carried to the next call. An utterance is at most max_samples long. It runs on native float and agrees with a double reference only to float precision. Trained on read English speech; it must not run on the audio thread.
Definition at line 55 of file HubertPhonemeModel.hpp.