MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::Yantra::HubertPhonemeModel Class Reference

HuBERT base fine tuned for English phonemes at 16 kHz: an utterance in, a distribution over phonemes per frame out. More...

#include <HubertPhonemeModel.hpp>

Collaboration diagram for MayaFlux::Yantra::HubertPhonemeModel:

Classes

struct  DeviceBuffer
 Where a result of the latest utterance is on the device. More...
struct  Impl

Public Member Functions

DeviceBuffer hidden_buffer ()
 The device buffer of the states, hidden_width rows of frames.
std::vector< float > hidden_states ()
 The state before the final layer for every frame of the latest utterance.
 HubertPhonemeModel (const HubertPhonemeModel &)=delete
 HubertPhonemeModel (const Kakshya::TensorSet &weights)
 Upload the weights and prepare the GPU resources.
 HubertPhonemeModel (const Kakshya::TensorSet &weights, std::shared_ptr< GpuDispatchRecorder > recording)
 As above, appending to a recording the caller shares with other work.
 HubertPhonemeModel (HubertPhonemeModel &&)=delete
HubertPhonemeModel & operator= (const HubertPhonemeModel &)=delete
HubertPhonemeModel & operator= (HubertPhonemeModel &&)=delete
DeviceBuffer posterior_buffer ()
 The device buffer of the distributions, classes rows of frames.
std::vector< float > process (std::span< const float > samples)
 Run one utterance.
size_t resident_frames () const noexcept
 Frames the latest utterance left on the device.
 ~HubertPhonemeModel ()

Static Public Member Functions

static std::string diagnose (const Kakshya::TensorSet &weights)
 Check that a tensor set holds the weights the network needs.
static uint32_t frame_count (size_t samples) noexcept
 Frames an utterance of this many samples gives, 0 below frame_span samples.
static bool is_vowel (uint32_t index) noexcept
 Whether a class is a vowel phoneme.
static std::string_view label (uint32_t index) noexcept
 Name of a class: the ARPABET symbol of a phoneme, or a bracketed name for the others.
static PhonemeVocabulary vocabulary () noexcept
 The vocabulary of this network: the names, the phoneme count, the blank class and the vowels.

Static Public Attributes

static constexpr uint32_t blank_class = 40
 Class the network emits between phonemes.
static constexpr uint32_t classes = 43
 Scores per frame.
static constexpr uint32_t frame_hop = 320
 Samples between the starts of successive frames.
static constexpr uint32_t frame_span = 400
 Samples one frame's features are computed from.
static constexpr uint32_t hidden_width = 768
 Width of the state before the final layer.
static constexpr size_t max_samples = 480000
 Longest utterance one call accepts, 30 s.
static constexpr uint32_t phonemes = 39
 Classes 0 to phonemes - 1 are phonemes.
static constexpr uint32_t sample_rate = 16000
 Sample rate in Hz the network was trained and runs at.

Private Attributes

std::unique_ptr< Impl > m_impl

Detailed Description

HuBERT base fine tuned for English phonemes at 16 kHz: an utterance in, a distribution over phonemes per frame out.

Built from the 212 tensors of hubert_base_phoneme_en.safetensors. Seven convolutions turn the waveform into one 512 wide feature every 320 samples, each seeing 400 samples. A convolution over 128 frames adds position, twelve attention layers let every frame read every other frame of the utterance, and a final layer gives 43 scores per frame: 39 ARPABET phonemes, an unknown class, the blank class and two unused markers.

The network was trained to emit each phoneme once, on one frame, and the blank class on the frames between, so the distribution says which phoneme is recognised near a frame, not where it starts or ends. The 768 wide state before the final layer is kept as well: frames that sound alike in context lie close together in it.

One call is one utterance. Every frame depends on the whole call, the input is brought to zero mean and unit variance over the call, and nothing is carried to the next call. An utterance is at most max_samples long. It runs on native float and agrees with a double reference only to float precision. Trained on read English speech; it must not run on the audio thread.

Definition at line 55 of file HubertPhonemeModel.hpp.


The documentation for this class was generated from the following files: