MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::Yantra::PhonemeAnalyzer< InputType, OutputType > Class Template Reference

Which phoneme is spoken near each frame of a mono 16 kHz utterance, from a learned network. More...

#include <PhonemeAnalyzer.hpp>

Inheritance diagram for MayaFlux::Yantra::PhonemeAnalyzer< InputType, OutputType >:
Collaboration diagram for MayaFlux::Yantra::PhonemeAnalyzer< InputType, OutputType >:

Public Types

using base_type = Analyzer<InputType, OutputType>
using input_type = Datum<InputType>
using output_type = Datum<OutputType>
using supports = Supports<PhonemeAnalyzerConfig>
Public Types inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType >
using base_type = ComputeOperation<InputType, OutputType>
using input_type = Datum<InputType>
using output_type = Datum<OutputType>
Public Types inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
using input_type = Datum<InputType>
using output_type = Datum<OutputType>

Public Member Functions

PhonemeAnalysis analyze_phonemes (const input_type &data)
 Type-safe phoneme analysis.
PhonemeAnalysis analyze_phonemes (const InputType &data)
ConfigRefs configs () const override
 The base view of the config structs this operation holds.
AnalysisType get_analysis_type () const override
 Gets the analysis type category for this analyzer.
std::string get_name () const override
 Get operation name for debugging/introspection.
PhonemeAnalysis get_phoneme_analysis () const
HubertPhonemeModel::DeviceBuffer hidden_buffer ()
 Where the states of the latest utterance are on the device, for a consumer on the same recording.
 PhonemeAnalyzer (const Kakshya::TensorSet &weights, PhonemeAnalyzerConfig config={})
 Construct from the network weights.
HubertPhonemeModel::DeviceBuffer posterior_buffer ()
 Where the probabilities of the latest utterance are on the device, for a consumer on the same recording.
size_t resident_frames () const noexcept
 Frames the latest utterance left on the device.
void share_recording (std::shared_ptr< GpuDispatchRecorder > recording)
 Append to a recording the caller shares with other work.
Public Member Functions inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType >
std::vector< std::any > analyze_batch (const std::vector< input_type > &inputs)
 Batch analysis for multiple inputs.
std::vector< std::any > analyze_batch (const std::vector< InputType > &inputs)
std::any analyze_data (const input_type &data)
 User-facing analysis method - returns analysis results directly.
std::any analyze_data (const InputType &data)
std::any get_current_analysis () const
 Access cached analysis from last operation.
OperationType get_operation_type () const override
 Returns the category of this operation for grammar and registry discovery.
bool has_current_analysis () const
 Check if analysis has been performed.
bool validate_input (const input_type &input) const override
 Validates input data meets analyzer requirements.
virtual ~Analyzer ()=default
Public Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
void add_dependency (std::shared_ptr< ComputeOperation > dep)
output_type apply_operation (const input_type &input)
 Public synchronous execution interface.
output_type apply_operation_with_dependencies (const input_type &input)
 Applies the operation with dependencies resolved.
OutputType apply_to_data (const InputType &data)
 Convenience overload that extracts just the data from result.
 ComputeOperation ()
 Constructor with data type validation warnings.
template<typename Config>
requires std::derived_from<Config, OperationConfig>
const Config & config () const
 The stored config struct of type Config, or its defaults when none was set.
output_type execute (const input_type &input)
 OpUnit interface - operations can act as units in dependency graphs.
const auto & get_dependencies () const
const ExecutionContext & get_last_execution_context () const
void merge_config (const Facets &incoming)
 Adds the config structs of incoming, replacing stored ones of the same type.
output_type operator() (const InputType &data)
 Convenience overload for direct data processing (backward compatibility).
template<typename Config>
requires std::derived_from<Config, OperationConfig>
const Config * resolved_config () const
 The stored config struct of type Config, or nullptr when absent.
template<typename Config>
requires std::derived_from<Config, OperationConfig>
void set_config (Config config)
 Stores a config struct, whole, under its own type.
void set_last_execution_context (const ExecutionContext &ctx)
void set_post_execution_hook (const OperationHookCallback &hook)
void set_pre_execution_hook (const OperationHookCallback &hook)
void set_reconstruction_callback (const ReconstructionCallback &callback)
template<typename... Configs>
ConfigRefs stored_configs (Supports< Configs... >) const
 The stored configs among those listed, in listed order.
virtual ~ComputeOperation ()=default
 Virtual destructor for proper cleanup of derived classes.

Static Public Member Functions

static std::string diagnose (const PhonemeAnalyzerConfig &config)
 Why a config cannot be analysed.
static float fitness (const input_type &)
 One while a device is up, zero otherwise.

Static Public Attributes

static constexpr Target target = targets::vulkan

Protected Member Functions

output_type analyze_implementation (const input_type &input) override
 Pure virtual analysis implementation - derived classes implement this.
Protected Member Functions inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType >
virtual output_type apply_granularity_formatting (const output_type &raw_output)
 Output formatting by granularity (override for custom formatting).
output_type operation_function (const input_type &input) override
 Core analysis implementation - must be overridden by derived classes.
template<typename AnalysisResultType>
void store_current_analysis (AnalysisResultType &&result) const
virtual bool validate_analysis_input (const input_type &) const
 Input validation (override for custom validation logic).
Protected Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
output_type apply_hooks (const input_type &input, const ExecutionContext &context)
virtual std::future< output_type > apply_operation_async (const input_type &input)
 Optional async implementation - default delegates to operation_function.
virtual output_type apply_operation_chained (const input_type &input, const ExecutionContext &ctx)
 Optional chain-aware implementation - default delegates to operation_function.
virtual output_type apply_operation_internal (const input_type &input, const ExecutionContext &context)
 Internal execution method - ComputeMatrix can access this.
virtual output_type apply_operation_parallel (const input_type &input, const ExecutionContext &ctx)
 Optional parallel-aware implementation - default delegates to operation_function.
output_type convert_result (std::vector< std::vector< double > > &result_data, DataStructureInfo &metadata)
 Convert processed double data back to OutputType using metadata and optional callback.

Private Member Functions

output_type make_output (const PhonemeAnalysis &analysis, PhonemeMethod method, DataStructureInfo &structure_info)
HubertPhonemeModel & network () const

Static Private Member Functions

static void bound_events (PhonemeAnalysis &analysis, size_t samples)
 Gives every event its sample range.
static void describe_frames (PhonemeAnalysis &analysis)
static output_type empty_output ()
static std::vector< std::vector< double > > select_values (const PhonemeAnalysis &analysis, PhonemeMethod method)

Private Attributes

std::unique_ptr< HubertPhonemeModel > m_model
std::shared_ptr< GpuDispatchRecorder > m_recording
std::shared_ptr< const Kakshya::TensorSet > m_weights

Additional Inherited Members

Protected Attributes inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType >
std::any m_current_analysis
std::optional< output_type > m_current_output
Protected Attributes inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
ExecutionContext m_last_execution_context

Detailed Description

template<ComputeData InputType = std::vector<Kakshya::DataVariant>, ComputeData OutputType = Eigen::MatrixXd>
class MayaFlux::Yantra::PhonemeAnalyzer< InputType, OutputType >

Which phoneme is spoken near each frame of a mono 16 kHz utterance, from a learned network.

Where WaveformAnalyzer measures how a waveform repeats and VadAnalyzer says whether a sound is speech, this says what is being said, as phonemes: every frame is judged with the whole utterance as its context, which no measure of a single window can do.

The analyzer is named for the task. Today it runs one network, a HuBERT base fine tuned on read English, recognised by its tensors; weights for any other network are rejected with the reason. Its classes are the 39 ARPABET phonemes, an unknown class, the blank class and two unused markers, named by HubertPhonemeModel::label.

The network emits each phoneme on one frame and the blank class between, so the result tells which phonemes occur and on which frames. Where each starts and ends is an estimate, described at PhonemeEvent. One call is one utterance of at most 30 s: nothing carries over between calls, and a longer input is refused with the reason. Cut a long recording at its pauses first, for instance with VadAnalyzer and IntervalExtractor, which also keeps the network off everything that is not speech. One channel only.

The weights are an explicit dependency: the caller loads the network's tensors and passes them. The network runs in native float on the vulkan target and has no CPU twin. It is not in the default registry, because it cannot be built without its weights; catalog it with a factory to use it in a grammar. It must not run on the audio thread.

With a matrix as OutputType the output data is the per frame values the method selects. HELD_CLASSES is the one made for IntervalExtractor: with samples_per_score 320 and min_run, min_gap and padding set for single frames it gives the vowel and consonant intervals of the utterance. With Kakshya::CompositeArray it is one record per recognised phoneme in the layout of phoneme_record_layout, whatever the method. The analysis is the same either way.

Definition at line 230 of file PhonemeAnalyzer.hpp.


The documentation for this class was generated from the following file: