|
MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
|
Which phoneme is spoken near each frame of a mono 16 kHz utterance, from a learned network. More...
#include <PhonemeAnalyzer.hpp>
Public Types | |
| using | base_type = Analyzer<InputType, OutputType> |
| using | input_type = Datum<InputType> |
| using | output_type = Datum<OutputType> |
| using | supports = Supports<PhonemeAnalyzerConfig> |
| Public Types inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType > | |
| using | base_type = ComputeOperation<InputType, OutputType> |
| using | input_type = Datum<InputType> |
| using | output_type = Datum<OutputType> |
| Public Types inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType > | |
| using | input_type = Datum<InputType> |
| using | output_type = Datum<OutputType> |
Public Member Functions | |
| PhonemeAnalysis | analyze_phonemes (const input_type &data) |
| Type-safe phoneme analysis. | |
| PhonemeAnalysis | analyze_phonemes (const InputType &data) |
| ConfigRefs | configs () const override |
| The base view of the config structs this operation holds. | |
| AnalysisType | get_analysis_type () const override |
| Gets the analysis type category for this analyzer. | |
| std::string | get_name () const override |
| Get operation name for debugging/introspection. | |
| PhonemeAnalysis | get_phoneme_analysis () const |
| HubertPhonemeModel::DeviceBuffer | hidden_buffer () |
| Where the states of the latest utterance are on the device, for a consumer on the same recording. | |
| PhonemeAnalyzer (const Kakshya::TensorSet &weights, PhonemeAnalyzerConfig config={}) | |
| Construct from the network weights. | |
| HubertPhonemeModel::DeviceBuffer | posterior_buffer () |
| Where the probabilities of the latest utterance are on the device, for a consumer on the same recording. | |
| size_t | resident_frames () const noexcept |
| Frames the latest utterance left on the device. | |
| void | share_recording (std::shared_ptr< GpuDispatchRecorder > recording) |
| Append to a recording the caller shares with other work. | |
| Public Member Functions inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType > | |
| std::vector< std::any > | analyze_batch (const std::vector< input_type > &inputs) |
| Batch analysis for multiple inputs. | |
| std::vector< std::any > | analyze_batch (const std::vector< InputType > &inputs) |
| std::any | analyze_data (const input_type &data) |
| User-facing analysis method - returns analysis results directly. | |
| std::any | analyze_data (const InputType &data) |
| std::any | get_current_analysis () const |
| Access cached analysis from last operation. | |
| OperationType | get_operation_type () const override |
| Returns the category of this operation for grammar and registry discovery. | |
| bool | has_current_analysis () const |
| Check if analysis has been performed. | |
| bool | validate_input (const input_type &input) const override |
| Validates input data meets analyzer requirements. | |
| virtual | ~Analyzer ()=default |
| Public Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType > | |
| void | add_dependency (std::shared_ptr< ComputeOperation > dep) |
| output_type | apply_operation (const input_type &input) |
| Public synchronous execution interface. | |
| output_type | apply_operation_with_dependencies (const input_type &input) |
| Applies the operation with dependencies resolved. | |
| OutputType | apply_to_data (const InputType &data) |
| Convenience overload that extracts just the data from result. | |
| ComputeOperation () | |
| Constructor with data type validation warnings. | |
| template<typename Config> requires std::derived_from<Config, OperationConfig> | |
| const Config & | config () const |
| The stored config struct of type Config, or its defaults when none was set. | |
| output_type | execute (const input_type &input) |
| OpUnit interface - operations can act as units in dependency graphs. | |
| const auto & | get_dependencies () const |
| const ExecutionContext & | get_last_execution_context () const |
| void | merge_config (const Facets &incoming) |
Adds the config structs of incoming, replacing stored ones of the same type. | |
| output_type | operator() (const InputType &data) |
| Convenience overload for direct data processing (backward compatibility). | |
| template<typename Config> requires std::derived_from<Config, OperationConfig> | |
| const Config * | resolved_config () const |
| The stored config struct of type Config, or nullptr when absent. | |
| template<typename Config> requires std::derived_from<Config, OperationConfig> | |
| void | set_config (Config config) |
| Stores a config struct, whole, under its own type. | |
| void | set_last_execution_context (const ExecutionContext &ctx) |
| void | set_post_execution_hook (const OperationHookCallback &hook) |
| void | set_pre_execution_hook (const OperationHookCallback &hook) |
| void | set_reconstruction_callback (const ReconstructionCallback &callback) |
| template<typename... Configs> | |
| ConfigRefs | stored_configs (Supports< Configs... >) const |
| The stored configs among those listed, in listed order. | |
| virtual | ~ComputeOperation ()=default |
| Virtual destructor for proper cleanup of derived classes. | |
Static Public Member Functions | |
| static std::string | diagnose (const PhonemeAnalyzerConfig &config) |
| Why a config cannot be analysed. | |
| static float | fitness (const input_type &) |
| One while a device is up, zero otherwise. | |
Static Public Attributes | |
| static constexpr Target | target = targets::vulkan |
Protected Member Functions | |
| output_type | analyze_implementation (const input_type &input) override |
| Pure virtual analysis implementation - derived classes implement this. | |
| Protected Member Functions inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType > | |
| virtual output_type | apply_granularity_formatting (const output_type &raw_output) |
| Output formatting by granularity (override for custom formatting). | |
| output_type | operation_function (const input_type &input) override |
| Core analysis implementation - must be overridden by derived classes. | |
| template<typename AnalysisResultType> | |
| void | store_current_analysis (AnalysisResultType &&result) const |
| virtual bool | validate_analysis_input (const input_type &) const |
| Input validation (override for custom validation logic). | |
| Protected Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType > | |
| output_type | apply_hooks (const input_type &input, const ExecutionContext &context) |
| virtual std::future< output_type > | apply_operation_async (const input_type &input) |
| Optional async implementation - default delegates to operation_function. | |
| virtual output_type | apply_operation_chained (const input_type &input, const ExecutionContext &ctx) |
| Optional chain-aware implementation - default delegates to operation_function. | |
| virtual output_type | apply_operation_internal (const input_type &input, const ExecutionContext &context) |
| Internal execution method - ComputeMatrix can access this. | |
| virtual output_type | apply_operation_parallel (const input_type &input, const ExecutionContext &ctx) |
| Optional parallel-aware implementation - default delegates to operation_function. | |
| output_type | convert_result (std::vector< std::vector< double > > &result_data, DataStructureInfo &metadata) |
| Convert processed double data back to OutputType using metadata and optional callback. | |
Private Member Functions | |
| output_type | make_output (const PhonemeAnalysis &analysis, PhonemeMethod method, DataStructureInfo &structure_info) |
| HubertPhonemeModel & | network () const |
Static Private Member Functions | |
| static void | bound_events (PhonemeAnalysis &analysis, size_t samples) |
| Gives every event its sample range. | |
| static void | describe_frames (PhonemeAnalysis &analysis) |
| static output_type | empty_output () |
| static std::vector< std::vector< double > > | select_values (const PhonemeAnalysis &analysis, PhonemeMethod method) |
Private Attributes | |
| std::unique_ptr< HubertPhonemeModel > | m_model |
| std::shared_ptr< GpuDispatchRecorder > | m_recording |
| std::shared_ptr< const Kakshya::TensorSet > | m_weights |
Additional Inherited Members | |
| Protected Attributes inherited from MayaFlux::Yantra::Analyzer< InputType, OutputType > | |
| std::any | m_current_analysis |
| std::optional< output_type > | m_current_output |
| Protected Attributes inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType > | |
| ExecutionContext | m_last_execution_context |
Which phoneme is spoken near each frame of a mono 16 kHz utterance, from a learned network.
Where WaveformAnalyzer measures how a waveform repeats and VadAnalyzer says whether a sound is speech, this says what is being said, as phonemes: every frame is judged with the whole utterance as its context, which no measure of a single window can do.
The analyzer is named for the task. Today it runs one network, a HuBERT base fine tuned on read English, recognised by its tensors; weights for any other network are rejected with the reason. Its classes are the 39 ARPABET phonemes, an unknown class, the blank class and two unused markers, named by HubertPhonemeModel::label.
The network emits each phoneme on one frame and the blank class between, so the result tells which phonemes occur and on which frames. Where each starts and ends is an estimate, described at PhonemeEvent. One call is one utterance of at most 30 s: nothing carries over between calls, and a longer input is refused with the reason. Cut a long recording at its pauses first, for instance with VadAnalyzer and IntervalExtractor, which also keeps the network off everything that is not speech. One channel only.
The weights are an explicit dependency: the caller loads the network's tensors and passes them. The network runs in native float on the vulkan target and has no CPU twin. It is not in the default registry, because it cannot be built without its weights; catalog it with a factory to use it in a grammar. It must not run on the audio thread.
With a matrix as OutputType the output data is the per frame values the method selects. HELD_CLASSES is the one made for IntervalExtractor: with samples_per_score 320 and min_run, min_gap and padding set for single frames it gives the vowel and consonant intervals of the utterance. With Kakshya::CompositeArray it is one record per recognised phoneme in the layout of phoneme_record_layout, whatever the method. The analysis is the same either way.
Definition at line 230 of file PhonemeAnalyzer.hpp.