MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::Yantra::PhonemeExtractor< InputType, OutputType > Class Template Reference

The phonemes of a whole mono 16 kHz recording, with the pauses between speech taken out. More...

#include <PhonemeExtractor.hpp>

Inheritance diagram for MayaFlux::Yantra::PhonemeExtractor< InputType, OutputType >:
Collaboration diagram for MayaFlux::Yantra::PhonemeExtractor< InputType, OutputType >:

Classes

struct  Piece

Public Types

using base_type = Extractor<InputType, OutputType>
using input_type = Datum<InputType>
using output_type = Datum<OutputType>
using supports = Supports<PhonemeExtractorConfig>
Public Types inherited from MayaFlux::Yantra::Extractor< InputType, OutputType >
using base_type = ComputeOperation<InputType, OutputType>
using input_type = Datum<InputType>
using output_type = Datum<OutputType>
Public Types inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
using input_type = Datum<InputType>
using output_type = Datum<OutputType>

Public Member Functions

ConfigRefs configs () const override
 The base view of the config structs this operation holds.
PhonemeExtraction extract_phonemes (const input_type &data)
 Type-safe phoneme extraction.
PhonemeExtraction extract_phonemes (const InputType &data)
ExtractionType get_extraction_type () const override
 Gets the extraction type category for this extractor.
 PhonemeExtractor (const Kakshya::TensorSet &vad_weights, const Kakshya::TensorSet &phoneme_weights, PhonemeExtractorConfig config={})
 Construct from the weights of both networks.
Public Member Functions inherited from MayaFlux::Yantra::Extractor< InputType, OutputType >
std::vector< OutputType > extract_batch (const std::vector< input_type > &inputs)
 Batch extraction for multiple inputs.
std::vector< OutputType > extract_batch (const std::vector< InputType > &inputs)
OutputType extract_data (const input_type &data)
 Type-safe extraction method.
OutputType extract_data (const InputType &data)
OutputType extract_with_scope (const input_type &data, ExtractionScope scope)
 Extract with specific scope, without changing the stored settings.
OutputType extract_with_scope (const InputType &data, ExtractionScope scope)
std::string get_name () const override
 Gets human-readable name for this extractor.
OperationType get_operation_type () const override
 Returns the category of this operation for grammar and registry discovery.
virtual ~Extractor ()=default
Public Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
void add_dependency (std::shared_ptr< ComputeOperation > dep)
output_type apply_operation (const input_type &input)
 Public synchronous execution interface.
output_type apply_operation_with_dependencies (const input_type &input)
 Applies the operation with dependencies resolved.
OutputType apply_to_data (const InputType &data)
 Convenience overload that extracts just the data from result.
 ComputeOperation ()
 Constructor with data type validation warnings.
template<typename Config>
requires std::derived_from<Config, OperationConfig>
const Config & config () const
 The stored config struct of type Config, or its defaults when none was set.
output_type execute (const input_type &input)
 OpUnit interface - operations can act as units in dependency graphs.
const auto & get_dependencies () const
const ExecutionContext & get_last_execution_context () const
void merge_config (const Facets &incoming)
 Adds the config structs of incoming, replacing stored ones of the same type.
output_type operator() (const InputType &data)
 Convenience overload for direct data processing (backward compatibility).
template<typename Config>
requires std::derived_from<Config, OperationConfig>
const Config * resolved_config () const
 The stored config struct of type Config, or nullptr when absent.
template<typename Config>
requires std::derived_from<Config, OperationConfig>
void set_config (Config config)
 Stores a config struct, whole, under its own type.
void set_last_execution_context (const ExecutionContext &ctx)
void set_post_execution_hook (const OperationHookCallback &hook)
void set_pre_execution_hook (const OperationHookCallback &hook)
void set_reconstruction_callback (const ReconstructionCallback &callback)
template<typename... Configs>
ConfigRefs stored_configs (Supports< Configs... >) const
 The stored configs among those listed, in listed order.
virtual bool validate_input (const input_type &) const
 Validates if the input data meets the operation's requirements.
virtual ~ComputeOperation ()=default
 Virtual destructor for proper cleanup of derived classes.

Static Public Member Functions

static std::string diagnose (const PhonemeExtractorConfig &config)
 Why a config cannot be extracted with.
static float fitness (const input_type &)
 One while a device is up, zero otherwise.

Static Public Attributes

static constexpr Target target = targets::vulkan

Protected Member Functions

output_type extract_implementation (const input_type &input) override
 Pure virtual extraction implementation - derived classes implement this.
std::string get_extractor_name () const override
 Get extractor-specific name (derived classes override this).
Protected Member Functions inherited from MayaFlux::Yantra::Extractor< InputType, OutputType >
virtual output_type apply_content_filtering (const output_type &raw_output)
 Apply content-based filtering (override for custom filtering).
virtual output_type apply_data_sampling (const output_type &raw_output)
 Apply data sampling (override for custom sampling).
virtual output_type apply_scope_filtering (const output_type &raw_output, ExtractionScope scope)
 Apply scope filtering to results.
virtual ExtractionScope extraction_scope () const
 The scope the stored config selects; FULL_DATA unless a family overrides it.
virtual output_type filter_to_target_regions (const output_type &raw_output)
 Filter results to target regions (override for custom filtering).
output_type operation_function (const input_type &input) override
 Core operation implementation - called by ComputeOperation interface.
virtual bool validate_extraction_input (const input_type &) const
 Input validation (override for custom validation logic).
Protected Member Functions inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
output_type apply_hooks (const input_type &input, const ExecutionContext &context)
virtual std::future< output_type > apply_operation_async (const input_type &input)
 Optional async implementation - default delegates to operation_function.
virtual output_type apply_operation_chained (const input_type &input, const ExecutionContext &ctx)
 Optional chain-aware implementation - default delegates to operation_function.
virtual output_type apply_operation_internal (const input_type &input, const ExecutionContext &context)
 Internal execution method - ComputeMatrix can access this.
virtual output_type apply_operation_parallel (const input_type &input, const ExecutionContext &ctx)
 Optional parallel-aware implementation - default delegates to operation_function.
output_type convert_result (std::vector< std::vector< double > > &result_data, DataStructureInfo &metadata)
 Convert processed double data back to OutputType using metadata and optional callback.

Private Member Functions

output_type build_output (const PhonemeExtraction &record, const std::vector< size_t > &rows, std::span< const double > samples, DataStructureInfo &info)
std::vector< std::pair< size_t, size_t > > find_voiced (std::span< const double > samples, const PhonemeExtractorConfig &cfg)
void recognise_phonemes (std::span< const double > samples, PhonemeExtraction &record)

Static Private Member Functions

static output_type empty_output ()
static bool gate_applies (const PhonemeExtractorConfig &cfg)
static bool is_selected (const PhonemeEvent &event, PhonemeSelection selection)
static PhonemeAnalyzerConfig phoneme_settings (const PhonemeExtractorConfig &config)
static std::vector< Piece > pieces_of (const PhonemeExtraction &record, const std::vector< size_t > &rows)
static std::vector< size_t > selected_rows (const PhonemeExtraction &record)

Private Attributes

StandardPhonemeAnalyzer m_phonemes
RegionSpeechSegmenter m_speech
std::optional< StandardWaveformAnalyzer > m_waveform
uint32_t m_waveform_hop { 0 }

Additional Inherited Members

Protected Attributes inherited from MayaFlux::Yantra::ComputeOperation< InputType, OutputType >
ExecutionContext m_last_execution_context

Detailed Description

template<ComputeData InputType = std::vector<Kakshya::DataVariant>, ComputeData OutputType = Kakshya::CompositeArray>
class MayaFlux::Yantra::PhonemeExtractor< InputType, OutputType >

The phonemes of a whole mono 16 kHz recording, with the pauses between speech taken out.

Finds speech with a SpeechSegmenter, then runs the phoneme network once on each speech interval, so no call exceeds its 30 s limit and nothing runs on silence. Phoneme ranges are estimates (PhonemeEvent). Pauses shorter than min_silence stay inside a range.

OutputType selects the output: Kakshya::CompositeArray (records, the default), Kakshya::RegionGroup (regions) or std::vector<Kakshya::DataVariant> (audio of the selected phonemes). Blocking, one channel, not in the default registry. It must not run on the audio thread.

Definition at line 152 of file PhonemeExtractor.hpp.


The documentation for this class was generated from the following file: