MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::IO::TensorReader Class Referenceabstract

Typed FileReader boundary for formats that store named tensors. More...

#include <TensorReader.hpp>

Inheritance diagram for MayaFlux::IO::TensorReader:
Collaboration diagram for MayaFlux::IO::TensorReader:

Public Member Functions

std::shared_ptr< Kakshya::SignalSourceContainer > create_container () override
 Create and initialize a container from the file.
virtual std::vector< Kakshya::TensorSpec > describe () const =0
 Specs of every tensor in the open file, in file order, without reading data.
std::optional< Kakshya::TensorSpec > find_spec (std::string_view name) const
 Spec of one tensor by name, without reading data.
virtual std::string format_name () const =0
 Format identifier such as safetensors or npy.
std::type_index get_container_type () const override
 Get the container type this reader creates.
std::type_index get_data_type () const override
 Get the data type this reader produces.
std::vector< uint64_t > get_dimension_sizes () const override
 Get size of each dimension in the file data.
std::string get_last_error () const override
 Get the last error message.
std::optional< FileMetadata > get_metadata () const override
 Get metadata from the open file.
size_t get_num_dimensions () const override
 Get the dimensionality of the file data.
uint64_t get_preferred_chunk_size () const override
 Get the preferred chunk size for streaming.
std::vector< uint64_t > get_read_position () const override
 Get current read position in primary dimension.
std::vector< FileRegion > get_regions () const override
 Get semantic regions from the file.
bool load_into_container (std::shared_ptr< Kakshya::SignalSourceContainer >) override
 Load file data into an existing container.
virtual std::optional< Kakshya::TensorData > read (std::string_view name)=0
 Read one tensor by name.
std::vector< Kakshya::DataVariant > read_all () override
 Read all data from the file into memory.
std::vector< Kakshya::DataVariant > read_region (const FileRegion &) override
 Read a specific region of data.
std::optional< Kakshya::TensorSet > read_tensors (const TensorReadOptions &options={})
 Read a selection of tensors together with the file's metadata.
bool seek (const std::vector< uint64_t > &) override
 Seek to a specific position in the file.
bool supports_streaming () const override
 Check if streaming is supported for the current file.
virtual std::map< std::string, std::string > tensor_metadata () const
 Free-form string metadata stored in the file, empty if the format has none.
 ~TensorReader () override=default
Public Member Functions inherited from MayaFlux::IO::FileReader
virtual bool can_read (const std::string &filepath) const =0
 Check if a file can be read by this reader.
virtual void close ()=0
 Close the currently open file.
virtual std::vector< std::string > get_supported_extensions () const =0
 Get supported file extensions for this reader.
virtual bool is_open () const =0
 Check if a file is currently open.
virtual bool open (const std::string &filepath, FileReadOptions options=FileReadOptions::ALL)=0
 Open a file for reading.
virtual ~FileReader ()=default

Protected Member Functions

bool fail (std::string reason)
 Record a failure reason and return false.

Protected Attributes

std::string m_last_error
std::string m_path

Additional Inherited Members

Static Public Member Functions inherited from MayaFlux::IO::FileReader
static std::string resolve_path (const std::string &filepath)
 Resolve a filepath against the project source root if not found as-is.
Static Protected Member Functions inherited from MayaFlux::IO::FileReader
static std::unordered_map< std::string, Kakshya::RegionGroup > regions_to_groups (const std::vector< FileRegion > &regions)
 Convert file regions to region groups.

Detailed Description

Typed FileReader boundary for formats that store named tensors.

A concrete reader parses only the file's index on open(), so describe() and tensor_metadata() answer without reading tensor data. read() then loads one tensor at its native element type, with byte order converted to the host's, so a large file can be inspected and read selectively. The inherited DataVariant and container methods are unsupported.

A new format derives from this class, registers a TensorFormat with the TensorFormatRegistry, and nothing else changes.

Usage:

auto reader = io.open_tensor_reader("model.safetensors");
for (const auto& spec : reader->describe()) { }
auto weight = reader->read("encoder.0.weight");
auto everything = reader->read_tensors();

Definition at line 38 of file TensorReader.hpp.


The documentation for this class was generated from the following files: