MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
Loading...
Searching...
No Matches
MayaFlux::IO::SafetensorsReader Class Reference

Reader for the safetensors format: a JSON index followed by packed tensor bytes. More...

#include <SafetensorsFormat.hpp>

Inheritance diagram for MayaFlux::IO::SafetensorsReader:
Collaboration diagram for MayaFlux::IO::SafetensorsReader:

Classes

struct  Entry

Public Member Functions

bool can_read (const std::string &filepath) const override
 Check if a file can be read by this reader.
void close () override
 Close the currently open file.
std::vector< Kakshya::TensorSpec > describe () const override
 Specs of every tensor in the open file, in file order, without reading data.
std::string format_name () const override
 Format identifier such as safetensors or npy.
std::vector< std::string > get_supported_extensions () const override
 Get supported file extensions for this reader.
bool is_open () const override
 Check if a file is currently open.
bool open (const std::string &filepath, FileReadOptions options=FileReadOptions::ALL) override
 Open a file for reading.
std::optional< Kakshya::TensorData > read (std::string_view name) override
 Read one tensor by name.
 SafetensorsReader ()=default
std::map< std::string, std::string > tensor_metadata () const override
 Free-form string metadata stored in the file, empty if the format has none.
 ~SafetensorsReader () override
Public Member Functions inherited from MayaFlux::IO::TensorReader
std::shared_ptr< Kakshya::SignalSourceContainer > create_container () override
 Create and initialize a container from the file.
std::optional< Kakshya::TensorSpec > find_spec (std::string_view name) const
 Spec of one tensor by name, without reading data.
std::type_index get_container_type () const override
 Get the container type this reader creates.
std::type_index get_data_type () const override
 Get the data type this reader produces.
std::vector< uint64_t > get_dimension_sizes () const override
 Get size of each dimension in the file data.
std::string get_last_error () const override
 Get the last error message.
std::optional< FileMetadata > get_metadata () const override
 Get metadata from the open file.
size_t get_num_dimensions () const override
 Get the dimensionality of the file data.
uint64_t get_preferred_chunk_size () const override
 Get the preferred chunk size for streaming.
std::vector< uint64_t > get_read_position () const override
 Get current read position in primary dimension.
std::vector< FileRegion > get_regions () const override
 Get semantic regions from the file.
bool load_into_container (std::shared_ptr< Kakshya::SignalSourceContainer >) override
 Load file data into an existing container.
std::vector< Kakshya::DataVariant > read_all () override
 Read all data from the file into memory.
std::vector< Kakshya::DataVariant > read_region (const FileRegion &) override
 Read a specific region of data.
std::optional< Kakshya::TensorSet > read_tensors (const TensorReadOptions &options={})
 Read a selection of tensors together with the file's metadata.
bool seek (const std::vector< uint64_t > &) override
 Seek to a specific position in the file.
bool supports_streaming () const override
 Check if streaming is supported for the current file.
 ~TensorReader () override=default
Public Member Functions inherited from MayaFlux::IO::FileReader
virtual ~FileReader ()=default

Static Public Member Functions

static bool probe (std::span< const std::byte > head, uint64_t file_size)
 Whether leading bytes and file size look like a safetensors header.
static void register_with_registry ()
 Register the format, with its writer, in the TensorFormatRegistry.
Static Public Member Functions inherited from MayaFlux::IO::FileReader
static std::string resolve_path (const std::string &filepath)
 Resolve a filepath against the project source root if not found as-is.

Private Member Functions

bool reject (std::string reason)

Private Attributes

uint64_t m_data_start {}
std::vector< Entry > m_entries
std::ifstream m_file
std::map< std::string, std::string > m_metadata
bool m_open {}

Additional Inherited Members

Protected Member Functions inherited from MayaFlux::IO::TensorReader
bool fail (std::string reason)
 Record a failure reason and return false.
Static Protected Member Functions inherited from MayaFlux::IO::FileReader
static std::unordered_map< std::string, Kakshya::RegionGroup > regions_to_groups (const std::vector< FileRegion > &regions)
 Convert file regions to region groups.
Protected Attributes inherited from MayaFlux::IO::TensorReader
std::string m_last_error
std::string m_path

Detailed Description

Reader for the safetensors format: a JSON index followed by packed tensor bytes.

open() parses the index only. Each tensor is then read with one seek, at its native element type, including bfloat16, float8 and sub-byte types. An index entry with an element type this build does not know is skipped with a warning; every other tensor stays readable.

Usage:

reader.open("model.safetensors");
auto bias = reader.read("layer.bias");
bool open(const std::string &filepath, FileReadOptions options=FileReadOptions::ALL) override
Open a file for reading.
std::optional< Kakshya::TensorData > read(std::string_view name) override
Read one tensor by name.

Definition at line 26 of file SafetensorsFormat.hpp.


The documentation for this class was generated from the following files: