|
MayaFlux 1.0.0
Digital-First Multimedia Processing Framework
|
Reader for the safetensors format: a JSON index followed by packed tensor bytes. More...
#include <SafetensorsFormat.hpp>
Classes | |
| struct | Entry |
Public Member Functions | |
| bool | can_read (const std::string &filepath) const override |
| Check if a file can be read by this reader. | |
| void | close () override |
| Close the currently open file. | |
| std::vector< Kakshya::TensorSpec > | describe () const override |
| Specs of every tensor in the open file, in file order, without reading data. | |
| std::string | format_name () const override |
| Format identifier such as safetensors or npy. | |
| std::vector< std::string > | get_supported_extensions () const override |
| Get supported file extensions for this reader. | |
| bool | is_open () const override |
| Check if a file is currently open. | |
| bool | open (const std::string &filepath, FileReadOptions options=FileReadOptions::ALL) override |
| Open a file for reading. | |
| std::optional< Kakshya::TensorData > | read (std::string_view name) override |
| Read one tensor by name. | |
| SafetensorsReader ()=default | |
| std::map< std::string, std::string > | tensor_metadata () const override |
| Free-form string metadata stored in the file, empty if the format has none. | |
| ~SafetensorsReader () override | |
| Public Member Functions inherited from MayaFlux::IO::TensorReader | |
| std::shared_ptr< Kakshya::SignalSourceContainer > | create_container () override |
| Create and initialize a container from the file. | |
| std::optional< Kakshya::TensorSpec > | find_spec (std::string_view name) const |
| Spec of one tensor by name, without reading data. | |
| std::type_index | get_container_type () const override |
| Get the container type this reader creates. | |
| std::type_index | get_data_type () const override |
| Get the data type this reader produces. | |
| std::vector< uint64_t > | get_dimension_sizes () const override |
| Get size of each dimension in the file data. | |
| std::string | get_last_error () const override |
| Get the last error message. | |
| std::optional< FileMetadata > | get_metadata () const override |
| Get metadata from the open file. | |
| size_t | get_num_dimensions () const override |
| Get the dimensionality of the file data. | |
| uint64_t | get_preferred_chunk_size () const override |
| Get the preferred chunk size for streaming. | |
| std::vector< uint64_t > | get_read_position () const override |
| Get current read position in primary dimension. | |
| std::vector< FileRegion > | get_regions () const override |
| Get semantic regions from the file. | |
| bool | load_into_container (std::shared_ptr< Kakshya::SignalSourceContainer >) override |
| Load file data into an existing container. | |
| std::vector< Kakshya::DataVariant > | read_all () override |
| Read all data from the file into memory. | |
| std::vector< Kakshya::DataVariant > | read_region (const FileRegion &) override |
| Read a specific region of data. | |
| std::optional< Kakshya::TensorSet > | read_tensors (const TensorReadOptions &options={}) |
| Read a selection of tensors together with the file's metadata. | |
| bool | seek (const std::vector< uint64_t > &) override |
| Seek to a specific position in the file. | |
| bool | supports_streaming () const override |
| Check if streaming is supported for the current file. | |
| ~TensorReader () override=default | |
| Public Member Functions inherited from MayaFlux::IO::FileReader | |
| virtual | ~FileReader ()=default |
Static Public Member Functions | |
| static bool | probe (std::span< const std::byte > head, uint64_t file_size) |
| Whether leading bytes and file size look like a safetensors header. | |
| static void | register_with_registry () |
| Register the format, with its writer, in the TensorFormatRegistry. | |
| Static Public Member Functions inherited from MayaFlux::IO::FileReader | |
| static std::string | resolve_path (const std::string &filepath) |
| Resolve a filepath against the project source root if not found as-is. | |
Private Member Functions | |
| bool | reject (std::string reason) |
Private Attributes | |
| uint64_t | m_data_start {} |
| std::vector< Entry > | m_entries |
| std::ifstream | m_file |
| std::map< std::string, std::string > | m_metadata |
| bool | m_open {} |
Additional Inherited Members | |
| Protected Member Functions inherited from MayaFlux::IO::TensorReader | |
| bool | fail (std::string reason) |
| Record a failure reason and return false. | |
| Static Protected Member Functions inherited from MayaFlux::IO::FileReader | |
| static std::unordered_map< std::string, Kakshya::RegionGroup > | regions_to_groups (const std::vector< FileRegion > ®ions) |
| Convert file regions to region groups. | |
| Protected Attributes inherited from MayaFlux::IO::TensorReader | |
| std::string | m_last_error |
| std::string | m_path |
Reader for the safetensors format: a JSON index followed by packed tensor bytes.
open() parses the index only. Each tensor is then read with one seek, at its native element type, including bfloat16, float8 and sub-byte types. An index entry with an element type this build does not know is skipped with a warning; every other tensor stays readable.
Usage:
Definition at line 26 of file SafetensorsFormat.hpp.