sketchgraphs.data.flat_array.FlatSerializedArray

class sketchgraphs.data.flat_array.FlatSerializedArray(offsets, pickle_data)

This class implements a flat pickle-serialized array of python objects backed by a byte buffer.

This class behaves as a standard immutable list of python objects, but materializes a new object on each access. This allows it to use memory-mapped byte arrays as its underlying storage, which enable random access and arrays larger than system memory.

__init__(offsets, pickle_data)

Initializes a new array from offsets and pickled data.

Note: users will typically wish to create a new instance of this class using the static method from_flat_array, which provides a view into the flat array.

Parameters

• offsets (array_like) – an array of offsets into the data representing each object.

• pickle_data (array_like) – an array of bytes representing the serialized data for all objects.

static from_flat_array(byte_data)

Creates a new FlatSerializedArray from the provided bytes.

Parameters

byte_data (array_like) – A bytes-like object which provides the underlying storage for flat array of objects. This is usually obtained through save_list_flat.

Returns

An instance of FlatSerializedArray providing a view into the data given by the provided bytes.

Return type

FlatSerializedArray

share_memory_()

Moves the underlying to a torch shared memory location.

Calling this method will transition the storage to torch tensors, and move them to a shared memory location. This will enable sharing the tensors across different processes without copying when using torch.multiprocessing.