When writing a file, data are loaded in chunks and the output bytes are saved in chunks.
The sizes of input and output chunks cna be specified by the user.
Size of an input chunk is currently expressed in the number of rows from a dataset.
Size of an output chunk is expressed in the number of bytes.
Both should be optimized for different shapes of datasets - combinations of different lengths (numbers of rows) and widths (number of columns, number of images, number of columns in images).
When writing a file, data are loaded in chunks and the output bytes are saved in chunks.
The sizes of input and output chunks cna be specified by the user.
Size of an input chunk is currently expressed in the number of rows from a dataset.
Size of an output chunk is expressed in the number of bytes.
Both should be optimized for different shapes of datasets - combinations of different lengths (numbers of rows) and widths (number of columns, number of images, number of columns in images).