Super-chunk#

This API describes the new Blosc 2 container, the super-chunk (or schunk for short).

struct blosc2_storage#

This struct is meant for holding storage parameters for a for a blosc2 container, allowing to specify, for example, how to interpret the contents included in the schunk.

Public Members

bool contiguous#: Whether the chunks are contiguous or sparse.

char *urlpath#: The path for persistent storage. If NULL, that means in-memory.

blosc2_cparams *cparams#

The compression params when creating a schunk.

If NULL, sensible defaults are used depending on the context.

blosc2_dparams *dparams#

The decompression params when creating a schunk.

If NULL, sensible defaults are used depending on the context.

blosc2_io *io#: Input/output backend.

struct blosc2_schunk#

This struct is the standard container for Blosc 2 compressed data.

This is essentially a container for Blosc 1 chunks of compressed data, and it allows to overcome the 32-bit limitation in Blosc 1. Optionally, a blosc2_frame can be attached so as to store the compressed chunks contiguously.

Public Members

uint8_t compcode#: The default compressor. Each chunk can override this.

uint8_t compcode_meta#: The default compressor metadata. Each chunk can override this.

uint8_t clevel#: The compression level and other compress params.

uint8_t splitmode#: The split mode.

int32_t typesize#: The type size.

int32_t blocksize#: The requested size of the compressed blocks (0; meaning automatic).

int32_t chunksize#: Size of each chunk. 0 if not a fixed chunksize.

uint8_t filters[BLOSC2_MAX_FILTERS]#: The (sequence of) filters. 8-bit per filter.

uint8_t filters_meta[BLOSC2_MAX_FILTERS]#: Metadata for filters. 8-bit per meta-slot.

int64_t nchunks#: Number of chunks in super-chunk.

int64_t current_nchunk#: The current chunk that is being accessed.

int64_t nbytes#: The data size (uncompressed).

int64_t cbytes#: The data size + chunks header size (compressed).

uint8_t **data#: Pointer to chunk data pointers buffer.

size_t data_len#: Length of the chunk data pointers buffer.

blosc2_storage *storage#: Pointer to storage info.

blosc2_frame *frame#

Pointer to frame used as store for chunks.

uint8_t* ctx; Context for the thread holder. NULL if not acquired.

blosc2_context *cctx#: Context for compression.

blosc2_context *dctx#: Context for decompression.

struct blosc2_metalayer *metalayers[16]#: The array of metalayers.

uint16_t nmetalayers#: The number of metalayers in the super-chunk.

int16_t nvlmetalayers#: The number of variable-length metalayers.

void *tuner_params#: Tune configuration.

blosc2_schunk *blosc2_schunk_new(blosc2_storage *storage)#

Create a new super-chunk.

Remark

In case that storage.urlpath is not NULL, the data is stored on-disk. If the data file(s) exist, they are overwritten.

Parameters

storage – The storage properties.

Returns

The new super-chunk.

int blosc2_schunk_free(blosc2_schunk *schunk)#

Release resources from a super-chunk.

Remark

All the memory resources attached to the super-chunk are freed. If the super-chunk is on-disk, the data continues there for a later re-opening.

Parameters

schunk – The super-chunk to be freed.

Returns

0 if success.

blosc2_schunk *blosc2_schunk_open(const char *urlpath)#

Open an existing super-chunk that is on-disk (frame).

No in-memory copy is made.

Parameters

urlpath – The file name.

Returns

The new super-chunk. NULL if not found or not in frame format.

blosc2_schunk *blosc2_schunk_open_offset(const char *urlpath, int64_t offset)#

Open an existing super-chunk that is on-disk (frame).

No in-memory copy is made.

Parameters

urlpath – The file name.
offset – The frame offset.

Returns

The new super-chunk. NULL if not found or not in frame format.

blosc2_schunk *blosc2_schunk_open_udio(const char *urlpath, const blosc2_io *udio)#

Open an existing super-chunk (no copy is made) using a user-defined I/O interface.

Parameters

urlpath – The file name.
udio – The user-defined I/O interface.

Returns

The new super-chunk.

blosc2_schunk *blosc2_schunk_copy(blosc2_schunk *schunk, blosc2_storage *storage)#

Create a copy of a super-chunk.

Parameters

schunk – The super-chunk to be copied.
storage – The storage properties.

Returns

The new super-chunk.

blosc2_schunk *blosc2_schunk_from_buffer(uint8_t *cframe, int64_t len, bool copy)#

Create a super-chunk out of a contiguous frame buffer.

Remark

If copy is false, the cframe buffer passed will be owned by the super-chunk and will be automatically freed when blosc2_schunk_free() is called. If the user frees it after the opening, bad things will happen. Don’t do that (or set copy).

Parameters

cframe – The buffer of the in-memory frame.
copy – Whether the super-chunk should make a copy of the cframe data or not. The copy will be made to an internal sparse frame.
len – The length of the buffer (in bytes).

Returns

The new super-chunk.

int64_t blosc2_schunk_to_buffer(blosc2_schunk *schunk, uint8_t **cframe, bool *needs_free)#

int64_t blosc2_schunk_to_file(blosc2_schunk *schunk, const char *urlpath)#

int64_t blosc2_schunk_append_file(blosc2_schunk *schunk, const char *urlpath)#

int blosc2_schunk_get_cparams(blosc2_schunk *schunk, blosc2_cparams **cparams)#

Return the cparams associated to a super-chunk.

Parameters

schunk – The super-chunk from where to extract the compression parameters.
cparams – The pointer where the compression params will be returned.

Returns

0 if succeeds. Else a negative code is returned.

Warning

A new struct is allocated, and the user should free it after use.

int blosc2_schunk_get_dparams(blosc2_schunk *schunk, blosc2_dparams **dparams)#

Return the dparams struct associated to a super-chunk.

Parameters

schunk – The super-chunk from where to extract the decompression parameters.
dparams – The pointer where the decompression params will be returned.

Returns

0 if succeeds. Else a negative code is returned.

Warning

A new struct is allocated, and the user should free it after use.

int blosc2_schunk_reorder_offsets(blosc2_schunk *schunk, int64_t *offsets_order)#

Reorder the chunk offsets of an existing super-chunk.

Parameters

schunk – The super-chunk whose chunk offsets are to be reordered.
offsets_order – The new order of the chunk offsets.

Returns

0 if succeeds. Else a negative code is returned.

int64_t blosc2_schunk_frame_len(blosc2_schunk *schunk)#

Get the length (in bytes) of the internal frame of the super-chunk.

Parameters

schunk – The super-chunk.

Returns

The length (in bytes) of the internal frame. If there is not an internal frame, an estimate of the length is provided.

int64_t blosc2_schunk_fill_special(blosc2_schunk *schunk, int64_t nitems, int special_value, int32_t chunksize)#

Quickly fill an empty frame with special values (zeros, NaNs, uninit).

Parameters

schunk – The super-chunk to be filled. This must be empty initially.
nitems – The number of items to fill.
special_value – The special value to use for filling. The only values supported for now are BLOSC2_SPECIAL_ZERO, BLOSC2_SPECIAL_NAN and BLOSC2_SPECIAL_UNINIT.
chunksize – The chunksize for the chunks that are to be added to the super-chunk.

Returns

The total number of chunks that have been added to the super-chunk. If there is an error, a negative value is returned.

int64_t blosc2_schunk_append_buffer(blosc2_schunk *schunk, const void *src, int32_t nbytes)#

Append a src data buffer to a super-chunk.

Parameters

schunk – The super-chunk where data will be appended.
src – The buffer of data to compress.
nbytes – The size of the src buffer.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int blosc2_schunk_get_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer)#

Fill buffer with a schunk slice.

Parameters

schunk – The super-chunk from where to extract a slice.
start – Index (0-based) where the slice begins.
stop – The first index (0-based) that is not in the selected slice.
buffer – The buffer where the data will be stored.

Returns

An error code.

Warning

You must make sure that you have enough space in buffer to store the uncompressed data.

int blosc2_schunk_set_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer)#

Update a schunk slice from buffer.

Parameters

schunk – The super-chunk where to set the slice.
start – Index (0-based) where the slice begins.
stop – The first index (0-based) that is not in the selected slice.
buffer – The buffer containing the data to set.

Returns

An error code.

void blosc2_schunk_avoid_cframe_free(blosc2_schunk *schunk, bool avoid_cframe_free)#

Set the private avoid_cframe_free field in a frame.

Parameters

schunk – The super-chunk referencing the frame.
avoid_cframe_free – The value to set in the blosc2_frame_s structure.

Warning

If you set it to true you will be responsible of freeing it.

Dealing with chunks#

int blosc2_schunk_get_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free)#

Return a compressed chunk that is part of a super-chunk in the chunk parameter.

Parameters

schunk – The super-chunk from where to extract a chunk.
nchunk – The chunk to be extracted (0 indexed).
chunk – The pointer to the chunk of compressed data.
needs_free – The pointer to a boolean indicating if it is the user’s responsibility to free the chunk returned or not.

Returns

The size of the (compressed) chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead.

Warning

If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the (compressed) chunk, and hence a free is needed. You can check whether the chunk requires a free with the needs_free parameter. If the chunk does not need a free, it means that a pointer to the location in the super-chunk (or the backing in-memory frame) is returned in the chunk parameter.

int blosc2_schunk_get_lazychunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free)#

Return a (lazy) compressed chunk that is part of a super-chunk in the chunk parameter.

Parameters

schunk – The super-chunk from where to extract a chunk.
nchunk – The chunk to be extracted (0 indexed).
chunk – The pointer to the (lazy) chunk of compressed data.
needs_free – The pointer to a boolean indicating if it is the user’s responsibility to free the chunk returned or not.

Returns

The size of the (compressed) chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead. Note that a lazy chunk is somewhat larger than a regular chunk because of the trailer section (for details see README_CHUNK_FORMAT.rst).

Note

For disk-based frames, a lazy chunk is always returned.

Warning

Currently, a lazy chunk can only be used by blosc2_decompress_ctx and blosc2_getitem_ctx.

Warning

If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the (compressed) chunk, and hence a free is needed. You can check whether requires a free with the needs_free parameter. If the chunk does not need a free, it means that a pointer to the location in the super-chunk (or the backing in-memory frame) is returned in the chunk parameter. In this case the returned chunk is not lazy.

int blosc2_schunk_decompress_chunk(blosc2_schunk *schunk, int64_t nchunk, void *dest, int32_t nbytes)#

Decompress and return the nchunk chunk of a super-chunk.

If the chunk is uncompressed successfully, it is put in the *dest pointer.

Parameters

schunk – The super-chunk from where the chunk will be decompressed.
nchunk – The chunk to be decompressed (0 indexed).
dest – The buffer where the decompressed data will be put.
nbytes – The size of the area pointed by *dest.

Returns

The size of the decompressed chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead.

Warning

You must make sure that you have enough space to store the uncompressed data.

int64_t blosc2_schunk_append_chunk(blosc2_schunk *schunk, uint8_t *chunk, bool copy)#

Append an existing chunk to a super-chunk.

Parameters

schunk – The super-chunk where the chunk will be appended.
chunk – The chunk to append. An internal copy is made, so chunk can be reused or freed if desired.
copy – Whether the chunk should be copied internally or can be used as-is.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_insert_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy)#

Insert a chunk at a specific position in a super-chunk.

Parameters

schunk – The super-chunk where the chunk will be appended.
nchunk – The position where the chunk will be inserted.
chunk – The chunk to insert. If an internal copy is made, the chunk can be reused or freed if desired.
copy – Whether the chunk should be copied internally or can be used as-is.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_update_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy)#

Update a chunk at a specific position in a super-chunk.

Parameters

schunk – The super-chunk where the chunk will be updated.
nchunk – The position where the chunk will be updated.
chunk – The new chunk. If an internal copy is made, the chunk can be reused or freed if desired.
copy – Whether the chunk should be copied internally or can be used as-is.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_delete_chunk(blosc2_schunk *schunk, int64_t nchunk)#

Delete a chunk at a specific position in a super-chunk.

Parameters

schunk – The super-chunk where the chunk will be deleted.
nchunk – The position where the chunk will be deleted.

Returns

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

Creating chunks#

int blosc2_chunk_zeros(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of zeros.

Parameters

cparams – The compression parameters.
nbytes – The size (in bytes) of the chunk.
dest – The buffer where the data chunk will be put.
destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

int blosc2_chunk_nans(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of nans.

Parameters

cparams – The compression parameters; only 4 bytes (float) and 8 bytes (double) are supported.
nbytes – The size (in bytes) of the chunk.
dest – The buffer where the data chunk will be put.
destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

Note

Whether the NaNs are floats or doubles will be given by the typesize.

int blosc2_chunk_repeatval(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize, const void *repeatval)#

Create a chunk made of repeated values.

Parameters

cparams – The compression parameters.
nbytes – The size (in bytes) of the chunk.
dest – The buffer where the data chunk will be put.
destsize – The size (in bytes) of the dest buffer.
repeatval – A pointer to the repeated value (little endian). The size of the value is given by cparams.typesize param.

Returns

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH + typesize). If negative, there has been an error and dest is unusable.

int blosc2_chunk_uninit(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of uninitialized values.

Parameters

cparams – The compression parameters.
nbytes – The size (in bytes) of the chunk.
dest – The buffer where the data chunk will be put.
destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

Frame specific functions#

int64_t *blosc2_frame_get_offsets(blosc2_schunk *schunk)#

Get the offsets of a frame in a super-chunk.

Parameters

schunk – The super-chunk containing the frame.

Returns

If successful, return a pointer to a buffer of the decompressed offsets. The number of offsets is equal to schunk->nchunks; the user is responsible to free this buffer. Else, return a NULL value.