Super-chunk#

This API describes the new Blosc 2 container, the super-chunk (or schunk for short).

struct blosc2_storage#

This struct is meant for holding storage parameters for a for a blosc2 container, allowing to specify, for example, how to interpret the contents included in the schunk.

Public Members

bool contiguous#

Whether the chunks are contiguous or sparse.

char *urlpath#

The path for persistent storage. If NULL, that means in-memory.

blosc2_cparams *cparams#

The compression params when creating a schunk.

If NULL, sensible defaults are used depending on the context.

blosc2_dparams *dparams#

The decompression params when creating a schunk.

If NULL, sensible defaults are used depending on the context.

blosc2_io *io#

Input/output backend.

struct blosc2_schunk#

This struct is the standard container for Blosc 2 compressed data.

This is essentially a container for Blosc 1 chunks of compressed data, and it allows to overcome the 32-bit limitation in Blosc 1. Optionally, a blosc2_frame can be attached so as to store the compressed chunks contiguously.

Public Members

uint8_t compcode#

The default compressor. Each chunk can override this.

uint8_t compcode_meta#

The default compressor metadata. Each chunk can override this.

uint8_t clevel#

The compression level and other compress params.

uint8_t splitmode#

The split mode.

int32_t typesize#

The type size.

int32_t blocksize#

The requested size of the compressed blocks (0; meaning automatic).

int32_t chunksize#

Size of each chunk. 0 if not a fixed chunksize.

uint8_t filters[BLOSC2_MAX_FILTERS]#

The (sequence of) filters. 8-bit per filter.

uint8_t filters_meta[BLOSC2_MAX_FILTERS]#

Metadata for filters. 8-bit per meta-slot.

int64_t nchunks#

Number of chunks in super-chunk.

int64_t current_nchunk#

The current chunk that is being accessed.

int64_t nbytes#

The data size (uncompressed).

int64_t cbytes#

The data size + chunks header size (compressed).

uint8_t **data#

Pointer to chunk data pointers buffer.

size_t data_len#

Length of the chunk data pointers buffer.

blosc2_storage *storage#

Pointer to storage info.

blosc2_frame *frame#

Pointer to frame used as store for chunks.

uint8_t* ctx; Context for the thread holder. NULL if not acquired.

blosc2_context *cctx#

Context for compression.

blosc2_context *dctx#

Context for decompression.

struct blosc2_metalayer *metalayers[16]#

The array of metalayers.

uint16_t nmetalayers#

The number of metalayers in the super-chunk.

int16_t nvlmetalayers#

The number of variable-length metalayers.

void *tuner_params#

Tune configuration.

blosc2_schunk *blosc2_schunk_new(blosc2_storage *storage)#

Create a new super-chunk.

Remark

In case that storage.urlpath is not NULL, the data is stored on-disk. If the data file(s) exist, they are overwritten.

Parameters:

  • storage – The storage properties.

Returns:

The new super-chunk.

int blosc2_schunk_free(blosc2_schunk *schunk)#

Release resources from a super-chunk.

Remark

All the memory resources attached to the super-chunk are freed. If the super-chunk is on-disk, the data continues there for a later re-opening.

Parameters:

  • schunk – The super-chunk to be freed.

Returns:

0 if success.

blosc2_schunk *blosc2_schunk_open(const char *urlpath)#

Open an existing super-chunk that is on-disk (frame).

No in-memory copy is made.

Parameters:

  • urlpath – The file name.

Returns:

The new super-chunk. NULL if not found or not in frame format.

blosc2_schunk *blosc2_schunk_open_offset(const char *urlpath, int64_t offset)#

Open an existing super-chunk that is on-disk (frame).

No in-memory copy is made.

Parameters:

  • urlpath – The file name.

  • offset – The frame offset.

Returns:

The new super-chunk. NULL if not found or not in frame format.

blosc2_schunk *blosc2_schunk_open_udio(const char *urlpath, const blosc2_io *udio)#

Open an existing super-chunk (no copy is made) using a user-defined I/O interface.

Parameters:

  • urlpath – The file name.

  • udio – The user-defined I/O interface.

Returns:

The new super-chunk.

blosc2_schunk *blosc2_schunk_copy(blosc2_schunk *schunk, blosc2_storage *storage)#

Create a copy of a super-chunk.

Parameters:

  • schunk – The super-chunk to be copied.

  • storage – The storage properties.

Returns:

The new super-chunk.

blosc2_schunk *blosc2_schunk_from_buffer(uint8_t *cframe, int64_t len, bool copy)#

Create a super-chunk out of a contiguous frame buffer.

Remark

If copy is false, the cframe buffer passed will be owned by the super-chunk and will be automatically freed when blosc2_schunk_free() is called. If the user frees it after the opening, bad things will happen. Don’t do that (or set copy).

Parameters:

  • cframe – The buffer of the in-memory frame.

  • copy – Whether the super-chunk should make a copy of the cframe data or not. The copy will be made to an internal sparse frame.

  • len – The length of the buffer (in bytes).

Returns:

The new super-chunk.

int64_t blosc2_schunk_to_buffer(blosc2_schunk *schunk, uint8_t **cframe, bool *needs_free)#
int64_t blosc2_schunk_to_file(blosc2_schunk *schunk, const char *urlpath)#
int64_t blosc2_schunk_append_file(blosc2_schunk *schunk, const char *urlpath)#
int blosc2_schunk_get_cparams(blosc2_schunk *schunk, blosc2_cparams **cparams)#

Return the cparams associated to a super-chunk.

Parameters:

  • schunk – The super-chunk from where to extract the compression parameters.

  • cparams – The pointer where the compression params will be returned.

Returns:

0 if succeeds. Else a negative code is returned.

Warning

A new struct is allocated, and the user should free it after use.

int blosc2_schunk_get_dparams(blosc2_schunk *schunk, blosc2_dparams **dparams)#

Return the dparams struct associated to a super-chunk.

Parameters:

  • schunk – The super-chunk from where to extract the decompression parameters.

  • dparams – The pointer where the decompression params will be returned.

Returns:

0 if succeeds. Else a negative code is returned.

Warning

A new struct is allocated, and the user should free it after use.

int blosc2_schunk_reorder_offsets(blosc2_schunk *schunk, int64_t *offsets_order)#

Reorder the chunk offsets of an existing super-chunk.

Parameters:

  • schunk – The super-chunk whose chunk offsets are to be reordered.

  • offsets_order – The new order of the chunk offsets.

Returns:

0 if succeeds. Else a negative code is returned.

int64_t blosc2_schunk_frame_len(blosc2_schunk *schunk)#

Get the length (in bytes) of the internal frame of the super-chunk.

Parameters:

  • schunk – The super-chunk.

Returns:

The length (in bytes) of the internal frame. If there is not an internal frame, an estimate of the length is provided.

int64_t blosc2_schunk_fill_special(blosc2_schunk *schunk, int64_t nitems, int special_value, int32_t chunksize)#

Quickly fill an empty frame with special values (zeros, NaNs, uninit).

Parameters:

  • schunk – The super-chunk to be filled. This must be empty initially.

  • nitems – The number of items to fill.

  • special_value – The special value to use for filling. The only values supported for now are BLOSC2_SPECIAL_ZERO, BLOSC2_SPECIAL_NAN and BLOSC2_SPECIAL_UNINIT.

  • chunksize – The chunksize for the chunks that are to be added to the super-chunk.

Returns:

The total number of chunks that have been added to the super-chunk. If there is an error, a negative value is returned.

int64_t blosc2_schunk_append_buffer(blosc2_schunk *schunk, const void *src, int32_t nbytes)#

Append a src data buffer to a super-chunk.

Parameters:

  • schunk – The super-chunk where data will be appended.

  • src – The buffer of data to compress.

  • nbytes – The size of the src buffer.

Returns:

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int blosc2_schunk_get_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer)#

Fill buffer with a schunk slice.

Parameters:

  • schunk – The super-chunk from where to extract a slice.

  • start – Index (0-based) where the slice begins.

  • stop – The first index (0-based) that is not in the selected slice.

  • buffer – The buffer where the data will be stored.

Returns:

An error code.

Warning

You must make sure that you have enough space in buffer to store the uncompressed data.

int blosc2_schunk_set_slice_buffer(blosc2_schunk *schunk, int64_t start, int64_t stop, void *buffer)#

Update a schunk slice from buffer.

Parameters:

  • schunk – The super-chunk where to set the slice.

  • start – Index (0-based) where the slice begins.

  • stop – The first index (0-based) that is not in the selected slice.

  • buffer – The buffer containing the data to set.

Returns:

An error code.

void blosc2_schunk_avoid_cframe_free(blosc2_schunk *schunk, bool avoid_cframe_free)#

Set the private avoid_cframe_free field in a frame.

Parameters:

  • schunk – The super-chunk referencing the frame.

  • avoid_cframe_free – The value to set in the blosc2_frame_s structure.

Warning

If you set it to true you will be responsible of freeing it.

Dealing with chunks#

int blosc2_schunk_get_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free)#

Return a compressed chunk that is part of a super-chunk in the chunk parameter.

Parameters:

  • schunk – The super-chunk from where to extract a chunk.

  • nchunk – The chunk to be extracted (0 indexed).

  • chunk – The pointer to the chunk of compressed data.

  • needs_free – The pointer to a boolean indicating if it is the user’s responsibility to free the chunk returned or not.

Returns:

The size of the (compressed) chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead.

Warning

If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the (compressed) chunk, and hence a free is needed. You can check whether the chunk requires a free with the needs_free parameter. If the chunk does not need a free, it means that a pointer to the location in the super-chunk (or the backing in-memory frame) is returned in the chunk parameter.

int blosc2_schunk_get_lazychunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t **chunk, bool *needs_free)#

Return a (lazy) compressed chunk that is part of a super-chunk in the chunk parameter.

Parameters:

  • schunk – The super-chunk from where to extract a chunk.

  • nchunk – The chunk to be extracted (0 indexed).

  • chunk – The pointer to the (lazy) chunk of compressed data.

  • needs_free – The pointer to a boolean indicating if it is the user’s responsibility to free the chunk returned or not.

Returns:

The size of the (compressed) chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead. Note that a lazy chunk is somewhat larger than a regular chunk because of the trailer section (for details see README_CHUNK_FORMAT.rst).

Note

For disk-based frames, a lazy chunk is always returned.

Warning

Currently, a lazy chunk can only be used by blosc2_decompress_ctx and blosc2_getitem_ctx.

Warning

If the super-chunk is backed by a frame that is disk-based, a buffer is allocated for the (compressed) chunk, and hence a free is needed. You can check whether requires a free with the needs_free parameter. If the chunk does not need a free, it means that a pointer to the location in the super-chunk (or the backing in-memory frame) is returned in the chunk parameter. In this case the returned chunk is not lazy.

int blosc2_schunk_decompress_chunk(blosc2_schunk *schunk, int64_t nchunk, void *dest, int32_t nbytes)#

Decompress and return the nchunk chunk of a super-chunk.

If the chunk is uncompressed successfully, it is put in the *dest pointer.

Parameters:

  • schunk – The super-chunk from where the chunk will be decompressed.

  • nchunk – The chunk to be decompressed (0 indexed).

  • dest – The buffer where the decompressed data will be put.

  • nbytes – The size of the area pointed by *dest.

Returns:

The size of the decompressed chunk or 0 if it is non-initialized. If some problem is detected, a negative code is returned instead.

Warning

You must make sure that you have enough space to store the uncompressed data.

int64_t blosc2_schunk_append_chunk(blosc2_schunk *schunk, uint8_t *chunk, bool copy)#

Append an existing chunk to a super-chunk.

Parameters:

  • schunk – The super-chunk where the chunk will be appended.

  • chunk – The chunk to append. An internal copy is made, so chunk can be reused or freed if desired.

  • copy – Whether the chunk should be copied internally or can be used as-is.

Returns:

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_insert_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy)#

Insert a chunk at a specific position in a super-chunk.

Parameters:

  • schunk – The super-chunk where the chunk will be appended.

  • nchunk – The position where the chunk will be inserted.

  • chunk – The chunk to insert. If an internal copy is made, the chunk can be reused or freed if desired.

  • copy – Whether the chunk should be copied internally or can be used as-is.

Returns:

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_update_chunk(blosc2_schunk *schunk, int64_t nchunk, uint8_t *chunk, bool copy)#

Update a chunk at a specific position in a super-chunk.

Parameters:

  • schunk – The super-chunk where the chunk will be updated.

  • nchunk – The position where the chunk will be updated.

  • chunk – The new chunk. If an internal copy is made, the chunk can be reused or freed if desired.

  • copy – Whether the chunk should be copied internally or can be used as-is.

Returns:

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

int64_t blosc2_schunk_delete_chunk(blosc2_schunk *schunk, int64_t nchunk)#

Delete a chunk at a specific position in a super-chunk.

Parameters:

  • schunk – The super-chunk where the chunk will be deleted.

  • nchunk – The position where the chunk will be deleted.

Returns:

The number of chunks in super-chunk. If some problem is detected, this number will be negative.

Creating chunks#

int blosc2_chunk_zeros(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of zeros.

Parameters:

  • cparams – The compression parameters.

  • nbytes – The size (in bytes) of the chunk.

  • dest – The buffer where the data chunk will be put.

  • destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns:

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

int blosc2_chunk_nans(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of nans.

Parameters:

  • cparams – The compression parameters; only 4 bytes (float) and 8 bytes (double) are supported.

  • nbytes – The size (in bytes) of the chunk.

  • dest – The buffer where the data chunk will be put.

  • destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns:

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

Note

Whether the NaNs are floats or doubles will be given by the typesize.

int blosc2_chunk_repeatval(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize, const void *repeatval)#

Create a chunk made of repeated values.

Parameters:

  • cparams – The compression parameters.

  • nbytes – The size (in bytes) of the chunk.

  • dest – The buffer where the data chunk will be put.

  • destsize – The size (in bytes) of the dest buffer.

  • repeatval – A pointer to the repeated value (little endian). The size of the value is given by cparams.typesize param.

Returns:

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH + typesize). If negative, there has been an error and dest is unusable.

int blosc2_chunk_uninit(blosc2_cparams cparams, int32_t nbytes, void *dest, int32_t destsize)#

Create a chunk made of uninitialized values.

Parameters:

  • cparams – The compression parameters.

  • nbytes – The size (in bytes) of the chunk.

  • dest – The buffer where the data chunk will be put.

  • destsize – The size (in bytes) of the dest buffer; must be BLOSC_EXTENDED_HEADER_LENGTH at least.

Returns:

The number of bytes compressed (BLOSC_EXTENDED_HEADER_LENGTH). If negative, there has been an error and dest is unusable.

Frame specific functions#

int64_t *blosc2_frame_get_offsets(blosc2_schunk *schunk)#

Get the offsets of a frame in a super-chunk.

Parameters:

  • schunk – The super-chunk containing the frame.

Returns:

If successful, return a pointer to a buffer of the decompressed offsets. The number of offsets is equal to schunk->nchunks; the user is responsible to free this buffer. Else, return a NULL value.