Basics: compressing data with the SChunk class

Python-Blosc2 is a thin wrapper for the C-Blosc2 format and compression library. It allows to easily and quickly create, append, insert, update and delete data and metadata in a super-chunk container (SChunk class).

import blosc2
import numpy as np

Create a new SChunk instance

Let’s configure the parameters that are different from defaults:

cparams = {
    "codec": blosc2.Codec.BLOSCLZ,
    "typesize": 4,
    "nthreads": 8,
}

dparams = {
    "nthreads": 16,
}

storage = {
    "contiguous": True,
    "urlpath": "myfile.b2frame",
    "mode": "w",  # create a new file
    "cparams": cparams,
    "dparams": dparams,
}

Now, we can already create a SChunk instance:

schunk = blosc2.SChunk(chunksize=10_000_000, **storage)
schunk

<blosc2.schunk.SChunk at 0x7f9d94a0bc80>

Great! So you have created your first super-chunk with your desired compression codec and typesize, that is going to be persistent on-disk.

Append and read data

We are going to add some data. First, let’s create the dataset (4 MB):

buffer = [i * np.arange(2_500_000, dtype="int32") for i in range(100)]

%%time
for i in range(100):
    nchunks = schunk.append_data(buffer[i])
    assert nchunks == (i + 1)

CPU times: user 737 ms, sys: 223 ms, total: 959 ms
Wall time: 680 ms

!ls -lh myfile.b2frame

-rw-r--r--  1 martaiborra  staff    53M Jun 20 10:57 myfile.b2frame

So, while we have added 100 chunks of 10 MB each, the data size of the frame on-disk is a little above 10 MB. This is how compression is helping you to use less resources.

Now, let’s read the chunks from disk:

dest = np.empty(2_500_000, dtype="int32")

%%time
for i in range(100):
    chunk = schunk.decompress_chunk(i, dest)

CPU times: user 325 ms, sys: 268 ms, total: 594 ms
Wall time: 277 ms

check = 99 * np.arange(2_500_000, dtype="int32")
np.testing.assert_equal(dest, check)

Updating and inserting

First, let’s update the first chunk:

data_up = np.arange(2_500_000, dtype="int32")
chunk = blosc2.compress2(data_up)

%%time
schunk.update_chunk(nchunk=0, chunk=chunk)

CPU times: user 305 µs, sys: 1.14 ms, total: 1.45 ms
Wall time: 1.36 ms

100

And then, insert another one at position 4:

%%time
schunk.insert_chunk(nchunk=4, chunk=chunk)

CPU times: user 255 µs, sys: 1.13 ms, total: 1.38 ms
Wall time: 2.54 ms

101

In this case the return value is the new number of chunks in the super-chunk.

Add user meta info

In Blosc2 there are to kind of meta information that you can add to a SChunk. One must be added during the creation of it, cannot be deleted and must always have the same bytes size. This one is known as meta, and works like a dictionary.

schunk = blosc2.SChunk(meta={"meta1": 234})
schunk.meta.keys()

['meta1']

schunk.meta["meta1"]

234

schunk.meta["meta1"] = 235
schunk.meta["meta1"]

235

The other one is known as vlmeta, which stands for “variable length metadata”, and, as the name suggests, it is meant to store general, variable length data (incidentally, this is more flexible than what you can store as regular data, which is always the same typesize). You can add an entry after the creation of the SChunk, update it with a different bytes size value or delete it.

vlmeta follows the dictionary interface, so adding info is as easy as:

schunk.vlmeta["info1"] = "This is an example"
schunk.vlmeta["info2"] = "of user meta handling"
schunk.vlmeta.getall()

{b'info1': 'This is an example', b'info2': 'of user meta handling'}

You can also delete an entry as you would do with a dictionary:

del schunk.vlmeta["info1"]
schunk.vlmeta.getall()

{b'info2': 'of user meta handling'}

That’s all for now. There are more examples in the examples directory of the git repository for you to explore. Enjoy!