C2Array¶
This is a class for remote arrays. This kind of array can also work as operand on a LazyExpr, LazyUDF or reduction.
Wrapped in a Proxy, a stored remote array is read at block granularity:
the proxy asks for the blocks a slice touches rather than the chunks they live
in, which for a multi-megabyte chunk is a small fraction of the bytes. That
rests on the server serving the dataset from a file, Range header and
auth cookie both honoured; a dataset it computes instead (a lazy expression, an
HDF5 leaf) is fetched a whole chunk at a time, as everything was before. Which
one this is takes at most one request to find out, and is decided once –
C2Array.block_source() is what answers it.
A stored remote array can also be filled, by as many writers at once as it has
chunks. The array is laid out first – blosc2.uninit writes a couple of
hundred bytes whatever its size – and then each writer posts the chunks it owns
with C2Array.update_chunk(). A slot nothing was written to is free, and a
write claims it; a second write to the same slot raises
blosc2.ChunkAlreadyWritten, so two writers that both believe they own a
chunk are resolved by the array rather than by anything either of them holds.
C2Array.written_chunks() reads how far the fill has got out of the frame’s
own offsets, which is a couple of range reads and no endpoint of its own.
- class blosc2.C2Array(path: str, /, urlbase: str | None = None, auth_token: str | None = None)[source]¶
Remote compressed NDArray accessed from a Caterva2 server.
- Attributes:
blocksThe blocks of the remote array
blocks_per_chunkHow many blocks a chunk of the remote array holds.
blocksizeThe block size (in bytes) for the remote container.
cbytesThe number of compressed bytes of the remote array
chunksThe chunks of the remote array
cparamsThe compression parameters of the remote array
cratioThe compression ratio of the remote array
deviceHardware device the array data resides on.
dtypeThe dtype of the remote array
infoPrint information about this remote array.
info_itemsA list of tuples with the information about the remote array.
max_rangesHow many ranges one request to this server may carry.
nbytesThe number of bytes of the remote array
ndimGet the number of dimensions of the
Operand.serves_blocksWhether blocks are worth asking this dataset for, as far as info can say.
shapeThe shape of the remote array
stampWhat names the exact remote bytes, for a Proxy to check a cache by.
vlmetaThe variable-length metadata of the remote array.
Methods
aclose()Close the underlying async HTTP client opened by
aget_chunk(), if any.aget_chunk(nchunk)Get the compressed unidimensional chunk of a C2Array asynchronously.
argmax([axis, keepdims])Returns the indices of the maximum values along a specified axis.
argmin([axis, keepdims])Returns the indices of the minimum values along a specified axis.
aupdate_chunk(nchunk, chunk)Write one compressed chunk asynchronously; see
update_chunk().block_plan(nchunk, nblocks)The range reads covering nblocks; see
ByteRangeNDSource.block_plan().The frame reader behind the block methods, or None if there is none.
chunk_layout(nchunk)Where the blocks of a chunk are; see
ByteRangeNDSource.chunk_layout().chunk_layouts(nchunks)The same for several chunks; see
ByteRangeNDSource.chunk_layouts().cumulative_prod([axis, dtype, include_initial])Calculates the cumulative product of elements in the input array ndarr.
cumulative_sum([axis, dtype, include_initial])Calculates the cumulative sum of elements in the input array ndarr.
get_chunk(nchunk)Get the compressed unidimensional chunk of a C2Array.
item()Copy an element of an array to a standard Python scalar and return it.
read_range(offset, size)The bytes at [offset, offset + size) of the remote frame.
read_ranges(spans)The bytes of every span, in one request where the server allows it.
Look at the array again, so that
stampspeaks for it now.save(urlpath[, contiguous])Persist the remote array reference using a CFrame-backed carrier.
slice(slice_)Get a slice of the array (returning blosc2 NDArray array).
Serialize the remote array reference as a CFrame-backed Blosc2 object.
to_device(device)Copy the array from the device on which it currently resides to the specified device.
update_chunk(nchunk, chunk)Write one compressed chunk into a slot of the remote array.
wants_blocks(nchunk, nwanted[, wave])Whether fetching nwanted blocks of a chunk beats fetching all of it.
where([value1, value2])Select
value1orvalue2values based onTrue/Falseforself.Which chunks of the remote array hold content; see
ByteRangeNDSource.written_chunks().- Special Methods:
__init__(path, /[, urlbase, auth_token])Create an instance of a remote NDArray.
__getitem__(slice_)Get a slice of the array (returning NumPy array).
Constructor¶
- __init__(path: str, /, urlbase: str | None = None, auth_token: str | None = None)[source]¶
Create an instance of a remote NDArray.
Remote NDArrays can be accessed via HTTP from a Caterva2 server (e.g., https://cat2.cloud). More information about Caterva2 at: https://ironarray.io/caterva2.
- Parameters:
- Returns:
out
- Return type:
Examples
>>> import blosc2 >>> urlbase = "https://cat2.cloud/demo" >>> path = "@public/examples/dir1/ds-3d.b2nd" >>> remote_array = blosc2.C2Array(path, urlbase=urlbase) >>> remote_array.shape (3, 4, 5) >>> remote_array.chunks (2, 3, 4) >>> remote_array.blocks (2, 2, 2) >>> remote_array.dtype dtype('float32')
Utility Methods¶
- __getitem__(slice_: int | slice | tuple | Sequence[int] | np.ndarray) np.ndarray[source]¶
Get a slice of the array (returning NumPy array).
- Parameters:
slice_¶ (int, slice, tuple of ints and slices, sequence of ints, or ndarray) – The slice to fetch. A sequence of integers or an integer or boolean array gathers those coordinates, as numpy reads them. A list of slices is not a key – numpy stopped reading one as a tuple – and raises IndexError rather than being read as something else.
- Returns:
out – A numpy.ndarray containing the data slice.
- Return type:
numpy.ndarray
Examples
>>> import blosc2 >>> urlbase = "https://cat2.cloud/demo" >>> path = "@public/examples/dir1/ds-2d.b2nd" >>> remote_array = blosc2.C2Array(path, urlbase=urlbase) >>> data_slice = remote_array[3:5, 1:4] >>> data_slice.shape (2, 3) >>> data_slice[:] array([[61, 62, 63], [81, 82, 83]], dtype=uint16)
- async aclose() None[source]¶
Close the underlying async HTTP client opened by
aget_chunk(), if any.
- async aget_chunk(nchunk: int) bytes[source]¶
Get the compressed unidimensional chunk of a C2Array asynchronously.
Same as
get_chunk(), but performs the HTTP GET with anhttpx.AsyncClientinstead of blocking the event loop. Used byProxy.afetch()to fetch multiple chunks concurrently. The underlying client is created lazily and reused across calls; close it explicitly withaclose()when done, e.g. when the event loop is about to be torn down.
- async aupdate_chunk(nchunk: int, chunk: bytes) dict[source]¶
Write one compressed chunk asynchronously; see
update_chunk().The same request, off the event loop, so a writer with many chunks to send can have several in flight. The server serializes them at the far end regardless – what overlaps is the round trip, which for a chunk-sized body is most of the cost.
- block_plan(nchunk: int, nblocks: Sequence[int]) list[tuple[int, int, tuple]][source]¶
The range reads covering nblocks; see
ByteRangeNDSource.block_plan().
- block_source() C2NDSource | None[source]¶
The frame reader behind the block methods, or None if there is none.
Built on the first request for it and never rebuilt. The fallback has to be permanent: a server that streams this dataset answers a range request with the whole body, so retrying would pay a full download to rediscover the same answer.
Every stored frame says yes: whether a given chunk of it is worth taking apart is decided per fetch, by wants_blocks, and not here. So this and
_index_source()now come to the same answer, and both remain because they ask for different reasons – one for the blocks of a chunk, one for the offsets that say where the chunks are. Neither remembers a no, since a dataset laid out empty becomes a stored one as it is filled.
- chunk_layout(nchunk: int)[source]¶
Where the blocks of a chunk are; see
ByteRangeNDSource.chunk_layout().
- chunk_layouts(nchunks: Sequence[int]) list[source]¶
The same for several chunks; see
ByteRangeNDSource.chunk_layouts().
- get_chunk(nchunk: int) bytes[source]¶
Get the compressed unidimensional chunk of a C2Array.
- Parameters:
nchunk¶ (int) – The index of the unidimensional chunk to retrieve.
- Returns:
out – The requested compressed chunk.
- Return type:
Examples
>>> import numpy as np >>> import blosc2 >>> urlbase = "https://cat2.cloud/demo" >>> path = "@public/examples/dir1/ds-3d.b2nd" >>> a = blosc2.C2Array(path, urlbase) >>> # Get the compressed chunk from array 'a' for index 0 >>> compressed_chunk = a.get_chunk(0) >>> f"Size of chunk {0} from a: {len(compressed_chunk)} bytes" Size of chunk 0 from a: 160 bytes >>> # Decompress the chunk and convert it to a NumPy array >>> decompressed_chunk = blosc2.decompress(compressed_chunk) >>> np.frombuffer(decompressed_chunk, dtype=a.dtype) array([ 0., 1., 5., 6., 20., 21., 25., 26., 2., 3., 7., 8., 22., 23., 27., 28., 10., 11., 0., 0., 30., 31., 0., 0., 12., 13., 0., 0., 32., 33., 0., 0.], dtype=float32)
- read_range(offset: int, size: int) bytes[source]¶
The bytes at [offset, offset + size) of the remote frame.
- read_ranges(spans: Sequence[tuple[int, int]]) list[bytes][source]¶
The bytes of every span, in one request where the server allows it.
- refresh_stamp() None[source]¶
Look at the array again, so that
stampspeaks for it now.meta is read when the handle is opened and, of itself, never again: a stamp off it names the array as this handle last saw it, which for a handle that has outlived someone else’s writes is not the array. A Proxy calls this before it reads the stamp it will judge its cache by, which is the one moment that difference decides anything.
One api/info, and none at all for an array already known to be complete – nothing can write to one of those, so nothing it reports can move.
- save(urlpath: str, contiguous: bool = True, **kwargs) None[source]¶
Persist the remote array reference using a CFrame-backed carrier.
- slice(slice_: int | slice | tuple | Sequence[int] | np.ndarray) blosc2.NDArray[source]¶
Get a slice of the array (returning blosc2 NDArray array).
- Parameters:
slice_¶ (int, slice, tuple of ints and slices, sequence of ints, or ndarray) – The slice to fetch. A sequence of integers or an integer or boolean array gathers those coordinates, as numpy reads them. A list of slices is not a key – numpy stopped reading one as a tuple – and raises IndexError rather than being read as something else.
- Returns:
out – A blosc2.NDArray containing the data slice.
- Return type:
Examples
>>> import blosc2 >>> urlbase = "https://cat2.cloud/demo" >>> path = "@public/examples/dir1/ds-2d.b2nd" >>> remote_array = blosc2.C2Array(path, urlbase=urlbase) >>> data_slice = remote_array.slice((slice(3,5), slice(1,4))) >>> data_slice.shape (2, 3) >>> type(data_slice) blosc2.ndarray.NDArray
- update_chunk(nchunk: int, chunk: bytes) dict[source]¶
Write one compressed chunk into a slot of the remote array.
The array has to exist and to be laid out already – blosc2.uninit and an upload is what makes one – and the slot has to be one nothing was ever written to. That is not a restriction the transport invents: a chunk written into an empty slot is appended to the frame and moves nothing, while one written over a chunk that is already there moves every byte after it, so a fill made of writes-once is the cheap one and the one whose offsets a concurrent reader can keep.
The chunk must match the array’s geometry – its chunkshape, its typesize and its blocksize – which is what compressing against
cparamsandblocksgives; the server checks it and refuses anything else rather than storing a chunk the array cannot read.- Parameters:
nchunk¶ (int) – Which chunk of the array to write, numbered as
NDArray.get_chunk()numbers them.chunk¶ (bytes) – The compressed chunk, as
SChunk.get_chunk()orblosc2.compress2()produce it.
- Returns:
out – What the server reports of the array’s state now. Carries
writtenandnchunkswhere it counts them, so a writer can see a fill finish without asking again.- Return type:
dict
- Raises:
ChunkAlreadyWritten – The slot already holds a chunk. The array is untouched.
Examples
>>> import math, blosc2, numpy as np >>> a = blosc2.C2Array("@personal/run.b2nd", urlbase) >>> data = np.arange(math.prod(a.chunks), dtype=a.dtype).reshape(a.chunks) >>> itemsize = a.dtype.itemsize >>> chunk = blosc2.compress2( ... data, typesize=itemsize, blocksize=math.prod(a.blocks) * itemsize ... ) >>> a.update_chunk(0, chunk) {'written': 1, 'nchunks': 320}
The blocksize is spelled out because
blosc2.compress2()picks its own when it is not: left to choose it takes the whole chunk, and a chunk blocked differently from the array is one the server refuses.
- wants_blocks(nchunk: int, nwanted: int, wave=None) bool[source]¶
Whether fetching nwanted blocks of a chunk beats fetching all of it.
- written_chunks() ndarray[source]¶
Which chunks of the remote array hold content; see
ByteRangeNDSource.written_chunks().Read out of the frame’s own offsets, which is where a fill records itself: no endpoint of its own, and nothing for the server to keep in step with the array. Read afresh every time, since the point of asking is to see what other writers have done since – which is a couple of range reads, the header first (a write moves the frame’s length, and the offsets are found through it) and then the offsets it locates.
Nothing else about the handle is disturbed: this asks what the array holds, not what this handle has done, so meta is left as it was and no api/info is spent on it.
- property blocks: tuple[int]¶
The blocks of the remote array
- property blocks_per_chunk: int¶
How many blocks a chunk of the remote array holds.
Geometry, and api/info already carries it, so this costs no request: chunks are padded to whole blocks, so every chunk holds the same number of them, edge chunks included.
- property blocksize: int¶
The block size (in bytes) for the remote container.
- property cbytes: int¶
The number of compressed bytes of the remote array
- property chunks: tuple[int]¶
The chunks of the remote array
- property cratio: float¶
The compression ratio of the remote array
- property dtype: dtype¶
The dtype of the remote array
- property info: InfoReporter¶
Print information about this remote array.
- property info_items: list¶
A list of tuples with the information about the remote array. Each tuple contains the name of the attribute and its value.
- max_concurrency = 8¶
How many fetches a Proxy over this array may run at once.
Every chunk or block is a request whose cost is mostly the round trip, so overlapping them is what a remote source has to gain;
Proxy.afetch()already used this figure for a C2Array, and fetch was serial only for want of somewhere to read it from. get_chunk and the range reads are thread-safe: they share one pooled HTTP client and hold no state of their own.
- property max_ranges: int¶
How many ranges one request to this server may carry.
- property nbytes: int¶
The number of bytes of the remote array
- property serves_blocks: bool¶
Whether blocks are worth asking this dataset for, as far as info can say.
What api/info already carries, and no request of its own: a dataset the server computes reports an expression where a stored one reports a geometry, and only a stored one has a frame to read ranges of. That is the whole question here. Whether taking a particular chunk apart pays is a different one, and it is asked per fetch by
ByteRangeNDSource.wants_blocks(), which knows what the slice touches; this cannot, since it is read before any slice exists.It used to answer no as well for a frame whose chunks averaged under
BLOCK_MIN_CBYTES, which decided from one number, once, that no future slice of that dataset would ever be worth taking apart. That forfeited the bytes the block path exists to save: measured against a Caterva2 server, a dataset of 193 KB chunks reads a slab of 81 of them in 6.4 MB against 13.3 MB whole, and one of 650 KB chunks a slab of 36 in 6.4 MB against 23.3 MB – 2.1x and 3.6x the traffic, on every such read, for the life of the dataset. Where the link is what is scarce, and a server’s uplink is shared by everyone reading through it, those are the bytes that decide how many readers it can hold. The judgement was never wrong, only made too early and too widely: a point read of a small chunk really does cost more than it saves, and wants_blocks still refuses it.Read off api/info again where this handle has written since it last looked, which is the one case where the answer moves under it: a dataset may be laid out before it is stored. That is one request to a handle that has just written, and none at all to a reader – which is what the promise below needs.
False is the whole answer; True is only that it is worth one request to find out, which
block_source()spends. A Proxy reads this when it is built, to decide whether its cache records blocks or chunks, so it must cost nothing and must not depend on what has been fetched.A server that reports
accept_rangesspares even that request where the answer is no: a dataset this server mounts from a peer reports the peer’s geometry, being stored there, but is fetched from its owner and re-serialized here, so a range read of it is refused. Nothing else in what api/info says can tell the two apart. That is asked here through_serves_ranges, and again where the source is actually built, so that reading the frame’s index is spared it too – but no released Caterva2 answers it yet, so today every dataset pays the one request.A dataset too small for blocks to pay anywhere in it is ruled out from api/info alone, and pays neither – see
blocks_could_ever_pay(), which is a bound over the whole frame and not the one-chunk judgement described above.
- property shape: tuple[int]¶
The shape of the remote array
- property stamp: str | None¶
What names the exact remote bytes, for a Proxy to check a cache by.
Geometry cannot tell a dataset that was replaced from the one a cache was filled from: a shape and a partitioning survive a rewrite, while every cached chunk – and, in block mode, every offset they were fetched by – goes stale. The server’s own mtime does tell, and api/info carries it, so this costs no request of its own; the compressed size goes in with it, since a rewrite within the same clock tick is what an mtime cannot see. What it names is the array as this handle last looked at it –
refresh_stamp()is how a caller that needs it to be the array now says so, and what a Proxy calls before judging a cache by it.Two questions, and they want different answers. Which array is this is answered by the nonce a server writes into an array’s vlmeta the first time a chunk is written to it: a size and an mtime can both be repeated by a different array that came to sit at the same path, and a cache served against one of those is stale without ever saying so. Has it changed since is answered by the mtime and the compressed size, as before.
The second question stops being worth asking once the array is complete. Every slot of a filled array is claimed, so every write to it is refused, and the bytes a cache holds cannot move again – so a complete array is stamped by its nonce and its size, and a cache of it survives an mtime that churned for reasons of its own.
An array still being filled is stamped freshly on every write, and has to be. A cache built while a chunk was unwritten holds that chunk as the zeros an unwritten chunk reads as, and holds its offset as the run-length one it had; when a writer fills that slot, both are wrong, and nothing in the cache marks them apart from the chunks that are still good.
None when the server reports no mtime and the array carries no nonce, which leaves the cache checked on its geometry alone, as every source without a stamp is.
- property vlmeta: dict¶
The variable-length metadata of the remote array.
Read again where this handle has written since it last looked: a fill records itself here, so a writer asking what it just did would otherwise be told what was true before it started.
- class blosc2.ChunkAlreadyWritten[source]¶
A chunk was written to a slot of a remote array that already held content.
A server that accepts chunk writes accepts each slot exactly once: the frame’s own offsets say whether a slot was ever written, and a second write would move every chunk that came after it. So a writer that finds this has lost a race, or is repeating work another writer already did; either way the array is intact and the chunk it carried is the one to drop.
- Attributes:
- args
Methods
add_note(object, /)Exception.add_note(note) -- add a note to the exception
with_traceback(object, /)Exception.with_traceback(tb) -- set self.__traceback__ to tb and return self.
C2NDSource class¶
- class blosc2.C2NDSource(array: C2Array, max_concurrency: int = 8)[source]¶
The frame behind a C2Array, read over HTTP byte ranges.
Caterva2 serves a stored dataset with a Starlette
FileResponse, which implements RFC 7233 by itself: a ranged request comes back 206 with only the bytes asked for, seeked to in the file rather than materialized, and the auth cookie composes with it. That is everything ByteRangeNDSource needs, so a slice costs the blocks it touches instead of the chunks they live in.A dataset the server builds – a lazy expression, an HDF5 leaf, a
.b2zmember – is streamed instead, and a streamed response ignores theRangeheader and answers with the whole body.read_range()refuses such an answer without reading it off the socket, and C2Array then keeps to whole chunks for good. Which is why this is built throughC2Array.block_source()rather than directly: the fallback belongs with the array, whoseapi/chunkpath works for every dataset there is.- Attributes:
blocksThe block shape of the source.
chunksThe chunk shape of the source.
cparamsThe compression parameters of the source.
dtypeThe dtype of the source.
shapeThe shape of the source.
- stamp
Methods
aget_chunk(nchunk)Same as
get_chunk(), but without blocking the caller's event loop.block_plan(nchunk, nblocks)The range reads that cover nblocks, near-adjacent ones merged.
chunk_layout(nchunk)Read where the blocks of a chunk are: its header, bstarts and extents.
chunk_layouts(nchunks)chunk_layout()for several chunks, in as few requests as they fit.get_chunk(nchunk)Return the compressed chunk in
self.invalidate_index()Forget where the chunks and blocks are, so the next read looks again.
read_range(offset, size)The bytes at [offset, offset + size) of the frame.
read_ranges(spans)Every span in one request, which HTTP has a shape for and S3 has not.
wants_blocks(nchunk, nwanted[, wave, nruns])Whether fetching nwanted blocks of a chunk beats fetching all of it.
written_chunks()Which chunks of the frame hold content, as a boolean per chunk.
- read_range(offset: int, size: int) bytes[source]¶
The bytes at [offset, offset + size) of the frame.
The whole of the transport: everything else here is the frame format. Fewer bytes may come back only at the end of the frame; anything else is an error, since the caller has no way to ask for the rest. Must be safe to call from several threads at once, which is what lets Proxy overlap the fetches of one slice.
- read_ranges(spans: Sequence[tuple[int, int]]) list[bytes][source]¶
Every span in one request, which HTTP has a shape for and S3 has not.
RFC 7233 lets a Range header name several spans, and the answer is a multipart/byteranges body carrying each with its own Content-Range. Starlette builds that, so a whole wave of block reads – across chunks, since they are all the same file – costs one round trip instead of one each. Measured against cat2.cloud: 32 spans in 0.136 s against 0.208 s for 32 requests eight at a time, and 1.530 s for them one at a time.
The server may serve fewer parts than were asked for: Starlette sorts the spans and merges the ones that touch, and answers a single 206 when they all merge into one. So the answer is taken apart by what each part says it holds, and each span read out of the part that covers it, rather than by trusting the order. An answer that does not carry the whole of what was asked for is retried a range at a time, and only a server that does that MULTIPART_STRIKES times is written off as unable to batch: an order of magnitude for the rest of the process is too much to pay for one truncated answer.
- max_ranges = 64¶
How many ranges one request of this transport may carry.
One means one request each, which is all any object store offers. A server answering
multipart/byterangestakes more – seeread_ranges()– and then a slice costs a couple of requests rather than a couple per chunk it touches.
URLPath class¶
- class blosc2.URLPath(path: str, /, urlbase: str | None = None, auth_token: str | None = None)[source]¶
__init__(path, /[, urlbase, auth_token])Create an instance of a remote data file (aka C2Array) urlpath.
- __init__(path: str, /, urlbase: str | None = None, auth_token: str | None = None)[source]¶
Create an instance of a remote data file (aka C2Array) urlpath. This is meant to be used in the
blosc2.open()function.The parameters are the same as for the
C2Array.__init__().
Context managers¶
- blosc2.c2context(*, urlbase: str | None = None, username: str | None = None, password: str | None = None, auth_token: str | None = None) None¶
Context manager that sets parameters in Caterva2 server requests.
A parameter not specified or set to
Nonewill inherit the value from the previous context manager, defaulting to an environment variable (see below) if supported by that parameter. Parameters set to an empty string will not be used in requests (without a default either).If the server requires authorization for requests, you can either provide an auth_token (which you should have obtained previously from the server), or both username and password to obtain the token by logging in to the server. The token will be reused until it is explicitly reset or requested again in a later context manager invocation.
Please note that this manager is reentrant but not safe for concurrent use.
- Parameters:
urlbase¶ (str | None) – The base URL to be used when a C2Array instance does not have a server URL base set. If not specified, it defaults to the value of the
BLOSC_C2URLBASEenvironment variable.username¶ (str | None) – The username for logging in to the server to obtain an authorization token. If not specified, it defaults to the value of the
BLOSC_C2USERNAMEenvironment variable.password¶ (str | None) – The password for logging in to the server to obtain an authorization token. If not specified, it defaults to the value of the
BLOSC_C2PASSWORDenvironment variable.auth_token¶ (str | None) – The authorization token to be used when a C2Array instance does not have an authorization token set.
- Yields:
out (None)