chunk

chunk(
    self ,
    overlap: int | float | str | numpy.datetime64 | pandas.Timestamp | None[int, float, str, datetime64, Timestamp, None] = None,
    keep_partial: bool = False,
    snap_coords: bool = True,
    tolerance: float = 1.5,
    conflict: Literal[‘drop’, ‘raise’, ‘keep_first’] = raise,
    group: str | collections.abc.Sequence[str, collections.abc.Sequence[str], None] = None,
    missing_dim: Literal[‘raise’, ‘drop’] = raise,
    **kwargs ,
)-> ‘Self’

Chunk the data in the spool along specified dimension.

Parameters

Parameter	Description
overlap	The amount of overlap between each segment, starting with the end of first patch. Negative values can be used to create gaps.
keep_partial	If True, keep the segments which are smaller than chunk size. This often occurs because of data gaps or at end of chunks.
snap_coords	If True (default), simplify the coordinates of joined patches to an evenly sampled range when doing so moves no coordinate value by more than `tolerance` samples. Merges whose gaps exceed that keep an exact segmented coordinate instead.
tolerance	The maximum number of samples a block of data can be spaced (gap) and still be considered contiguous.
conflict	Indicates how to handle conflicts in attributes other than those indicated by dim (eg tag, history, acquisition_key, etc). If “drop” simply drop conflicting attributes, or attributes not shared by all models. If “raise” raise an [AttributeMergeError](`dascore.exceptions.AttributeMergeError`] when issues are encountered. If “keep_first”, just keep the first value for each attribute.

group	Attributes which partition patches into separate outputs (their values differing is never an error). Defaults to the config option `groupby_attrs`; unlike the default, explicitly passed names must exist on at least one patch. Dimensions and coordinate identities always partition implicitly.
missing_dim	What to do when patches lack the chunked dimension: “raise” (default) or “drop” (exclude them from the output).
kwargs	kwargs are used to specify the dimension along which to chunk, eg: `time=10` chunks along the time axis in 10 second increments. The value may also be a quantity: one of the coordinate’s own units (`time=10 * s`) or a data size (`time=25 * megabytes`), which chunks so each patch’s data array is about that large. `overlap` accepts the same forms.

Examples

import dascore as dc
from dascore.units import s, megabytes

spool = dc.get_example_spool("random_das")
# get spools with time duration of 10 seconds
time_chunked = spool.chunk(time=10, overlap=1)
# the same, with the units stated explicitly
unit_chunked = spool.chunk(time=10 * s)
# get patches whose data arrays are at most ~1 MB
size_chunked = spool.chunk(time=1 * megabytes)
# merge along time axis
time_merged = spool.chunk(time=...)

Note

A data size measures the patch’s data array only; coordinates and attrs are extra, as are any copies a later processing step makes, so the patch as a whole is somewhat larger. The sample count is rounded down, so the data never exceeds the requested size, and a merge of patches with different dtypes is sized against the dtype they upcast to.

Spool.concatenate performs a similar operation but disregards the coordinate values.

To inspect what a chunk call will do before running it — which output patches it produces and which slice of which source patch feeds each one — use Spool.chunk_plan, which takes the same arguments and returns the plan without touching any data.