chunk

chunk(
    self ,
    overlap: int | float | str | numpy.datetime64 | pandas.Timestamp | None[int, float, str, datetime64, Timestamp, None] = None,
    keep_partial: bool = False,
    snap_coords: bool = True,
    tolerance: float | pint.registry.Quantity | numpy.timedelta64[float, Quantity, timedelta64] = 1.5,
    conflict: Literal[‘drop’, ‘raise’, ‘keep_first’] = raise,
    group: str | collections.abc.Sequence[str, collections.abc.Sequence[str], None] = None,
    missing_dim: Literal[‘raise’, ‘drop’] = raise,
    **kwargs ,
)-> ‘Self’

Chunk the data in the spool along specified dimension.

Parameters

Parameter	Description
overlap	The amount of overlap between each segment, starting with the end of first patch. Negative values can be used to create gaps.
keep_partial	If True, keep the segments which are smaller than chunk size. This often occurs because of data gaps or at end of chunks.
snap_coords	If True (default), simplify the coordinates of joined patches to an evenly sampled range when doing so moves no coordinate value by more than `tolerance` (samples, or the length itself when the tolerance states one). Merges whose gaps exceed that keep an exact segmented coordinate instead.
tolerance	The maximum number of samples a block of data can be spaced (gap) and still be considered contiguous. A quantity or timedelta states that limit in the coordinate’s own units instead (eg `tolerance=1 * s`), which also works for patches whose sampling interval is unknown. Either way a boundary of one sample is contiguous, so a tolerance below one sample never splits adjacent patches.
conflict	Indicates how to handle attributes which hold conflicting values across the patches being combined (eg data_type, data_units, custom attrs). A missing value (None, NaN, ““) is a value like any other: it equals another missing one and nothing else, so a patch which never stated an attribute conflicts with one which did. History and the ids are never compared. If”raise” (default) raise an AttributeMergeError for conflicting values. If “drop”, omit the conflicting attributes from the output. If “keep_first”, keep the first patch’s value of each.

group	Attributes which partition patches into separate outputs: conflicting values are never an error, the patches simply land in different outputs. A missing value (null or ““) is a value like any other, so patches which never stated an attribute are grouped together and apart from those which did. Defaults to the config option `patch_kind_attrs`; unlike the default, explicitly passed names must exist on at least one patch. Dimensions and coordinate identities always partition implicitly.
missing_dim	What to do when patches lack the chunked dimension: “raise” (default) or “drop” (exclude them from the output).
kwargs	kwargs are used to specify the dimension along which to chunk, eg: `time=10` chunks along the time axis in 10 second increments. The value may also be a quantity: one of the coordinate’s own units (`time=10 * s`) or a data size (`time=25 * megabytes`), which chunks so each patch’s data array is about that large. `overlap` accepts the same forms.

Examples

import dascore as dc
from dascore.units import s, megabytes

spool = dc.get_example_spool("random_das")
# get spools with time duration of 10 seconds
time_chunked = spool.chunk(time=10, overlap=1)
# the same, with the units stated explicitly
unit_chunked = spool.chunk(time=10 * s)
# get patches whose data arrays are at most ~1 MB
size_chunked = spool.chunk(time=1 * megabytes)
# merge along time axis
time_merged = spool.chunk(time=...)

Note

A data size measures the patch’s data array only; coordinates and attrs are extra, as are any copies a later processing step makes, so the patch as a whole is somewhat larger. The sample count is rounded down, so the data never exceeds the requested size, and a merge of patches with different dtypes is sized against the dtype they upcast to.

Spool.concatenate performs a similar operation but disregards the coordinate values.

To inspect what a chunk call will do before running it — which output patches it produces and which slice of which source patch feeds each one — use Spool.chunk_plan, which takes the same arguments and returns the plan without touching any data.