chunk

chunk(
    self ,
    overlap: int | float | str | numpy.datetime64 | pandas.Timestamp | None[int, float, str, datetime64, Timestamp, None] = None,
    keep_partial: bool = False,
    snap_coords: bool = True,
    tolerance: float | pint.registry.Quantity | numpy.timedelta64[float, Quantity, timedelta64] = 1.5,
    conflict: Literal[‘drop’, ‘raise’, ‘keep_first’] = raise,
    group: str | collections.abc.Sequence[str, collections.abc.Sequence[str], None] = None,
    missing_dim: Literal[‘raise’, ‘drop’] = raise,
    fill_value = None,
    on_incomplete: Literal[‘warn’, ‘raise’, ‘ignore’] = raise,
    **kwargs ,
)-> ‘Self’

Chunk the data in the spool along specified dimension.

Parameters

Parameter	Description
overlap	The amount of overlap between each segment, starting with the end of first patch. Negative values can be used to create gaps.
keep_partial	If True, keep the segments which are smaller than chunk size. This often occurs because of data gaps or at end of chunks.
snap_coords	If True (default), simplify the coordinates of joined patches to an evenly sampled range, absorbing the sub-sample jitter of labels rounded on their way to a file. A merge across a hole keeps an exact segmented coordinate however wide the tolerance: missing samples are absent data, not a slower sampling rate.
tolerance	The maximum number of samples a block of data can be spaced (gap) and still be considered contiguous. A quantity or timedelta instead bounds the excess over one sample in the coordinate’s own units (eg `tolerance=1 * s` admits a spacing of one step plus a second), which also works for patches whose sampling interval is unknown. Either way a boundary of one sample is contiguous. A hole inside a patch whose coordinate is segmented into runs of one step (up to 256) is a gap like any other, so each run can end an output or join a neighbouring patch. See `dascore.utils.gaps.GapTolerance`.
conflict	Indicates how to handle attributes which hold conflicting values across the patches being combined (eg data_type, data_units, custom attrs). A missing value (None, NaN, ““) is a value like any other: it equals another missing one and nothing else, so a patch which never stated an attribute conflicts with one which did. History and the ids are never compared. If”raise” (default) raise an AttributeMergeError for conflicting values. If “drop”, omit the conflicting attributes from the output. If “keep_first”, keep the first patch’s value of each.

group	Attributes which partition patches into separate outputs: conflicting values are never an error, the patches simply land in different outputs. A missing value (null or ““) is a value like any other, so patches which never stated an attribute are grouped together and apart from those which did. Defaults to the config option `patch_kind_attrs`; unlike the default, explicitly passed names must exist on at least one patch. Dimensions and coordinate identities always partition implicitly.
missing_dim	What to do when patches lack the chunked dimension: “raise” (default) or “drop” (exclude them from the output).
on_incomplete	For explicit `(n, 2)` windows, `"raise"` (default) raises `ChunkError` for an unmet request, `"warn"` reports and skips it, and `"ignore"` skips it silently. With `keep_partial=True`, nonempty available pieces are accepted before this policy is applied; wholly absent requests still follow it. Other chunk modes retain their existing behavior.
fill_value	If given, the value written into the samples missing from a merge, so an output spanning a hole is evenly sampled rather than segmented. `tolerance` still decides which holes are bridged at all: a hole it does not span separates patches as before, and nothing is filled across it. The value has to survive a cast to the data’s own dtype, so `np.nan` needs float data; fill integer data with an integer, or cast it first.
kwargs	kwargs are used to specify the dimension along which to chunk, eg: `time=10` chunks along the time axis in 10 second increments. The value may also be a quantity: one of the coordinate’s own units (`time=10 * s`) or a data size (`time=25 * megabytes`), which chunks so each patch’s data array is about that large. `overlap` accepts the same forms. An `(n, 2)` array instead requests bounded absolute coordinate windows with inclusive endpoints, in input order. Overlapping and duplicate windows produce separate outputs. Explicit windows do not accept `overlap`; quantities in their bounds are absolute points.

Examples

import numpy as np

import dascore as dc
from dascore.units import s, megabytes

spool = dc.get_example_spool("random_das")
# get spools with time duration of 10 seconds
time_chunked = spool.chunk(time=10, overlap=1)
# the same, with the units stated explicitly
unit_chunked = spool.chunk(time=10 * s)
# get patches whose data arrays are at most ~1 MB
size_chunked = spool.chunk(time=1 * megabytes)
# merge along time axis
time_merged = spool.chunk(time=...)
# merge across holes of up to 9 missing samples, filling them
gapless = spool.chunk(time=..., tolerance=10, fill_value=np.nan)
# Request two independent absolute windows and inspect the plan.
start = spool.get_contents()["time_min"].min()
step = spool.get_contents()["time_step"].iloc[0]
windows = np.array([
    [start, start + 4 * step],
    [start + 8 * step, start + 12 * step],
])
selected = spool.select(time=windows)
explicit = spool.chunk(time=windows, on_incomplete="ignore")
explicit_plan = spool.chunk_plan(time=windows, on_incomplete="ignore")
assert len(explicit_plan.outputs) == len(explicit)

Note

Data-size chunks measure only the data array and round sample counts down. Mixed dtypes are sized after promotion.

Spool.concatenate performs a similar operation but disregards the coordinate values.

To inspect what a chunk call will do before running it — which output patches it produces and which slice of which source patch feeds each one — use Spool.chunk_plan, which takes the same arguments and returns the plan without touching any data.

Keywords

spool, chunking, overlapping chunks, gaps, archive