chunk

chunk(
    self ,
    overlap: int | float | str | numpy.datetime64 | pandas.Timestamp | None[int, float, str, datetime64, Timestamp, None] = None,
    keep_partial: bool = False,
    snap_coords: bool = True,
    tolerance: float | pint.registry.Quantity | numpy.timedelta64[float, Quantity, timedelta64] = 1.5,
    conflict: Literal[‘drop’, ‘raise’, ‘keep_first’] = raise,
    group: str | collections.abc.Sequence[str, collections.abc.Sequence[str], None] = None,
    on_missing_dim: Literal[‘raise’, ‘drop’] = raise,
    fill_value = None,
    on_incomplete: Literal[‘warn’, ‘raise’, ‘ignore’] = raise,
    **kwargs ,
)-> ‘Self’

Chunk the data in the spool along specified dimension.

Parameters

Parameter	Description
overlap	The amount of overlap between each segment, starting with the end of first patch. Negative values can be used to create gaps.
keep_partial	If True, keep the segments which are smaller than chunk size. This often occurs because of data gaps or at end of chunks.
snap_coords	If True (default), simplify the coordinates of joined patches to an evenly sampled range, absorbing the sub-sample jitter of labels rounded on their way to a file. If False, joined coordinates keep their exact labels.
tolerance	The maximum number of samples a block of data can be spaced (gap) and still be considered contiguous. A quantity or timedelta instead bounds the excess over one sample in the coordinate’s own units (eg `tolerance=1 * s` admits a spacing of one step plus a second), which also works for patches whose sampling interval is unknown. Either way a boundary of one sample is contiguous. Without `fill_value` a missing sample always ends an output, so a looser tolerance only matters with one. See `dascore.utils.gaps.GapTolerance`.
conflict	Indicates how to handle attributes which hold conflicting values across the patches being combined (eg data_type, data_units, custom attrs). A missing value (None, NaN, ““) is a value like any other: it equals another missing one and nothing else, so a patch which never stated an attribute conflicts with one which did. History and the ids are never compared. If”raise” (default) raise an AttributeMergeError for conflicting values. If “drop”, omit the conflicting attributes from the output. If “keep_first”, keep the first patch’s value of each.

group	Attributes which partition patches into separate outputs: conflicting values are never an error, the patches simply land in different outputs. A missing value (null or ““) is a value like any other, so patches which never stated an attribute are grouped together and apart from those which did. Defaults to the config option `patch_kind_attrs`; unlike the default, explicitly passed names must exist on at least one patch. Dimensions and coordinate identities always partition implicitly.
on_missing_dim	What to do when patches lack the chunked dimension: “raise” (default) or “drop” (exclude them from the output).
on_incomplete	For explicit `(n, 2)` windows, `"raise"` (default) raises `ChunkError` for an unmet request, `"warn"` reports and skips it, and `"ignore"` skips it silently. With `keep_partial=True`, nonempty available pieces are accepted before this policy is applied; wholly absent requests still follow it. Other chunk modes retain their existing behavior.
fill_value	If given, the value written into the samples missing from a merge, so an output may span a hole `tolerance` spans. Without it no output holds a hole. The value has to survive a cast to the data’s own dtype, so `np.nan` needs float data; fill integer data with an integer, or cast it first.
kwargs	kwargs are used to specify the dimension along which to chunk, eg: `time=10` chunks along the time axis in 10 second increments. Each window is half-open, `[start, start + length)`, so windows without overlap share no sample; only a final partial window is dropped, unless `keep_partial=True`. The value may also be a quantity: one of the coordinate’s own units (`time=10 * s`) or a data size (`time=25 * megabytes`), which chunks so each patch’s data array is about that large. `overlap` accepts the same forms. An `(n, 2)` array instead requests absolute coordinate windows with inclusive endpoints, in input order. Overlapping and duplicate windows produce separate outputs. Explicit windows do not accept `overlap`; quantities in their bounds are absolute points. `None`, `...`, NaN, NaT or a signed infinity is open. Arrays on several dimensions need equal row counts, row `i` of each one window: the first is chunked, the others trim (see the notes).

Examples

import numpy as np

import dascore as dc
from dascore.units import s, megabytes

spool = dc.get_example_spool("random_das")
# get spools with time duration of 10 seconds
time_chunked = spool.chunk(time=10, overlap=1)
# the same, with the units stated explicitly
unit_chunked = spool.chunk(time=10 * s)
# get patches whose data arrays are at most ~1 MB
size_chunked = spool.chunk(time=1 * megabytes)
# merge along time axis
time_merged = spool.chunk(time=...)
# merge across holes of up to 9 missing samples, filling them
gapless = spool.chunk(time=..., tolerance=10, fill_value=np.nan)
# Request two independent absolute windows and inspect the plan.
start = spool.get_contents()["time_min"].min()
step = spool.get_contents()["time_step"].iloc[0]
windows = np.array([
    [start, start + 4 * step],
    [start + 8 * step, start + 12 * step],
])
selected = spool.select(time=windows)
explicit = spool.chunk(time=windows, on_incomplete="ignore")
explicit_plan = spool.chunk_plan(time=windows, on_incomplete="ignore")
assert len(explicit_plan.outputs) == len(explicit)
# the same windows, each trimmed to distances 10 through 20
boxes = spool.chunk(time=windows, distance=np.array([[10, 20]] * 2))

Note

Data-size chunks measure only the data array and round sample counts down. Mixed dtypes are sized after promotion.

Spool.concatenate performs a similar operation but disregards the coordinate values.

To inspect what a chunk call will do before running it — which output patches it produces and which slice of which source patch feeds each one — use Spool.chunk_plan, which takes the same arguments and returns the plan without touching any data.

Keywords

spool, chunking, overlapping chunks, gaps, archive