Spool Selection

Spool.select uses one selector model for memory and directory spools. Patch-list and directory spools compose selections in a PatchCatalog; ordinary metadata predicates are pushed into SQLite and remain lazy until contents, length, indexing, or iteration requires rows.

Bare selector names resolve to attributes first and then coordinates. _attrs and _coords provide explicit namespaces when needed: either a name -> selector mapping (the fully general form, required when a name cannot be a Python keyword) or a name/collection of names tagging which bare keyword arguments belong to that namespace (e.g. select(sensor=(1, 10), _coords="sensor")). Unknown names raise immediately instead of being ignored.

Attribute equality, membership, ranges, and glob predicates are evaluated by the index. Regular expressions use a SQL candidate predicate and an exact residual filter; chained regular expressions are combined with AND. Quantities are converted to the canonical unit recorded by the index, and dimensionally incompatible queries raise rather than silently returning incorrect matches. Values stored without units can never be proven incompatible, so they remain candidates for quantity selectors rather than being silently excluded.

Coordinate predicates select by range — a (start, stop) tuple or slice, with None/... for an open end. Scalar, value-membership, and boolean-sample-mask coordinate selectors have no exact patch-level meaning spool-wide and are rejected (apply masks per patch, e.g. spool.map(lambda p: p.select(...)); boolean arrays over patches, spool[mask], still select membership). Numeric coordinate envelopes are stored in each coordinate’s original units, so bare numeric range bounds mean the coordinate’s own units — exactly what Patch.select means by them — and quantities convert themselves to whatever units each stored definition uses. In an archive whose files state different units for one coordinate, a bare range therefore selects a per-file interval; pass a quantity to mean one physical interval. The exact per-patch trim defers its representation until each patch is known, so a mixed archive of unit-bearing and unitless patches is handled correctly in one selection.

Coordinate predicates first select patches whose summary envelopes can overlap the request. The loaded patch is then selected exactly. samples=True never excludes a patch at the index stage. With relative=True, bounds resolve independently against each patch: positive offsets count from the coordinate minimum and negative offsets from its maximum. The first bound is the lower limit and the second is the upper limit. Crossed bounds select nothing; equal bounds include a sample at that value when one exists. Attribute predicates in the same call remain unchanged.

Relative selection excludes patches without the requested coordinate and drops rows whose indexed envelopes establish that the selected window is empty. For a patch spanning three seconds, both select(time=(5, -1), relative=True) and select(time=(5, ...), relative=True) therefore return an empty spool. Patch.select returns a patch with zero samples for either request. Spool length, contents, indexing, and iteration use the surviving rows. These metadata checks preserve lazy loading: a window overlapping an envelope may still fall between samples or inside a gap, so its exact result is determined when the patch loads. Sample-index selection continues to retain empty patch entries.

A selection pushed into a reader gives the loaded patch the data_id of the window it read — the id the same selection on the loaded patch gives — so the replayed select finds its work done and records nothing. select does not append a history entry. A window is named by the samples it holds, not by the call which reached it, so the spelling of a bound no longer matters for a file-backed patch: ..., None, and a quantity in any unit all reach the same id. A selection the source cannot load — stepped, fancy, or on a coordinate the file’s array cannot be sliced along — derives from the whole array and the select call instead, where those spellings still tell apart. A single native-unit range can be pushed into the reader on either an even or uneven coordinate. Composed ranges on uneven or float16/float32 coordinates replay against the loaded patch because their envelopes cannot reliably identify which samples earlier selections left. Selections involving associated coordinates also replay there, so a trim on one coordinate can make a later trim on another coordinate sharing its axis a no-op. Attribute predicates, including data_id, always query indexed source metadata; they do not query provenance generated when a selected patch loads.

Sample selections leave indexed sample count and data_id unknown until loading, including full-span sample ranges. For relative selections on coordinates the index records as float16/float32, indexed sample count and data_id remain unknown until loading: index arithmetic cannot reliably predict whether rounding removes a sample. The loaded patch still applies the exact selection and provides its resulting identity.

Restructuring operations that create new patch identities (chunking, concatenation) materialize a derived in-memory catalog whose rows are the plan outputs, so selection on a chunked spool runs the identical catalog engine. Sorting, slicing, and array selection never restructure: they compose lazy order and membership specs on the current catalog. Exact selections already attached to a parent view still apply when member source patches load.

Catalog views share their source state. Adding, removing, or rescanning sources invalidates realized metadata so existing views observe the updated catalog under their composed predicates.

The core contract, executed here so drift fails the doc build: names resolve attributes-first then coordinates, unknown names raise, and coordinate ranges are exact on the loaded patches (candidacy by envelope at the index, exactness at load):

import pytest

import dascore as dc
from dascore.exceptions import InvalidSpoolQueryError

spool = dc.get_example_spool("diverse_das")
with pytest.raises(InvalidSpoolQueryError):
    spool.select(not_a_name="anything")

selected = spool.select(acquisition_key="DAS2.*")
assert all(p.attrs.acquisition_key.startswith("DAS2.") for p in selected)

df = spool.get_contents()
t0 = df["time_min"].min()
window = (t0, t0 + dc.to_timedelta64(1))
for patch in spool.select(time=window):
    coord = patch.get_coord("time")
    assert coord.min() >= window[0] and coord.max() <= window[1]

Bare numeric coordinate bounds mean the coordinate’s own units, matching Patch.select, and scalar/membership coordinate selectors raise:

ft_patch = dc.get_example_patch().convert_units(distance="ft")
coord = dc.spool([ft_patch]).select(distance=(20, 60))[0].get_coord("distance")
assert float(coord.min()) >= 20 and float(coord.max()) <= 60  # 20-60 ft

with pytest.raises(InvalidSpoolQueryError):
    dc.spool([ft_patch]).select(distance=100)  # scalar has no range meaning

Relative trimming never swaps crossed bounds to recover data from the other end of a short patch:

short_patch = dc.get_example_patch().select(time=(0, 3), relative=True)
for bounds in [(5, -1), (5, ...)]:
    assert short_patch.select(time=bounds, relative=True).size == 0
    assert len(dc.spool([short_patch]).select(time=bounds, relative=True)) == 0

Independent coordinate windows

Pass a rectangular (n, 2) array to select several bounded absolute ranges along one coordinate. Each row is applied to each matching source patch in input order. Overlapping, repeated, and shared-endpoint ranges retain separate pieces, so selecting two rows over a file boundary can return more than two patches. The endpoints are inclusive coordinate bounds; an empty (0, 2) array selects nothing. Ordinary selectors on other attributes or coordinates can accompany the array. Array windows currently require samples=False and relative=False.

The view is deferred and follows source catalog updates. Its contents report the selected sample envelopes when the source metadata holds the full coordinate values, including uneven coordinates; file scans can read coordinate payloads without loading the measurement arrays. A single (start, stop) selector keeps its existing semantics.

import numpy as np

patch = dc.get_example_patch()
source = dc.spool(patch)
windows = np.array([[2, 4], [4, 6], [2, 4]])
pieces = source.select(distance=windows)
assert len(pieces) == 3
assert [(p.get_coord("distance").min(), p.get_coord("distance").max()) for p in pieces] == [(2, 4), (4, 6), (2, 4)]