Coordinate Internals

This note explains the current internal coordinate model in DASCore, with particular attention to exact values, runs, summaries, and scan payloads.

Coordinate classes

get_coord(...) returns the simplest coordinate that fits the supplied inputs:

Class	Purpose
`NumericCoord`	Every numeric, datetime, and timedelta coordinate, of any shape, including one whose labels are unknown (a single blank run).
`CoordString`	String or categorical labels.

A NumericCoord holds its labels as an ordered tuple of runs, each either a Grid or a Labels:

Run Purpose

Grid An evenly sampled stretch described by numbers rather than stored. Nanosecond time and integer grids are exact: the labels are origin + (phase + k * step_num) // step_den ticks, so a fractional step such as 1/1024 s never drifts and step_exact reports it, while step stays the nearest whole-tick spacing. A float grid (step_den of zero) keeps a scalar step and linear spacing; slices also retain parent_count, k0, and stride to evaluate the original labels, which is also what time in coarser or finer units and an integer span that does not divide by its count (get_coord(start=0, stop=10, shape=3) is float) fall back to.

Labels A window – offset, count and (possibly negative) stride – into one array of labels held as it is, read-only, of any shape. The array itself lives in the coordinate’s sources mapping under the id it was hashed with on the way in, so a slice, a stride, a reversal or a move to another coordinate only moves the window and the labels are copied and hashed once however far they travel. Converting units scales a window’s labels into a new entry, which is a genuinely new array.

Run	Purpose
`Grid`	An evenly sampled stretch described by numbers rather than stored. Nanosecond time and integer grids are exact: the labels are `origin + (phase + k * step_num) // step_den` ticks, so a fractional step such as 1/1024 s never drifts and `step_exact` reports it, while `step` stays the nearest whole-tick spacing. A float grid (`step_den` of zero) keeps a scalar step and linear spacing; slices also retain `parent_count`, `k0`, and `stride` to evaluate the original labels, which is also what time in coarser or finer units and an integer span that does not divide by its count (`get_coord(start=0, stop=10, shape=3)` is float) fall back to.
`Labels`	A window – offset, count and (possibly negative) stride – into one array of labels held as it is, read-only, of any shape. The array itself lives in the coordinate’s `sources` mapping under the id it was hashed with on the way in, so a slice, a stride, a reversal or a move to another coordinate only moves the window and the labels are copied and hashed once however far they travel. Converting units scales a window’s labels into a new entry, which is a genuinely new array.

One grid is the evenly sampled coordinate; one stored run is an arbitrary or N-dimensional array; several runs are a coordinate with holes in it, where each boundary records a break in sampling without changing any label. runs_count says how many there are and segments hands each back as a coordinate of its own.

How a coordinate is sampled is its step (None with no grid) and two independent flags, as the coordinate guidelines define them: gapped, where positions of the step’s grid hold no sample, and jittered, where stored labels sit off it by under half a step. evenly_sampled means a step, neither jittered nor gapped; stored labels sitting on their step’s grid within float rounding count, though only a grid answers questions by arithmetic. get_coord(dim, require_evenly_sampled=True) on a patch is the one requirement check.

get_coord(data=..., snap=False) reads labels exactly (rule 2 of the guidelines). Labels on a fitted grid at consecutive positions are evenly sampled: integer and time labels are judged exactly in their own tick unit, so 1024 Hz labels stored in microseconds find their 15625/16 µs grid, and a fractional step must span at least 100 ticks; floats are judged within the larger of _GRID_RTOL of the step and 4 ulp of the largest label, in the dtype they were stored in (float64 holding float32 values counts as float32). Labels whose offsets from the fitted grid, and whose spacings’ errors, all stay under half a step keep that step and are jittered; anything else has no step. Inference never invents holes. A stated step places the labels on its grid instead: skipped positions become holes, and two labels on one position, or labels drifting half a step from the grid, raise. No label moves, except that labels which a grid restates exactly are held as that grid.

The default, snap=None, still snaps labels within a relative 0.1% of one spacing onto a grid, as DASCore 0.1.24 did, with a FutureWarning where that differs from reading them exactly; from 0.1.26 it reads them exactly. Where the released median step would move a label half a step or more (#1419), jittered labels are snapped from their end labels and count instead, and others keep no step. snap=True snaps the same way without the warning. Readers do not use it: each builds its snapped grids with stored_coord and reads every other label exactly.

Cell convention

Labels conventionally centre half-open cells one step wide; this is a recommendation for producers, not an enforced or stored property, and adds no coordinate method. The auxiliary names {dim}_start / {dim}_stop are reserved for explicit low and high cell edges, just as {dim}_min / {dim}_max / {dim}_step describe the envelope. This expresses CF cell bounds as two one-dimensional coordinates associated with the dimension. When both are attached, the manager validates their length, association, units, monotonic direction, and start <= label < stop; a lone name has no special meaning. STFT and stacked tiles emit physical bounds; this replaces their former unitless sample-index pair. Their private _tile_index_{dim} coordinate retains the original reconstruction indices. Old development-version stacks with sample-index public bounds cannot be read and must be regenerated. Arbitrary tile permutations require explicitly dropping the public pair to satisfy its monotonic-order contract; private indices still permit reconstruction. Spectrogram rendering omits source-window bounds and draws one cell per hop at the window-centre label. Selection and raw decimation select its rows; grid-building operations discard it through the manager’s shared grid update; translation and conversion transform it with the labels. Waterfall uses explicit outer bounds when present and otherwise extends evenly sampled image extents by half the declared step. Label-based selection and coordinate summaries retain their existing meanings.

Segmented coordinates

Holding labels as runs keeps sampling breaks rather than flattening them into a single array or idealizing them as one grid. A grid which is exactly the next samples of its predecessor fuses automatically, and so do two windows which read on through one stored array; windows into different arrays never fuse, since concatenating their labels would cost a copy and lose both ids. Concatenation unions the stores, and an entry no window reads is dropped. A coordinate of several runs reports a step only when every run sits on the same one (its labels then follow one grid with positions missing between the runs, so it is gapped) and None when they disagree; a summary carries a step only for an evenly sampled coordinate, since a summary with a step rebuilds one.

There are two primary construction paths:

concat_coords(...) combines already-known coordinate blocks, normalizes their ordering, and records non-contiguous boundaries.
get_coord(data=..., snap=False) keeps an array as one grid when it reproduces every label exactly, otherwise as one stored-label run, with a step only where it is jittered, so it reports no holes. Declaring step (get_coord(data=..., step=...)) places the labels on that grid, making each stretch of consecutive positions a grid of that step (one stored run if they are jittered), and missing() and the spool’s gap methods then see the holes between them. Selecting samples by mask, array or position keeps the step the same way, so the removed samples are holes; only a stride gives a coarser step.

get_discontinuities(...) reports segment boundaries without materializing the full coordinate, and for a monotonic array every spacing that is not its declared step (its median spacing when it declares none). kind="all" returns every boundary and kind="gaps" keeps only the ones the tolerance calls gaps. The tolerance is a GapTolerance (dascore.utils.gaps): a bare number or quantity is an absolute excess over one step, GapTolerance.samples(k) counts steps. Spool.chunk, get_gaps, get_coverage, and waterfall(gap_factor=) apply the same predicate, so a GapTolerance gets one verdict wherever it is asked; only the bare-number spelling differs, a sample count to the spool methods and an absolute excess to a coordinate’s get_discontinuities.

fuse(...) and snap(...) make different promises. fuse may refit runs only when no value moves more than the supplied tolerance; snap forces one uniform grid and has no bound on interior movement. With tolerance=None or 0, fuse performs lossless simplification only.

A spool holds a patch whose dimensional coordinates have holes as one row; its index stores the coordinate’s runs (see the spool index), so reports and plans see the holes. patch.split_gaps() cuts such a patch into its contiguous pieces. write(...) hands a patch to its writer whole, which keeps its holes or refuses it.

String coordinates

CoordString is the dedicated coordinate type for string or categorical values. String coordinates do not support units, numeric range operations, relative selection, or slice-style range selection. They do support exact matching, wildcard matching, compiled regular expressions, boolean-mask selection, ordering, and lossy lexicographic min/max summaries.

Plain string selectors reserve * and ? for wildcard behavior. Compiled re.Pattern instances provide explicit regular-expression matching. Any construction path for string coordinates should reject unit-bearing inputs explicitly rather than silently dropping units.

Selection and indexing

Patch.select, Patch.sel, Patch.isel, and Patch.order share the coordinate manager’s _select execution path. The wrappers normalize their queries; existing coordinate select/order methods resolve DASCore queries, while sel resolves labels with pandas’ xarray-compatible rules. All resulting positional indexers go through the same validation, dependent-coordinate updates, dimension reduction, and backend-aware array indexing.

select and order retain dimensions; scalar sel/isel indexers reduce them. Label queries on compact ranges verify the existing coordinate lookup estimates, with batched binary searches as a fallback and a pandas index containing only candidate labels when needed for compatibility. Exact numeric label arrays resolve positions directly without constructing that index. Neither small nor bulk queries expand the input range: lookup memory is proportional to the number of requested labels, and search rounds grow logarithmically with axis length. Grids below floating-point resolution may contain repeated labels; array or nearest lookup rejects ambiguous long grids instead of scanning or materializing them to establish global uniqueness. Positional indexing remains available.

Slice results remain compact for integer, temporal, and floating coordinates, following existing select behavior. Floating slices retain the original grid expression and its length, plus an integer offset and stride. Slicing, striding, and reversal therefore preserve the selected labels bit-for-bit without expanding the coordinate, including through lazy xarray indexing and DASDAE version 2 storage. Scalar and array indexers in sel/isel evaluate only the requested output labels. These bounds apply to the new indexing methods; legacy array-based select/order paths can still materialize ranges. String label slices and exact matching in sel follow pandas’ rules, while coordinate-level select retains DASCore’s wildcard and range behavior.

Summaries are envelopes, not full coordinates

CoordSummary contains min, max, step, dtype, units, associated dims, length, and the coordinate’s exact data_id. The canonical construction path is coord.to_summary(), which starts from a live coordinate; build one with get_coord(...) first when starting from a raw value array.

The summary itself is intentionally lossy. An evenly sampled coordinate has a non-null step and can be reconstructed with summary.to_coord(); an exact grid also carries step_numerator, step_denominator, and origin_offset, so a fractional step rebuilds exactly, and the spool index stores these fields with the summary. Irregular, string, and multi-run coordinates use step=None; their envelopes and ids can support indexing and change detection, but neither field reconstructs their full values.

PatchSummary keeps non-coordinate PatchAttrs, a mapping of coordinate summaries, patch dimensions/shape/dtype, and reload provenance. PatchSummary.flat_dump(...) produces fields such as time_min, time_max, time_step, and time_data_id for dataframes and indexes. Those flattened fields are output records, not supported input for synthesizing a new structured PatchSummary or live coordinate.

Metadata and summary boundaries

Every FiberIO.get_metadata(...) result is a PatchMeta containing attrs, a full CoordManager, dtype, and optionally a logical source key. Dimensions and shape derive from the coordinate manager. The inherited FiberIO.scan uses the same metadata hook.

dc.scan_payloads(...) exposes that metadata and attaches source provenance separately from attrs. snap names the dimensions whose stored labels come back on a grid (rule 3 of the guidelines): by default the format’s own names, shown in its get_metadata signature; True every dimension; False none, every coordinate exactly as stored. Header-defined start/step/count grids are already exact, so the option changes nothing for them. Every mode describes the same written sample extent.

dc.scan(...) consumes the same metadata patches but collapses their coordinates to CoordSummary envelopes, under the same snap. dc.scan_to_df(...) flattens the same summaries for tabular/index use.

This separation keeps directory indexing compact while allowing targeted callers to probe a file’s true coordinate values without loading its data array. Data-less patches can still retain large coordinate arrays, so they should be requested for specific resources and discarded promptly.

DASDAE coordinate storage and scanning

DASDAE stores coordinates separately from patch data. Format version 2, the default when writing, stores each coordinate as it is described: one grid as its start, extent, and step (the exact grid where the coordinate holds one, so a fractional sampling rate rebuilds the same labels) at the cost of a few numbers however long it is, and any other coordinate as an array of values, read back without snapping when it holds several irregular runs. The node tag "CoordRange" is the format’s own name for a grid. A patch with holes is written whole, as its runs. Version 1 files, and file_version="1" writes, store every coordinate as an array beside a whole-tick step, so version 1 refuses a fractional step rather than rounding it, and holes or a step its labels do not restate rather than dropping them. Releases before version 2 existed cannot read a version 2 file; write with file_version="1" for them.

Full reads rebuild those descriptions into a CoordManager exactly as written; DASDAE snaps no dimension unless asked. Scans also construct a real CoordManager, but avoid loading the data array itself; explicit stored start/step metadata remains a range fast path.

String-array serialization is separate from coordinate semantics. HDF5 backends cannot all store NumPy unicode arrays directly, so DASDAE encodes only arrays that are actually string-like and restores them before coordinate construction.

Practical guidance

When changing coordinate or scan internals:

choose tolerant get_coord(data=...) only when idealizing small jitter is acceptable
use truth-preserving construction for any snap=False or exact-value contract
keep run boundaries in live coordinates; a summary states only the envelope, never a coordinate’s runs
treat flat coordinate-summary fields as persisted output, not coordinate reconstruction input
keep string-coordinate rules explicit instead of routing them through generic numeric logic
put shared exact-construction behavior in coordinate or IO helpers instead of duplicating it across formats