Coordinate Guidelines
These rules govern DASCore’s coordinates. If code disagrees with them, fix the code, or change this note first in its own pull request and log why.
Not yet on dev. The 2026-11-12 simplification (see the decisions log) is being implemented in stages; until each lands, the code differs from these rules here (delete each line as its PR merges):
- windows leaving
chunkforSpool.cut, andon_incompletegoing (rules 8, 9); - independent segments from
concatenateand sample removal, equality by labels, the multi-segment refusal,fill_gaps’ half-step move,Spool.fill_gaps, andfill_valueleavingchunk(rules 1, 4, 7, 10); - removing the
gappedandjitteredflags, hole positions,Missing,fuseand the run table’spositioncolumn (rules 2, 3, 11, 12, 13); - predating 2026-11-12:
Spool.snap_coords,snap_coords(max_shift=)andsnap_coords(step=)(rule 10).
The model
A coordinate is one evenly sampled grid, one list of labels, or several of those in order, called segments. Nothing is moved across the boundary between two segments. chunk joins patches that continue one another and splits where they don’t, so every output is one segment. concatenate joins patches in order into one patch and keeps the boundaries between them. Only when asked, fill_gaps joins a coordinate’s segments onto one grid across gaps up to the size it is given, and split_gaps returns each segment as its own patch. time below is only an example; no dimension name is special.
whole = spool.concatenate(time=None) # one patch; gaps kept
filled = whole.fill_gaps(time=10) # gaps up to 10 s filled
pieces = spool.chunk(time=None) # one patch per stretchScope
The sampling rules apply to one-dimensional, monotonic, numeric or time-like (datetime or timedelta) dimension coordinates; others have no step.
Associated coordinates run along a dimension without being one. Whatever an operation does along a dimension, it does to them; if it can’t, it drops them with a warning (fill_gaps gives them NaN or NaT where it can). Selecting by a multi-dimensional one raises and suggests a mask. Units always survive.
Terms
- Step: the spacing the labels follow. Integer and datetime steps are exact fractions of a tick, so rates like 1024 Hz are exact (labels are grid values rounded down to whole ticks). Float grids reproduce
linspacebit for bit. - Grid: a start, a step and a count; evenly sampled labels held as numbers.
- Labels: a stored array of values, with no step.
- Segment: one grid or one list of labels; a coordinate is one segment, or several in order.
- Seam: where one patch, or segment, ends and the next begins. Its width is the distance from the last label before it to the first after it, in steps; contiguous samples meet at a seam one step wide.
- Gap: a seam wider than the seam test allows (rule 5).
- Stretch: patches joined across seams that pass the seam test, which
chunkreturns as one segment.
coord.step is set, and coord.evenly_sampled is true, only for a coordinate of one grid. A coordinate of one list of labels is irregular.
Rules
Labels
Labels are what was recorded. Only
snap_coords,fill_gaps, a reader’ssnap(rule 3) andchunk’s grid for a stretch (rule 6) change them; storing a label as a grid value within float precision doesn’t count. Equal labels mean equal coordinates, however they are split into segments. A grid that starts exactly one step after the grid before it ends, on the same step, is part of that grid. Segments arise only fromconcatenate(rule 7) and from removing samples from the middle of a coordinate (a mask, an index array); each remaining piece is its own segment on the original step, and only an explicit stride gives a coarser step. A format that stores segments reads them back (rule 3).Inference finds a grid or labels.
get_coord(data=...)checks every label against a fitted grid, not just the spacings, so drift can’t hide: labels all on the grid at consecutive positions are that grid; anything else is labels, with no step. Inference never makes segments. With a stated step, labels on its grid at consecutive positions are that grid. Integer and datetime grids are found exactly in their own tick unit; floats are judged in their stored dtype.Readers snap the coordinates
snapnames.dc.read,dc.scan,dc.spooland every reader takesnap:None(default): the format’s default names, shown in its reader’s signature: the dimensions it stores as labels restating a grid (none where a header states every grid; DASDAE snaps only stored labels withinmax_shiftof a grid; DASCore’s other formats snap none, so what they wrote reads back);True: every one-dimensional numeric or time-like dimension;False: nothing; every coordinate exactly as stored;- a tuple such as
("time",): those names; a missing or unsnappable name raises.
For each coordinate it snaps, a reader returns a grid when the stored labels are within
max_shiftof one, otherwise the labels as stored. Readers never split a coordinate into segments, returning segments only where a file stores them; gaps show up between files, wherechunksees them. One shared helper builds every snapped grid, from a reliable stated start and acquisition rate, else from the first and last stored labels and the count (a whole number of ticks where that keeps the ends within their stored precision, else an exact fraction of one), so interrogators with unlocked clocks still come back evenly sampled. It checks the grid against stored labels (the ends, plus a few interior ones for end-point grids). Where one sits further thanmax_shiftoff it, beyond what its stored precision resolves, the file holds a gap or a clock running apart, so the helper keeps the stored labels and warns, naming the file. The index records itssnapand rescans when opened with another. A conformance test holds every format to this rule.Operations check what they need. Operations that compute across samples refuse a coordinate of more than one segment, naming
fill_gapsandsplit_gaps. FFTs, filters and others needing a step useget_coord(dim, require_evenly_sampled=True), which also refuses labels and namessnap_coords. One list of labels is an irregular axis thattaper,sobel_filter, filtereddecimate,differentiateandintegratecompute on by its labels.
Joining
max_shiftis the seam test. Two patches join when the later starts one step after the earlier ends, give or takemax_shiftand the precision the labels are stored in; the step is the earlier patch’s. A wider seam is a gap; a narrower one is an overlap, which rule 6 trims before the seam is judged.max_shiftbounds each seam, not how far labels move in total.tolerance, off by default, widens the test as released: a seam up totolerancesteps wide also joins. As a quantity or timedelta, which v0.1.24 did not take,toleranceis that width as a distance; pieces with no step join where they overlap or touch, and across a seam only under that form.chunkjoins and splits. It never fills, and every output is one segment.- It joins compatible patches (
notes/patch_compatibility.qmd) whose steps agree withinsampling_group_tolerance, along the coordinate’s direction, judging each seam, including those between a patch’s segments, from the recorded labels on either side. A gap starts a new stretch. A single sample takes its neighbours’ step. A stretch holding a piece with no step keeps every label as recorded and has no grid. - Overlaps. Each sample is kept once and the earlier patch wins. A later patch that starts sooner than the seam test allows continues from the first of its samples that lies far enough past the earlier patch’s end (one step less
max_shift) and joins under rule 5; its earlier samples are dropped. If no sample does, its samples past the earlier patch’s end start a new stretch. - Grid. Each stretch gets one grid, fitted before windows are cut, so chunk outputs rejoin exactly. It starts at the first patch’s first label. Its step is the roundest step any patch states (fewest significant digits in the rate or the step; ties go to the earliest patch) that fits: numbering the stretch’s samples consecutively, every patch’s first and last label must lie within
max_shiftof its numbered position on the grid. If no stated step fits, the step spreads the stretch evenly from its first label to its last, as v0.1.24 does where the stated steps agree. Only a spread needing a fractional step under 100 ticks gives the stretch no grid: it keeps its recorded labels as one list, as a stretch holding a piece with no step does. A spread too fine for int64 ticks takes the closest step they can number, so time stretches of any practical length get a grid. A seam joined because oftoleranceis closed by the same fit, and one wider than 1.5 steps warns, as released. - After a file ending at 99 s with a 1 s step, one starting at 100.1 s joins it, and one starting at 101.3 s starts a new stretch unless
toleranceis at least 2.3. chunk_planrecords per piece why each output starts, how far the fit moved its labels, and how many samples were dropped.
- It joins compatible patches (
concatenatejoins exactly, in order, into one patch, keeping every label; it never moves, splits or fills. Each patch it joins stays its own segment unless it continues the one before exactly (rule 1), so every gap is kept as a boundary. Overlapping labels give an unordered coordinate and a warning.
Selecting and windows
- Ranges and windows.
- Picking by value (
select,sel, ranges, annotations) includes both ends; positions (isel,samples=True) and windows exclude the end, as in xarray. Reversed bounds are swapped; on a descending coordinate, start and end mean the lower and upper values. chunktakes lengths only, in every form v0.1.24 took, or as a data size such astime=25 * megabytes;Nonekeeps each stretch whole.chunk(dim=length, overlap=o)starts windows atstart + k * (length - o), measured from the stretch’s first label, and each holds the samples whose labels on the stretch’s grid fall inside it, so counts may differ by one; negative overlap skips;samples=Truecounts samples.Spool.selectaccepts ranges,(n, 2)arrays of ranges, or annotation sets, and returns each source’s matching part, unjoined; arrays on several dimensions pair row by row.Spool.cutowns explicit windows, joining each aschunkjoins, into one output or labelled fragments if it splits. OnPatch.selectan array still lists values to pick, as released.
- Picking by value (
- Completeness applies to
chunkwindows only. A window running past the end of its stretch is dropped unlesskeep_partial=True, as released; windows of a stretch with no step are kept.selectjust clips.
Gaps
fill_gaps,split_gapsandsnap_coords.fill_gaps(**{dim: size})joins segments onto one grid, on the earlier segment’s step, across gaps up to that size, in the coordinate’s units (all gaps, if none is given); a larger gap stays a segment boundary. It insertsfill_value(NaN) samples into the gaps it joins and moves the later segment onto the grid, by at most half a step, reporting the move. It raises when the later segment’s samples do not fit consecutive positions of that grid within half a step (a different step, or none) or the dtype can’t holdfill_value.Spool.fill_gapsalso fills the gaps between its patches, so a laterchunkjoins across them.split_gaps()returns one patch per segment.snap_coords(*dims)works as released: sort, keep the end labels and the count, spread samples evenly; evenly sampled coordinates are left alone.max_shift=raises on larger moves, and moves over one step warn, since a gap is being spread.step=places labels on that grid instead.- Without names they apply to every eligible dimension. On spools they are lazy, using the index where possible and otherwise reading only the coordinate;
snap_coordsacts on each patch alone.
Reports and storage
Reports match
chunk.get_gapslists wherechunkwith the same arguments splits, with span and missing-sample count.get_coveragecounts missing samples (unknown without a step).get_contentsgives{dim}_steponly for a coordinate of one grid.The index mirrors the coordinate: one row per patch, a coordinate table (step, envelope, segment count) and a run table with one row, called a run, per segment of each coordinate that has more than one segment; single-segment coordinates have no runs. Gap logic is written once over runs, so memory and index agree. Undecidable cases read the coordinate lazily. Beyond
max_index_runs, a coordinate is stored as an envelope with a warning naming the file and the fix.Writers keep steps and segments, or refuse, saying to fill or split first. They never silently split, stretch, fill or drop a step.
Constants
These are the only thresholds.
| Name | Value | Use |
|---|---|---|
dascore.config.max_shift |
0.2 steps | The seam test and grid fit (rules 5, 6) and the reader check (rule 3): above measured jitter (0.17), well short of a missing sample; always under 0.5. snap_coords has no default limit. |
tolerance warning |
1.5 steps | The released default: a wider seam joined by tolerance warns (rule 6). |
dascore.config.sampling_group_tolerance |
5% | How far the steps of patches in one stretch may differ (rule 6). |
dascore.config.max_index_runs |
256 | Runs stored per coordinate in the index. |
| Float exactness | max(_GRID_RTOL × |step|, 4 ulp) |
In the stored dtype; float64 holding float32 values counts as float32. |
| Fractional steps | ≥ 100 ticks | So an integer array missing one value isn’t read as a fractional grid. |
Methods that change coordinates
| Job | Patch | Spool | Notes |
|---|---|---|---|
| Pick | select, sel, isel, unselect |
select, unselect |
select for compatibility, sel/isel for xarray users. |
| Reorder | order, sort_coords |
sort |
|
| Change coordinates | update_coords |
All changes go through it, including coords_from_df, add_distance_to, enrich. |
|
| Rebuild | replace |
Everything at once; update and new remain. |
|
| Remove, rename | drop_coords, rename_coords |
drop_coords(to_attrs=True) replaces squeeze_coords. |
|
| Assemble | chunk, concatenate, cut |
chunk by length, cut by window, concatenate keeping gaps. |
|
| Split at gaps | split_gaps |
dc.spool([patch]) no longer splits. |
|
| Regularize | snap_coords, fill_gaps |
snap_coords, fill_gaps |
|
| Sum patches | stack |
Along dim_vary, keeps the first patch’s labels. |
Processing that changes a grid (resample, decimate, dft) states the step it produces.
Deprecations and breaks
Released behaviour is deprecated with a warning for at least one release rather than changed outright, unless there is a compelling reason, which the decisions log records. The deprecations below that promised a later switch take effect in 1.0, the release after v0.1.24 (see the decisions log). Unreleased features change freely, and removing one is not a deprecation; they include Spool.cut, squeeze_coords, sampling_group_tolerance, half-open annotation ranges, chunk’s fill_value, explicit windows, on_incomplete, group and on_missing_dim, the gapped and jittered flags, Missing, fuse and the run table.
| In v0.1.24 | Deprecation |
|---|---|
chunk joining any seam up to tolerance=1.5 steps wide by default and spreading joined labels evenly |
Switched in 1.0 to rules 5 and 6: a later patch more than max_shift off the earlier one’s grid, such as one starting 0.5 or 1.3 steps after it ends, joins only under tolerance, and a stretch keeps a stated step that fits; tolerance=1.5 restores the released joining |
chunk(snap_coords=...) |
Warns and is ignored from 1.0: chunk always fits one grid per stretch (rule 6); use Spool.concatenate to keep recorded labels |
get_coord(data=...) snapping within 0.1% |
Switched in 1.0: labels without a stated step are read by rule 2; snap=True keeps the released snap |
taper, sobel_filter, filtered decimate, differentiate and integrate on coordinates which are not evenly sampled |
Raise CoordError from 1.0 for more than one segment; one list of labels is an irregular axis and still computes by its labels |
order dropping missing values |
Raises CoordError from 1.0 |
coords_from_df overwriting coordinates |
Raises ParameterError from 1.0 |
concatenate_patches, merge_patches |
Removed in 1.0; use Spool.concatenate and Spool.chunk |
Terra15 read(snap_dims=...) |
Warns; means snap |
Spool.stack relabelling dim_vary to start at zero |
Keeps the first patch’s labels from 1.0 |
From 1.0, a coordinate of several segments refuses filters until filled or split, and writing segments to formats that can’t store them raises. Some reader changes are not deprecated first (see the decisions log): stored labels a grid disagrees with are kept, with a warning, rather than stretched onto it (Terra15 v5 and NeubrexRFS times among the registry files); NeubrexRFS times and Sentek distances read as stored; and Febus G1 sweep times and NeubrexRFS distances snap.
Future work
Coordinates will hold a mapping of keys to any array-like, real or virtual, carried through snap_coords and similar so recorded labels survive. Not yet designed.
Open questions
Not yet decided:
- whether one list of labels and the same labels as segments, which rule 1 makes equal, should chunk and compute alike (rules 4, 6), so that
chunklooks for gaps inside a list of labels; - whether a stretch whose stated steps differ by more than 0.1% is spread evenly or keeps its labels, as v0.1.24 kept them (rule 6);
- whether a stated step whose labels skip grid positions gives segments or raises (rules 2, 10);
- whether
groupstays inchunk’s public signature; - whether
max_shiftalso takes a unit form (it is a fraction of a step for now).
Decisions log
- 2026-10-04: Assembly and regularization are separate; a design with join modes proved too complex.
- 2026-10-04: An explicit
tolerancejoins across gaps, and patches may hold holes, reversing those parts of #1236. - 2026-10-05: The index returns to a run table, reversing #1284, so memory and index share one model.
- 2026-10-05:
snap_coordskeeps its released behaviour, addingmax_shift(after silent large moves in #803, #896, #1418) andstep. - 2026-10-05: Boundaries follow xarray: picks include both ends, positions exclude the end.
- 2026-10-06: Selection keeps the step (a draft that coarsened it erased real holes); completeness checks edge positions (a draft using
tolerancethere let truncated windows pass). - 2026-10-06: A prototype on ~5,800 Febus files set
max_shiftto 0.2 (half a step let a missing sample snap) and seams by recorded labels (projecting from the first piece dropped real samples). Filed #1419, #1420. - 2026-10-07: Readers assume no holes inside a file (clocks may be unlocked from sampling);
snap=Nonemeans the format’s defaults. - 2026-10-07: Jitter and holes are independent flags, replacing a level ladder;
require_evenly_sampledstays. - 2026-10-07:
max_shift=0replaces asnap_jitterflag. - 2026-10-07: Released behaviour is deprecated rather than changed outright, unless there is a compelling reason.
- 2026-10-08: The released
get_coord(data=...)snap is not kept where its median step moves a label half a step or more, which relabels samples (#1419): an exact grid is used, jittered labels are snapped from their end labels and count, and others get no step. Where rule 2 itself finds a grid (a fractional step, or labels within float exactness), that grid is used without a warning: the coordinate stays evenly sampled, and only labels the snap had moved return to their recorded values. - 2026-10-09: A format’s default names are the dimensions it stores as labels restating a grid. Sentek’s distances are sensor positions 1.94 to 2.85 m apart, which the released reader moved up to 0.78 m onto a grid one step short, so they read as stored. Its repeated times are float32 rounding of the 27.264 ms grid its header states, and snap without a warning, so Sentek time stays ordered rather than becoming unordered as first written. DASDAE snaps only coordinates it stored jittered: near-even labels snap as released (filters still run on such files) and the holes and irregular labels it wrote read back as written.
- 2026-10-09: The reader check allows the precision labels were stored in (one unit in the last place, in their stored float dtype): a grid spanning two rounded labels inherits their rounding. A spanning step is a whole number of ticks where it keeps the ends within that precision, as released readers gave it, and an exact fraction otherwise, since whole ticks drift past
max_shiftover an hour at 1 kHz. - 2026-10-10: Where the check fails, the reader keeps the stored labels and warns, rather than returning a grid that moves them further: a ten-sweep gap inside a Febus BSL file moved labels 25 s onto a grid. Unlocked clocks, measured at 0.17 steps or less, still snap. Not deprecated first: the released grids for Terra15 v5 (1.3 steps) and NeubrexRFS DSS (2.0) and DTS (9.2) times relabel samples.
- 2026-10-10: NeubrexRFS time is not a default name: its labels are irregular measurement times (minutes to days apart), which v0.1.24 moved by up to 50 h onto a grid. Febus G1 sweep times and NeubrexRFS float32 distances, which v0.1.24 kept as stored (their jitter exceeds its 0.1%), now snap by default within
max_shift;snap=Falsekeeps them. - 2026-10-11:
Spool.stackrelabelleddim_varyto start at zero (1970 for times) in v0.1.24, which rule 1 forbids and the methods table did not record; it warns for one release, then keeps the first patch’s labels. - 2026-11-06: 1.0 is the release after v0.1.24, so the switches promised for the next release take effect in 1.0 rather than after a release of warnings, as pre-v1.0 review findings s01-F3 and s09-F2 asked:
orderand the operations needing even sampling raise,coords_from_dfrefuses to overwrite,stackkeeps the first patch’s labels, andconcatenate_patchesandmerge_patchesare removed.get_coord(data=...)reads labels without a stated step exactly by default; a stated step still absorbs float rounding and a zero-dimensional array still makes a scalar coordinate, since reading those exactly broke selection and coordinate construction well beyond the warning. - 2026-11-10: The 1.0 switch for the operations needing even sampling (
taper,sobel_filter, filtereddecimate,differentiate,integrate) refuses only holes and jitter, as the check refused only holes before #1427. A coordinate with no step, such as irregular sensor positions, stays an irregular axis they compute on by its labels, as in 0.1.x, so the 2026-11-06 switch does not remove that released support. - 2026-11-12: Coordinates and
chunkare simplified, after two rounds of review found the design too complex to maintain. Reversed: holes on a shared grid (2026-10-04) and thejitteredandgappedflags (2026-10-07) give way to independent segments, with no hole positions,Missingorfuse, and the run table loses itspositioncolumn; DASDAE’s default snap is judged bymax_shiftrather than a stored flag (2026-10-09);max_shiftbecomes the seam test rather than a bound on shifting a piece (2026-10-06);chunkagain fits one grid per stretch, and an explicittolerancecloses a gap by refitting, as released, rather than keeping holes (2026-10-04). Windows move toSpool.cutandfill_valuetoSpool.fill_gaps;on_incompleteandjoin_dimgo withchunk’s windows. Measured ondev: ten 1,000-sample files each 0.15 step late joined into one patch with labels moved 1.35 steps; a two-grid coordinate and the same labels as one array compared unequal, andchunk(time=None)split one ([2000, 2000]) but not the other ([4000]);Spool.concatenatereturned labels, across whose hour-long gap filtereddecimateran silently; and mask and index-arrayselectalready returned two grids. A bound on total label movement was rejected, since it would split a day of drifting one-minute files into hundreds of patches. The default seam slack falls from 0.5 step (tolerance=1.5) to 0.2 (max_shift), so a later patch more than 0.2 step off the earlier one’s grid, as across a seam 1.2 to 1.5 steps wide, stops joining unlesstoleranceis given. The other removals were unreleased. - 2026-11-12:
chunk’s released joining switches in 1.0 without a release of warnings first, as the 2026-11-06 switches do, so seams 1.2 to 1.5 steps wide and later patches more thanmax_shiftoff the earlier one’s grid stop joining by default;tolerance=1.5restores the released joining.snap_coordsis a method, not achunkflag:chunk(snap_coords=...)warns and is ignored, sincechunkalways fits one grid per stretch,Spool.concatenatekeeps recorded labels, andPatch.snap_coordsregularizes them. The only stretch that keeps its labels for want of a grid is one needing a fractional step under 100 ticks (or holding a piece with no step), which it keeps as one list; a spread too fine for int64 ticks takes the closest step they can number, so time stretches of any practical length get a grid.