Spool Index

Directory and in-memory spools use the same metadata model. The persisted directory index is one SQLite file named .dascore_index.sqlite3; in-memory spools use the same schema in an in-memory SQLite database. The index stores summaries and source identities, not patch data.

Why eight tables?

The schema separates records with different lifetimes and cardinalities. This avoids repeating source and coordinate metadata on every patch and gives SQLite enforceable ownership boundaries.

Table	One row per	Purpose
meta_data	index	Identifies the file and its schema version, and counts its patches
sources	file or directory-format source	Tracks the source path, format, size, modification time, hive-style path attrs (JSON), and presentation ordinal
patches	patch within a source	Stores patch identity, the data array’s type and sample count, and common time/distance envelopes
attrs	patch	Stores typed attribute values in dynamically added columns
attr_meta	attribute name and value kind	Maps original attribute names to typed storage columns and canonical units
coord_defs	unique coordinate value definition	Stores coordinate summaries and deduplicates identical definitions
patch_coords	coordinate attached to a patch, and each run of a segmented one	Links patches to coordinate definitions while retaining the coordinate name and dimensions; run_index 0 is the coordinate as a whole
coord_variants	distinct coordinate name, dtype, kind, units, and relativity	Counts the patches stating each variant, so coordinate discovery reads a few rows rather than every link

coord_defs and patch_coords are deliberately separate. Many patches can share a distance coordinate, so coord_defs stores it once and patch_coords supplies the many-to-many attachment. Range coordinates receive the coordinate’s exact data_id, reconstructed from the coordinate values when a scan did not supply one. Non-range coordinates only receive a merge-compatibility identity when the scan provides one. A coordinate is hashed in the units it was written in, so one length spelled in metres and in centimetres is two definitions — as it must be, since the stored envelopes are in the original spelling too. This distinction prevents a matching envelope from being mistaken for matching coordinate values while retaining the deduplication needed by future merge planning.

coord_variants and meta_data.patch_count are summaries rather than records. Triggers update them in the transaction that changes the rows they count, so every connection keeps them exact, including one writing straight through SQL, and a spool’s coordinate names, kinds and units, and a root spool’s length, cost the same on a large archive as on a small one.

attrs and attr_meta are also complementary. Attribute names are open-ended, so the index cannot define every typed column in advance. attr_meta records the stable mapping and units needed to interpret the columns added to attrs as data is ingested.

Time queries

Absolute-time range queries use the nanosecond time_min and time_max envelopes already stored on patches. Database writes derive these cached time bounds from the coordinate records, including auxiliary time coordinates in planned outputs; duration and numeric time coordinates leave the absolute-time bounds empty. These bounds select candidate patches; exact coordinate trimming still happens when a patch loads. Duration, numeric, string, and other coordinate queries retain the general coordinate-table path; which kinds and units a coordinate has comes from coord_variants.

Lifecycle and validation

The index is an incrementally updated cache. A directory update scans new or changed sources, transactionally replaces their patch rows, and removes rows for deleted sources. Foreign keys cascade source deletion through patches, attributes, and patch-coordinate links. Unreferenced coordinate definitions may remain available for reuse.

The schema version is validated before any mutation. An unrelated or incomplete database raises an error with instructions to delete and rebuild it; DASCore does not silently repair or migrate it. A file that identifies itself as a DASCore spool index of an older schema version is rebuilt automatically by the directory indexer — the index is a disposable cache whose truth is the files. An index written by a newer DASCore is left untouched and raises an error instead: upgrade DASCore, or give the older version its own index_path.

Hive-style path attributes

Stored source paths are parsed for key=value pairs (each directory segment containing the source, with __-separated pairs allowed in a segment). The last segment names the source itself — a file, or a directory-format unit’s directory — and is never parsed: a file extension is indistinguishable from a value ending in one, which the acquisition_key=XX.R2D1..RAW partition in the example below would be if it named a source. The pairs are merged into every patch’s attrs at ingest as plain string values — the path wins over a same-named file attr, since renaming a path is the user’s way of correcting metadata. The applied dict is also persisted on the source row (sources.path_attrs, JSON) and surfaced privately as _path_attrs in the flat relation: derived and union catalogs absolutize paths, so provenance must ride with the row for patch loading to stamp the attrs and for moves to rewrite them. Reserved/underscore key names follow the same skip rules as file attrs; values are never type-inferred.

Renames are detected during a directory update by exact (mtime_ns, size_bytes) stat identity (the rename-invariant manifest signature for directory-format units) between a disappeared and a brand-new path, each unique on its side. A detected move rewrites the source path and its path-derived attr values in SQL without re-reading file contents, so renaming a partition directory over a large archive is cheap. A rename that removes a hive key falls back to a rescan — the file’s own value for that attr is only recoverable by reading it — and ambiguous stat matches also rescan rather than guess.

import tempfile
from pathlib import Path

import dascore as dc

root = Path(tempfile.mkdtemp())
(root / "acquisition_key=XX.R2D1..RAW" / "cable=A").mkdir(parents=True)
sub = root / "acquisition_key=XX.R2D1..RAW" / "cable=A"
dc.get_example_patch().io.write(sub / "das_file.h5", "DASDAE")

spool = dc.spool(root).update()
row = spool.get_contents().iloc[0]
assert (row["acquisition_key"], row["cable"]) == ("XX.R2D1..RAW", "A")
assert spool[0].attrs.get("cable") == "A"

# a directory rename is a pure metadata rewrite
sub.rename(sub.parent / "cable=B")
assert dc.spool(root).update().get_contents()["cable"].iloc[0] == "B"

Ordering

The base ordering contract is (sources.ordinal, patch_row): ordinals are assigned at ingest, a replaced source keeps its position while new sources append, so merging catalogs concatenates and duplicate sources keep their first-occurrence position with last-occurrence metadata (dict-merge semantics). Spools built from in-memory patches therefore iterate in construction order on every path.

Directory catalogs additionally carry a per-patch default presentation order: rows present by time (time IS NULL last, then time, with ordinal and patch row as deterministic tiebreaks), because source-grain ordinals alone cannot interleave a multi-patch file whose span straddles a patch of another file. The default order is a catalog contract, not view state — a directory root still updates — and an explicit sort(...) replaces it. The directory indexer still renumbers source ordinals to time order (earliest patch per source, path as tiebreak) after each sync, which keeps the ordinal grain stable for replacement and union dedup.

Sorting by a coordinate stated under several value kinds, such as numbers in some patches and text in others, orders each kind’s values among themselves in every view: numbers, then text, then times.

SQLite permits concurrent readers and serializes writers. Initialization and updates use an immediate write transaction and a 30-second busy timeout. This relies on correct local-filesystem locking; reliable operation on network filesystems with weak locking is not promised.

Bounded metadata access

For an indexed spool, the first integer access projects the selected patch’s metadata row, and forward contiguous slices limit the membership query in SQL. Use spool[:10].get_contents() for a small preview. A second integer access on the same view realizes and caches its metadata frame, so repeated positional reads do not repeatedly pay for SQL and dataframe setup. An update resets this behavior. Regex predicates and coordinate selections whose residual operations can remove candidate rows still require exact filtering before a positional limit can be applied.

split() fetches the ordered candidate membership once and reuses it for every batch. A batch can be empty after exact filtering; candidate rows are not discarded merely because a refined count is smaller. Chunk planning and full iteration continue to realize metadata when their existing execution paths require it.

The flat relation

Spool-facing operations consume the tables through one flat relation: a dataframe with one row per patch carrying {dim}_min/max/step envelopes, private structural columns (_patch_row, _{name}_def_key coordinate identities, and _{name}_units original unit spellings, normalized per dimensionality by the chunk planner before partitioning), the dims signature, and one column per attribute. Attrs and coords are independent namespaces, so an attr may share a name with a coordinate envelope column (e.g. an attr channel_step alongside a channel coordinate); in that rare genuine collision the envelope column owns the flat name, the attr column is omitted with a warning, and the attr remains queryable through the _attrs namespace. The private columns are the index’s own bookkeeping, so the relation a caller receives from get_contents (and from get_gaps and get_coverage) is the public one. present_columns builds it from an explicit list of steps (dascore.utils.pd.PRESENTERS): each step gives one private column back its public spelling — the unit spellings become {name}_units, _dtype and _data_size become dtype and data_size — and the last step drops every underscore-prefixed column that is left, so a column no step claims never leaves.

_dtype and _data_size (the samples in the data array, the product of its shape) are private for the same reason: chunk’s merge-compatibility grouping and conflict policing compare every public column, and patches of different element types — or, far more often, different lengths — must still merge. A row states no size when it does not know one. A patch assembled from several, a piece cut from one, and a row a selection trims all describe a patch which does not exist yet, and how many samples it comes out with is known only once it is loaded: those rows leave data_size null rather than repeating a member’s count. An output which is one whole, untrimmed member is that patch, so it keeps that patch’s size, and so does a row a selection leaves whole. Only a selection the presented envelopes cannot describe — sample indices, or a selector which is not a range — blanks every row it presents, since nothing then marks which rows it cut. The chunk planner and selection both operate on the full relation (see the Spool Chunking and Spool Selection notes).

A patch’s own origin_id and data_id are ordinary indexed attrs, so a patch can be found by the id it carries without loading anything. The storage schema’s row number, patch_row, is renamed _patch_row before attr columns are applied, and patch_row is a reserved attr name, so no attr can collide with it. Neither id takes part in chunk’s merge compatibility: every patch states a different one, and a merged patch derives its ids from its members’ (a Merge operation over their data_ids, in order) rather than inheriting any one of them.

The cell below runs against the real catalog so this description cannot silently drift.

import dascore as dc
from dascore.io.index.catalog import PatchCatalog

catalog = PatchCatalog.from_patches(list(dc.get_example_spool("random_das")))
df = catalog.to_df()
required = {
    "_patch_row",
    "origin_id",
    "_time_def_key",
    "_time_units",
    "_dtype",
    "_data_size",
    "dims",
    "time_min",
    "time_max",
    "time_step",
}
assert required.issubset(df.columns)

Patch identity and spool set semantics

A spool of in-memory patches has set semantics by patch identity: constructing a spool from a sequence keeps one entry per distinct patch instance, and spool + spool unions membership. Each patch carries an instance identity minted eagerly at construction; because patches are immutable, copies (including deep copies and unpickled patches) share that identity, while every patch operation produces a new instance with its own. The identity is the patch’s synthetic memorypatch:// source path in the index, so identity, deduplication, and resolution all agree:

import copy

patch = dc.get_example_patch()

# a sequence with the same instance twice is one spool entry
assert len(dc.spool([patch, patch])) == 1

# copies share identity regardless of when they are made — identity is
# eager, so there is no access-order dependence
clone = copy.deepcopy(patch)
assert clone._instance_id == patch._instance_id
assert len(dc.spool([patch, clone])) == 1

# any operation mints a new, distinct identity
assert len(dc.spool([patch, patch.new()])) == 2

# the identity is the row's synthetic path, and it resolves back to the
# very same object
cat = PatchCatalog.from_patches([patch, patch])  # one entry, not two
row = cat.to_df().iloc[0]
assert len(cat.to_df()) == 1
assert row["source_path"].startswith("memorypatch://")
assert cat.resolve_row(row.to_dict()) is patch

File-backed patches are identified by (base_uri, source_path, source_patch_key) instead; their rows resolve through readers rather than the registry.

Federation

spool + spool merges catalogs table-to-table: source records are reconstructed from the member backends and re-ingested, so coordinate definitions deduplicate by definition key, the same source appearing in several members keeps a single entry, and file paths are absolutized so members with different roots coexist. A composite resolver routes in-memory rows to the shared registry, plan-output rows to their plan resolvers, and everything else through file readers.

Row-membership state (attribute predicates, slice windows, patch-row arrays) transfers as rows, preserving the set semantics above. State that only lives Python-side — coordinate/samples residual trims, sort specs, and a directory default order that record-grain transfer would actually scramble — is first baked into an identity-plan derived catalog (table work only; no patch data loads), so each operand contributes exactly its current contents in its current order. Baking mints new patch identities, so a trimmed operand no longer deduplicates against its source: its contents genuinely differ. Spool equality follows the same philosophy — it compares effective contents (residual trims folded into the envelopes, representation artifacts like def keys and backing excluded), so a trimmed view equals its union-materialized twin.

other = dc.get_example_patch().new()
union = dc.spool([patch]) + dc.spool([other])
assert len(union) == 2
assert len(dc.spool([patch]) + dc.spool([patch])) == 1  # same patch dedups

# A trimmed operand contributes its trimmed contents (baked, not dropped),
# and equality compares those effective contents.
t = patch.get_coord("time")
trimmed = dc.spool([patch]).select(time=(t.min() + 10 * t.step, t.min() + 20 * t.step))
combined = trimmed + dc.spool([])
assert combined[0].shape == trimmed[0].shape
assert combined == trimmed

Scope

The index answers metadata selection, identifies candidate patches, and plans chunk operations — all without touching patch data. Exact coordinate selection is applied again when a patch is loaded because summary envelopes cannot prove arbitrary-coordinate membership. Materialized derived data (persisting chunked results) remains future work.

The index retains source data_id values, original numeric attribute and coordinate dtypes, and whether scalar attributes fully describe a source patch (history remains excluded). Temporal dtype names follow the existing summary normalization and temporal envelopes use nanoseconds. Coordinate names are retained even when their values have no supported envelope representation; such names are not selectable. This metadata lets consumers distinguish rows they can reconstruct from those that require loading the source patch. Plan output rows require resolution through their members. Index version 15 added this reconstruction metadata; version 16 stores each range coordinate’s exact grid (step_numerator, step_denominator, origin_offset, with length authoritative for the sample count), marks with is_exact the rows which rebuild every value of their coordinate, and names the envelope columns by storage type (min_int/max_int/step_int for nanosecond times and durations, min_float/max_float/step_float for numbers, min_str/max_str for text). Version 17 links a segmented coordinate to each of its runs as well as to itself: patch_coords.run_index is 0 for the coordinate as a whole, which every query about a patch’s coordinate reads, and 1, 2, … for its runs (up to 256; a coordinate with more is indexed as a whole only). Gap reports and chunk planning read runs through a partial index holding just those links, so an archive of contiguous patches pays nothing for them; chunk plans each run as a trimmed member, and plans and merge exports carry runs along. Version 18 adds coord_variants and meta_data.patch_count. Version 23 records a coordinate’s dtype with the unit it counts in, so a row rebuilds the coordinate the file holds rather than one in nanoseconds. Version 24 identifies a coordinate by its runs, so an index written before it states other ids for the same labels. Older indexes are rebuilt when opened.