Spool Index

Directory and in-memory spools use the same metadata model. The persisted directory index is one SQLite file named .dascore_index.sqlite3; in-memory spools use the same schema in an in-memory SQLite database. The index stores summaries and source identities, not patch data.

Why seven tables?

The schema separates records with different lifetimes and cardinalities. This avoids repeating source and coordinate metadata on every patch and gives SQLite enforceable ownership boundaries.

Table	One row per	Purpose
meta_data	index	Identifies the file and its schema version
sources	file or directory-format source	Tracks the source path, format, size, modification time, hive-style path attrs (JSON), and presentation ordinal
patches	patch within a source	Stores patch identity and common time/distance envelopes
attrs	patch	Stores typed attribute values in dynamically added columns
attr_meta	attribute name and value kind	Maps original attribute names to typed storage columns and canonical units
coord_defs	unique coordinate value definition	Stores coordinate summaries and deduplicates identical definitions
patch_coords	coordinate attached to a patch	Links patches to coordinate definitions while retaining the coordinate name and dimensions

The last two tables are deliberately separate. Many patches can share a distance coordinate, so coord_defs stores it once and patch_coords supplies the many-to-many attachment. Range coordinates receive an exact semantic fingerprint, reconstructed from the coordinate values when a scan did not supply one. Non-range coordinates only receive a merge-compatibility identity when the scan provides an exact fingerprint. This distinction prevents a matching envelope from being mistaken for matching coordinate values while retaining the deduplication needed by future merge planning.

attrs and attr_meta are also complementary. Attribute names are open-ended, so the index cannot define every typed column in advance. attr_meta records the stable mapping and units needed to interpret the columns added to attrs as data is ingested.

Lifecycle and validation

The index is an incrementally updated cache. A directory update scans new or changed sources, transactionally replaces their patch rows, and removes rows for deleted sources. Foreign keys cascade source deletion through patches, attributes, and patch-coordinate links. Unreferenced coordinate definitions may remain available for reuse.

The current schema version is validated before any mutation. An unrelated or incomplete database raises an error with instructions to delete and rebuild it; DASCore does not silently repair or migrate it. A file that identifies itself as a DASCore spool index of a different schema version is rebuilt automatically by the directory indexer — the index is a disposable cache whose truth is the files.

Hive-style path attributes

Stored source paths are parsed for key=value pairs (each directory segment, and __-separated pairs in any segment including the extension-stripped file name). The pairs are merged into every patch’s attrs at ingest as plain string values — the path wins over a same-named file attr, since renaming a path is the user’s way of correcting metadata. The applied dict is also persisted on the source row (sources.path_attrs, JSON) and surfaced privately as _path_attrs in the flat relation: derived and union catalogs absolutize paths, so provenance must ride with the row for patch loading to stamp the attrs and for moves to rewrite them. Reserved/underscore key names follow the same skip rules as file attrs; values are never type-inferred.

Renames are detected during a directory update by exact (mtime_ns, size_bytes) stat identity (the rename-invariant manifest signature for directory-format units) between a disappeared and a brand-new path, each unique on its side. A detected move rewrites the source path and its path-derived attr values in SQL without re-reading file contents, so renaming a partition directory over a large archive is cheap. A rename that removes a hive key falls back to a rescan — the file’s own value for that attr is only recoverable by reading it — and ambiguous stat matches also rescan rather than guess.

import tempfile
from pathlib import Path

import dascore as dc

root = Path(tempfile.mkdtemp())
(root / "acquisition_key=XX.R2D1..RAW" / "cable=A").mkdir(parents=True)
sub = root / "acquisition_key=XX.R2D1..RAW" / "cable=A"
dc.get_example_patch().io.write(sub / "tag=raw.h5", "DASDAE")

spool = dc.spool(root).update()
row = spool.get_contents().iloc[0]
assert (row["acquisition_key"], row["cable"], row["tag"]) == ("XX.R2D1..RAW", "A", "raw")
assert spool[0].attrs.get("cable") == "A"

# a directory rename is a pure metadata rewrite
sub.rename(sub.parent / "cable=B")
assert dc.spool(root).update().get_contents()["cable"].iloc[0] == "B"

Ordering

The base ordering contract is (sources.ordinal, patch_id): ordinals are assigned at ingest, a replaced source keeps its position while new sources append, so merging catalogs concatenates and duplicate sources keep their first-occurrence position with last-occurrence metadata (dict-merge semantics). Spools built from in-memory patches therefore iterate in construction order on every path.

Directory catalogs additionally carry a per-patch default presentation order: rows present by time (time IS NULL last, then time, with ordinal and patch id as deterministic tiebreaks), because source-grain ordinals alone cannot interleave a multi-patch file whose span straddles a patch of another file. The default order is a catalog contract, not view state — a directory root still updates — and an explicit sort(...) replaces it. The directory indexer still renumbers source ordinals to time order (earliest patch per source, path as tiebreak) after each sync, which keeps the ordinal grain stable for replacement and union dedup.

SQLite permits concurrent readers and serializes writers. Initialization and updates use an immediate write transaction and a 30-second busy timeout. This relies on correct local-filesystem locking; reliable operation on network filesystems with weak locking is not promised.

The flat relation

Spool-facing operations consume the tables through one flat relation: a dataframe with one row per patch carrying {dim}_min/max/step envelopes, private structural columns (_patch_id, _{name}_def_key coordinate identities, and _{name}_units original unit spellings, presented publicly as {name}_units by get_contents and normalized per dimensionality by the chunk planner before partitioning), the dims signature, and one column per attribute. Attrs and coords are independent namespaces, so an attr may share a name with a coordinate envelope column (e.g. an attr channel_step alongside a channel coordinate); in that rare genuine collision the envelope column owns the flat name, the attr column is omitted with a warning, and the attr remains queryable through the _attrs namespace. The chunk planner and selection both operate on this relation (see the Spool Chunking and Spool Selection notes). The cell below runs against the real catalog so this description cannot silently drift.

import dascore as dc
from dascore.io.index.catalog import PatchCatalog

catalog = PatchCatalog.from_patches(list(dc.get_example_spool("random_das")))
df = catalog.to_df()
required = {
    "_patch_id",
    "_time_def_key",
    "_time_units",
    "dims",
    "time_min",
    "time_max",
    "time_step",
}
assert required.issubset(df.columns)

Patch identity and spool set semantics

A spool of in-memory patches has set semantics by patch identity: constructing a spool from a sequence keeps one entry per distinct patch instance, and spool + spool unions membership. Each patch carries an instance identity minted eagerly at construction; because patches are immutable, copies (including deep copies and unpickled patches) share that identity, while every patch operation produces a new instance with its own. The identity is the patch’s synthetic memorypatch:// source path in the index, so identity, deduplication, and resolution all agree:

import copy

patch = dc.get_example_patch()

# a sequence with the same instance twice is one spool entry
assert len(dc.spool([patch, patch])) == 1

# copies share identity regardless of when they are made — identity is
# eager, so there is no access-order dependence
clone = copy.deepcopy(patch)
assert clone._instance_id == patch._instance_id
assert len(dc.spool([patch, clone])) == 1

# any operation mints a new, distinct identity
assert len(dc.spool([patch, patch.new()])) == 2

# the identity is the row's synthetic path, and it resolves back to the
# very same object
cat = PatchCatalog.from_patches([patch, patch])  # one entry, not two
row = cat.to_df().iloc[0]
assert len(cat.to_df()) == 1
assert row["source_path"].startswith("memorypatch://")
assert cat.resolve_row(row.to_dict()) is patch

File-backed patches are identified by (base_uri, source_path, source_patch_id) instead; their rows resolve through readers rather than the registry.

Federation

spool + spool merges catalogs table-to-table: source records are reconstructed from the member backends and re-ingested, so coordinate definitions deduplicate by definition key, the same source appearing in several members keeps a single entry, and file paths are absolutized so members with different roots coexist. A composite resolver routes in-memory rows to the shared registry, plan-output rows to their plan resolvers, and everything else through file readers.

Row-membership state (attribute predicates, slice windows, patch-id arrays) transfers as rows, preserving the set semantics above. State that only lives Python-side — coordinate/samples residual trims, sort specs, and a directory default order that record-grain transfer would actually scramble — is first baked into an identity-plan derived catalog (table work only; no patch data loads), so each operand contributes exactly its current contents in its current order. Baking mints new patch identities, so a trimmed operand no longer deduplicates against its source: its contents genuinely differ. Spool equality follows the same philosophy — it compares effective contents (residual trims folded into the envelopes, representation artifacts like def keys and backing excluded), so a trimmed view equals its union-materialized twin.

other = dc.get_example_patch().new()
union = dc.spool([patch]) + dc.spool([other])
assert len(union) == 2
assert len(dc.spool([patch]) + dc.spool([patch])) == 1  # same patch dedups

# A trimmed operand contributes its trimmed contents (baked, not dropped),
# and equality compares those effective contents.
t = patch.get_coord("time")
trimmed = dc.spool([patch]).select(time=(t.min() + 10 * t.step, t.min() + 20 * t.step))
combined = trimmed + dc.spool([])
assert combined[0].shape == trimmed[0].shape
assert combined == trimmed

Scope

The index answers metadata selection, identifies candidate patches, and plans chunk operations — all without touching patch data. Exact coordinate selection is applied again when a patch is loaded because summary envelopes cannot prove arbitrary-coordinate membership. Materialized derived data (persisting chunked results) remains future work.