import dascore as dc
spool = dc.get_example_spool()Spool
A Spool manages patches from memory, files, directories, or remote resources. File-backed spools scan metadata first and load arrays only when patches are accessed.
Data sources
Use dc.spool(...) for any supported input:
memory_spool = dc.spool([dc.get_example_patch()])
file_spool = dc.spool("examples://terra15_das_1_trimmed.hdf5")Remote paths use the same constructor:
from upath import UPath
remote_file = UPath("memory://dascore/tutorial/spool-file.h5")
dc.write(dc.get_example_patch(), remote_file, "DASDAE")
remote_spool = dc.spool(remote_file)Directory spools maintain an index for fast metadata queries. Call update() after files change.
# Write example patches to a directory to get a usable path.
directory = dc.examples.spool_to_directory(dc.get_example_spool())
directory_spool = dc.spool(directory).update()To place the index outside the data directory:
import tempfile
from pathlib import Path
index_path = Path(tempfile.mkdtemp()) / "index.h5"
external_index = dc.spool(directory, index_path=index_path).update()Pass index_path= when the data directory is not writable. New spools normally rediscover an external index; an index inside the data directory takes precedence. Remote UPath resources also work; see Working with remote patches for cache policy.
Path attributes
Directory names written as key=value become string attributes on index rows and loaded patches. Multiple pairs in one segment use __; path values override attributes stored in the file.
root = Path(tempfile.mkdtemp())
data_path = root / "experiment=demo" / "cable=north"
data_path.mkdir(parents=True)
dc.get_example_patch().io.write(data_path / "patch.h5", "DASDAE")
partitioned = dc.spool(root).update()
assert partitioned.get_contents()["experiment"].iloc[0] == "demo"
assert partitioned[0].attrs["cable"] == "north"Renaming a partition directory and calling update() refreshes path attributes without rereading file contents. Removing a path attribute may require rescanning the affected file so its stored value can be restored.
Only directories below the spool root are parsed; the root and source filename are not. Values remain strings and support the normal attribute selectors. When a path key matches a coordinate, bare select targets the attribute; use _coords for the coordinate. See the directory index note for update and indexing details.
Accessing patches
Indexing and iteration load patches. Slicing, integer arrays, and boolean masks return a new spool.
patch = spool[0]
subset = spool[1:]
for patch in spool:
...Integer arrays reorder patches and may repeat them; boolean arrays keep positions marked True:
import numpy as np
reordered = spool[np.array([2, 0])]
mask = np.ones(len(spool), dtype=bool)
mask[1] = False
without_second = spool[mask]Lists and Pandas Series follow the same positional rules. Slicing remains lazy: it narrows the spool plan without loading selected arrays.
Rows from get_contents() align with spool positions, so a dataframe condition can select the same patches. Do not reuse a mask from a differently ordered spool.
diverse = dc.get_example_spool("diverse_das")
contents = diverse.get_contents()
tagged = diverse[contents["tag"] == "some_tag"]Inspecting and selecting
Displaying a spool shows its patch count, coordinate extents, and tracks for distinct patch kinds. Large spools omit the expensive summary according to display_max_patches.
A track groups patches with the same configured kind attributes and measures their span along time, or another available dimension. Single-kind spools omit redundant tracks because their overall extent already carries the same information. Tracks describe extents, not coverage; use get_coverage for missing-data statistics.
dc.get_example_spool("diverse_das")Dimensions (time, distance)
time: 1989-05-04T00:00:00 to 2030-01-01T00:00:07.996 <40.7 y> distance: 0.000 to 299.000 m <299.000 m>
Tracks (7 along time)
big_gaps 3 patches 2020-01-03 <26 s> overlaps 3 patches 2020-01-03 <24 s> random 3 patches 2020-01-03 <24 s> smallg 3 patches 2020-01-03 <24 s> some_tag 1 patch 2020-01-03 <8 s> wayout 2 patches 1989-05-04 <40.7 y> DAS2.R2D1..RAW ยท random 3 patches 2020-01-03 <24 s> (coverage: spool.viz.coverage())
Spool.get_contents returns one metadata row per patch without loading its data array. File-backed rows include source_path, source_format, and source_patch_key.
contents = diverse.get_contents()
contents[["time_min", "time_max", "source_format"]].head()| time_min | time_max | source_format | |
|---|---|---|---|
| 0 | 2020-01-03 00:00:00 | 2020-01-03 00:00:07.996 | memory |
| 1 | 2020-01-03 00:00:08 | 2020-01-03 00:00:15.996 | memory |
| 2 | 2020-01-03 00:00:16 | 2020-01-03 00:00:23.996 | memory |
| 3 | 2020-01-03 00:00:00 | 2020-01-03 00:00:07.996 | memory |
| 4 | 2020-01-03 00:00:08 | 2020-01-03 00:00:15.996 | memory |
The dataframe is a realized relation, so requesting every row can be expensive for a very large remote archive. Apply select first whenever the index backend can push down the predicate.
Spool.select filters metadata and trims matching patches along coordinates.
recent = diverse.select(time=("2020-01-03", ...))
sources = diverse.select(acquisition_key={"DAS2.R2D1..RAW", "DAS3.R2D1..RAW"})
tagged = diverse.select(tag="some*")Coordinate selections both discard patches outside the range and trim intersecting patches. Attribute selections only filter rows. Collections, Unix-style globs, and compiled regular expressions are accepted for string metadata.
import re
regex_tags = diverse.select(tag=re.compile(r"^some"))Finding a patch by ID
File-backed patch_id values are indexed, so a recorded ID can locate its source without loading every patch:
wanted = spool.get_contents()["patch_id"].iloc[0]
found = spool.select(patch_id=wanted)
assert found[0].attrs.patch_id == wantedprocessing_id is not indexed because loading advances processing provenance. Built or chunked spools may have no patch_id, and moving a source invalidates its derived ID until the directory is rescanned. See Patch identity for the identity rules.
An empty ID after a file move is intentional: retaining the path-derived value would point at a source which no longer exists. Calling update() rebuilds the row from its new location.
Unselect
Spool.unselect returns patches rejected by the same attribute selection.
not_tagged = diverse.unselect(tag="some_tag")Patch coordinates are excluded because complementing an interior range would split every patch. Inventory coordinates are allowed and may increase the number of resulting patches.
from dascore.examples import inventory_patch_pair
inventory_patch, inventory = inventory_patch_pair()
inventory_spool = dc.spool(inventory_patch).attach_inventory(inventory)
outside_zone = inventory_spool.unselect(zone="zone_1")Inventory complements may cut one input patch into several retained pieces, so an unselect result can contain more patches than its input.
Chunking and coverage
Spool.chunk lazily changes patch boundaries. It can merge contiguous patches, create fixed windows with overlap, or limit array size.
from dascore.units import megabytes, s
windowed = spool.chunk(time=3 * s, overlap=1 * s, keep_partial=True)
merged = spool.chunk(time=None)
bounded = spool.chunk(time=1 * megabytes)overlap accepts the same coordinate-unit, quantity, or data-size forms as the chunk length. keep_partial=True retains a shorter final window. Passing time=None asks DASCore to merge every contiguous compatible segment.
Data-size limits cover the array only, not coordinates, metadata, or later processing copies. MB is decimal and MiB is binary.
Gaps and coverage
Spool.get_gaps reports boundaries that chunk will not merge. Spool.get_coverage summarizes the same compatible groups.
import numpy as np
from dascore.examples import random_spool
gappy = random_spool(time_gap=np.timedelta64(1, "s"), length=3)
gaps = gappy.get_gaps()
coverage = gappy.get_coverage()second_scale_gaps = gappy.get_gaps(tolerance=1 * s)gap_size spans from the last sample before a gap to the first after it, so subtract one sample step for the missing extent. The tolerance argument, shared with chunk, controls which boundaries count. Pass another dimension as in get_gaps("distance"); time is only the default. Coverage uses patch envelopes; holes inside a patch are not visible. group_id links coverage rows to their gaps.
At the default tolerance of 1.5 samples, one missing sample counts as a gap. A quantity or timedelta expresses the limit in coordinate units instead. Overlaps are not gaps; coverage calculates the union of available spans.
Coverage groups use patch kind, dimensions, sampling rate, and coordinate structure. One metadata tag can therefore produce several rows when its patches cannot be combined safely.
Use Spool.viz.coverage or Spool.viz.calendar to plot these summaries.
Concatenate
Spool.concatenate joins each compatible group even across gaps or overlaps. Unlike chunk, it does not require continuity.
patch = dc.get_example_patch()
one_hour = dc.to_timedelta64(3600)
later = patch.update_coords(time_min=patch.get_coord("time").max() + one_hour)
joined = dc.spool([patch, later]).concatenate(time=None)concatenate can also create a new dimension when given a name not already present. Use this to stack compatible patches whose samples should remain distinct rather than merging along an existing axis.
Patches that cannot concatenate remain in separate output groups. Values are ordered along the concatenated dimension when it has a step; otherwise spool order is retained. Other coordinates must agree, while coordinates along the concatenated dimension are joined. For conflicting non-coordinate attributes, conflict= can raise (the default), drop them, or keep the first value.
Mapping
Spool.map eagerly applies a function to every patch. Chunk first to control work size.
def distance_max(patch):
"""Return each channel's maximum."""
return patch.aggregate("time", "max")
mapped = spool.chunk(time=5, overlap=1).map(distance_max)
result = dc.spool(mapped).concatenate(time=None)[0]map returns a list by default. Wrap it with dc.spool(...) to continue using spool operations, as above. Active runtime configuration is propagated to thread and process workers.
See the parallel processing recipe for executors and larger workflows.
Inventory
A spool can attach a DASDAE inventory. Inventory labels become available to select and expand_by; enrich copies observing-system metadata onto extracted patches.
patch, inventory = inventory_patch_pair()
with_inventory = dc.spool(patch).attach_inventory(inventory)
assert len(with_inventory.expand_by("zone")) == 2Directory spools automatically load .inventory, .inventory.yaml, .inventory.yml, or .inventory.json stored beside the data.