import dascore as dc
spool = dc.get_example_spool()Spool
Keywords
spool, chunking, overlapping chunks, gaps, archive
A Spool manages patches from memory, files, directories, or remote resources. File-backed spools scan metadata first and load arrays only when patches are accessed.
Data sources
Use dc.spool(...) for any supported input:
memory_spool = dc.spool([dc.get_example_patch()])
file_spool = dc.spool("examples://terra15_das_1_trimmed.hdf5")Remote paths use the same constructor:
from upath import UPath
remote_file = UPath("memory://dascore/tutorial/spool-file.h5")
dc.write(dc.get_example_patch(), remote_file, "DASDAE")
remote_spool = dc.spool(remote_file)Directory spools maintain an index for fast metadata queries. Call update() after files change.
# Write example patches to a directory to get a usable path.
directory = dc.examples.spool_to_directory(dc.get_example_spool())
directory_spool = dc.spool(directory).update()To place the index outside the data directory:
import tempfile
from pathlib import Path
index_path = Path(tempfile.mkdtemp()) / "index.h5"
external_index = dc.spool(directory, index_path=index_path).update()Pass index_path= when the data directory is not writable. New spools normally rediscover an external index; an index inside the data directory takes precedence. Remote UPath resources also work; see Working with remote patches for cache policy.
Path attributes
Directory names written as key=value become string attributes on index rows and loaded patches. Multiple pairs in one segment use __; path values override attributes stored in the file.
root = Path(tempfile.mkdtemp())
data_path = root / "experiment=demo" / "cable=north"
data_path.mkdir(parents=True)
dc.get_example_patch().io.write(data_path / "patch.h5", "DASDAE")
partitioned = dc.spool(root).update()
assert partitioned.get_contents()["experiment"].iloc[0] == "demo"
assert partitioned[0].attrs["cable"] == "north"Renaming a partition directory and calling update() refreshes path attributes without rereading file contents. Removing a path attribute may require rescanning the affected file so its stored value can be restored.
Only directories below the spool root are parsed; the root and source filename are not. Values remain strings and support the normal attribute selectors. When a path key matches a coordinate, bare select targets the attribute; use _coords for the coordinate. See the directory index note for update and indexing details.
Accessing patches
Indexing and iteration load patches. Slicing, integer arrays, and boolean masks return a new spool.
patch = spool[0]
subset = spool[1:]
for patch in spool:
...Integer arrays reorder patches and may repeat them; boolean arrays keep positions marked True:
import numpy as np
reordered = spool[np.array([2, 0])]
mask = np.ones(len(spool), dtype=bool)
mask[1] = False
without_second = spool[mask]Lists and Pandas Series follow the same positional rules. Slicing remains lazy: it narrows the spool plan without loading selected arrays.
Rows from get_contents() align with spool positions, so a dataframe condition can select the same patches. Do not reuse a mask from a differently ordered spool.
diverse = dc.get_example_spool("diverse_das")
contents = diverse.get_contents()
tagged = diverse[contents["tag"] == "some_tag"]Inspecting and selecting
Displaying a spool shows its patch count, coordinate extents, and tracks for distinct patch kinds. Large spools omit the expensive summary according to display_max_patches.
A track groups patches with the same configured kind attributes and measures their span along time, or another available dimension. Single-kind spools omit redundant tracks because their overall extent already carries the same information. Tracks describe extents, not coverage; use get_coverage for missing-data statistics.
dc.get_example_spool("diverse_das")Dimensions (time, distance)
time: 1989-05-04T00:00:00 to 2030-01-01T00:00:07.996 <40.7 y> distance: 0.000 to 299.000 m <299.000 m>
Tracks (7 along time)
big_gaps 3 patches 2020-01-03 <26 s> overlaps 3 patches 2020-01-03 <24 s> random 3 patches 2020-01-03 <24 s> smallg 3 patches 2020-01-03 <24 s> some_tag 1 patch 2020-01-03 <8 s> wayout 2 patches 1989-05-04 <40.7 y> DAS2.R2D1..RAW Β· random 3 patches 2020-01-03 <24 s> (coverage: spool.viz.coverage())
Spool.get_contents returns one metadata row per patch without loading its data array. File-backed rows include source_path, source_format, and source_patch_key.
contents = diverse.get_contents()
contents[["time_min", "time_max", "source_format"]].head()| time_min | time_max | source_format | |
|---|---|---|---|
| 0 | 2020-01-03 00:00:00 | 2020-01-03 00:00:07.996 | memory |
| 1 | 2020-01-03 00:00:08 | 2020-01-03 00:00:15.996 | memory |
| 2 | 2020-01-03 00:00:16 | 2020-01-03 00:00:23.996 | memory |
| 3 | 2020-01-03 00:00:00 | 2020-01-03 00:00:07.996 | memory |
| 4 | 2020-01-03 00:00:08 | 2020-01-03 00:00:15.996 | memory |
The dataframe is a realized relation, so requesting every row can be expensive for a very large remote archive. Apply select first whenever the index backend can push down the predicate.
Spool.select filters metadata and trims matching patches along coordinates.
recent = diverse.select(time=("2020-01-03", ...))
sources = diverse.select(acquisition_key={"DAS2.R2D1..RAW", "DAS3.R2D1..RAW"})
tagged = diverse.select(tag="some*")Coordinate selections both discard patches outside the range and trim intersecting patches. Attribute selections only filter rows. Collections, globs, and compiled regular expressions are accepted for string metadata. A glob means what SQLiteβs GLOB means, since that is what the index applies: * and ? are wildcards, [abc] is a character class, and [^abc] negates one.
import re
regex_tags = diverse.select(tag=re.compile(r"^some"))Finding a patch by ID
File-backed origin_id values are indexed, so a recorded ID can locate its source without loading every patch:
wanted = spool.get_contents()["origin_id"].iloc[0]
found = spool.select(origin_id=wanted)
assert found[0].attrs.origin_id == wanteddata_id is also indexed. Attribute queries match the stored source ID. A value trim clears the ID in presented contents because the loaded patch names the window it read and so carries a different value; sample selections defer it until loading. Built or chunked spools may have no origin_id, and moving a source invalidates its derived IDs until the directory is rescanned. See Patch identity for the identity rules.
An empty ID after a file move is intentional: retaining the path-derived value would point at a source which no longer exists. Calling update() rebuilds the row from its new location.
Unselect
Spool.unselect returns patches rejected by the same attribute selection.
not_tagged = diverse.unselect(tag="some_tag")Patch coordinates are excluded because complementing an interior range would split every patch. Inventory coordinates are allowed and may increase the number of resulting patches.
from dascore.examples import inventory_patch_pair
inventory_patch, inventory = inventory_patch_pair()
inventory_spool = dc.spool(inventory_patch).attach_inventory(inventory)
outside_zone = inventory_spool.unselect(zone="zone_1")Inventory complements may cut one input patch into several retained pieces, so an unselect result can contain more patches than its input.
Chunking and coverage
Spool.chunk lazily changes patch boundaries. It can merge contiguous patches, create fixed windows with overlap, or limit array size.
from dascore.units import megabytes, s
windowed = spool.chunk(time=3 * s, overlap=1 * s, keep_partial=True)
merged = spool.chunk(time=None)
bounded = spool.chunk(time=1 * megabytes)overlap accepts the same coordinate-unit, quantity, or data-size forms as the chunk length. keep_partial=True retains a shorter final window. Passing time=None asks DASCore to merge every contiguous compatible segment.
To request particular absolute windows, pass an (n, 2) array. select returns each matching source piece separately; chunk assembles compatible pieces inside each requested row. Rows stay in input order, including overlaps and duplicates. Chunking requires every requested sampled window by default, and chunk_plan shows the accepted outputs without loading measurement arrays. on_incomplete="warn" or "ignore" skips unmet outputs, while keep_partial=True accepts available nonempty pieces before that policy applies.
import numpy as np
small = dc.spool(dc.get_example_patch().select(distance=(0, 9)))
windows = np.array([[2, 4], [6, 8]])
source_pieces = small.select(distance=windows)
requested_chunks = small.chunk(distance=windows)
requested_plan = small.chunk_plan(distance=windows)
assert len(source_pieces) == len(requested_chunks) == len(requested_plan.outputs) == 2Eligible whole-file merges read arrays directly and reconstruct coordinates and scalar attributes from the index. Members requiring trims, associated coordinates, or metadata the index cannot reproduce use the patch reader. Index-built results retain patch and processing identities and omit stored history. To retain stored history, load the members first, as in dc.spool(list(file_spool)).chunk(time=None).
Data-size limits cover the array only, not coordinates, metadata, or later processing copies. MB is decimal and MiB is binary.
Filling bridged holes
A tolerance wide enough to span a hole merges the patches on either side, but the samples in between are still missing, so the merged patch is unevenly sampled and its coordinate stays segmented. fill_value writes those samples instead, and the output comes back as an ordinary evenly sampled patch.
import numpy as np
from dascore.examples import random_spool
with_gaps = random_spool(time_gap=np.timedelta64(1, "s"), length=3)
filled = with_gaps.chunk(time=None, tolerance=300, fill_value=np.nan)tolerance alone decides which holes are bridged: a hole it does not span separates patches as before, and nothing is filled across it. When a chunk length is given as well, the windows cover the whole bridged span, so one lying entirely inside a hole comes back as an all-fill patch and one the sources only partly feed is padded rather than left short. The fill value has to survive a cast to the dataβs own dtype, so np.nan needs float data; fill integer data with an integer, or cast it first. Patch.fill_gaps does the same to a patch you already hold.
Gaps and coverage
Spool.get_gaps reports boundaries that chunk will not merge. Spool.get_coverage summarizes the same compatible groups.
import numpy as np
from dascore.examples import random_spool
gappy = random_spool(time_gap=np.timedelta64(1, "s"), length=3)
gaps = gappy.get_gaps()
coverage = gappy.get_coverage()second_scale_gaps = gappy.get_gaps(tolerance=1 * s)gap_size spans from the last sample before a gap to the first after it, so subtract one sample step for the missing extent. The tolerance argument, shared with chunk, controls which boundaries count. Pass another dimension as in get_gaps("distance"); time is only the default. Coverage uses patch envelopes, and for a patch whose coordinate is segmented into runs of one step (such as a gapped patch merged in memory or read whole from a DASDAE file; one with more than 256 runs is read whole) the envelope of each run, so holes inside such a patch are visible too. group_id links coverage rows to their gaps.
At the default tolerance of 1.5 samples, one missing sample counts as a gap. A quantity or timedelta bounds the excess over one step in coordinate units instead: a spacing is a gap when it exceeds step + tolerance. Coordinates judge their own gaps by the same rule (see coordinates). Overlaps are not gaps; coverage calculates the union of available spans.
Coverage groups use patch kind, dimensions, sampling rate, and coordinate structure. One metadata tag can therefore produce several rows when its patches cannot be combined safely.
Use Spool.viz.coverage or Spool.viz.calendar to plot these summaries.
Concatenate
Spool.concatenate joins each compatible group even across gaps or overlaps. Unlike chunk, it does not require continuity.
patch = dc.get_example_patch()
one_hour = dc.to_timedelta64(3600)
later = patch.update_coords(time_min=patch.get_coord("time").max() + one_hour)
joined = dc.spool([patch, later]).concatenate(time=None)concatenate can also create a new dimension when given a name not already present. Use this to stack compatible patches whose samples should remain distinct rather than merging along an existing axis.
Patches that cannot concatenate remain in separate output groups. Values are ordered along the concatenated dimension when it has a step; otherwise spool order is retained. Other coordinates must agree, while coordinates along the concatenated dimension are joined. For conflicting non-coordinate attributes, conflict= can raise (the default), drop them, or keep the first value.
Loading ahead
Use Spool.iterate to load upcoming patches in a background thread while processing the current patch:
from contextlib import closing
with closing(spool.iterate(max_in_flight=2)) as patches:
for patch in patches:
processed = patch.detrend("time")A positive window requires thread support; use ordinary iteration or max_in_flight=0 in WebAssembly. The limit counts upcoming patches; the current patch and any retained results are additional. Closing the iterator releases its loading thread even if the loop exits early. Ordinary iteration stays synchronous. The parallel processing recipe also shows how to scan directory updates with a reusable process executor.
Mapping
Spool.map eagerly applies a function to every patch. Chunk first to control work size.
def distance_max(patch):
"""Return each channel's maximum."""
return patch.aggregate("time", "max")
mapped = spool.chunk(time=5, overlap=1).map(distance_max)
result = dc.spool(mapped).concatenate(time=None)[0]map returns a list by default. Wrap it with dc.spool(...) to continue using spool operations, as above. Active runtime configuration is propagated to thread and process workers.
See the parallel processing recipe for executors and larger workflows.
Inventory
A spool can attach a DASDAE inventory. Inventory labels become available to select and expand_by; enrich copies observing-system metadata onto extracted patches.
patch, inventory = inventory_patch_pair()
with_inventory = dc.spool(patch).attach_inventory(inventory)
assert len(with_inventory.expand_by("zone")) == 2Directory spools automatically load .inventory, .inventory.yaml, .inventory.yml, or .inventory.json stored beside the data.