Working with Files

Open in JupyterLite

DASCore reads and writes local pathlib.Path objects and remote UPath resources supported by each format. See Working with remote patches for cache policy.

Writing and reading

Write through Patch.io, then open the result as a spool:

from pathlib import Path

import dascore as dc

patch = dc.get_example_patch()
path = Path("output_file.h5")
patch.io.write(path, "DASDAE")
round_trip = dc.spool(path)[0]

Remote destinations use the same interface through universal-pathlib, whose UPath objects support URLs implemented by fsspec:

from upath import UPath

remote_path = UPath("memory://dascore/tutorial/output.pkl")
patch.io.write(remote_path, "pickle")
remote_patch = dc.spool(remote_path)[0]

Format support varies by backend: a formatter that requires a local seekable file may use the remote cache, while formats built on fsspec-compatible stores can operate directly.

Scanning metadata

dc.scan returns PatchSummary objects without loading patch arrays.

summary_index = 0
summary = dc.scan("examples://terra15_das_1_trimmed.hdf5")[summary_index]

print(summary.source_path)
print(summary.get_coord_summary("time").min)

/home/runner/work/dascore/dascore/.test_data_cache/0.0.0/terra15_das_1_trimmed.hdf5
2021-10-11T22:43:37.380047104

Scanning also reports source_format, source_version, and source_patch_key, which together identify the formatter and patch inside a multi-patch file.

Use the source path and the summary’s position to load the same patch through its spool. Scan results and file-spool rows have the same order, and the spool uses each row’s source_patch_key when materializing it. For compact directory listings, use scan or scan_to_df.

source_spool = dc.spool(
    summary.source_path,
    file_format=summary.source_format,
    file_version=summary.source_version,
)
loaded = source_spool[summary_index]

dc.scan_payloads(...) also returns each formatter’s CoordManager. With snap=False, formats that store per-sample coordinates expose their exact values:

payload = dc.scan_payloads(summary.source_path, snap=False)[0]
time = payload["coords"].get_coord("time")

Payload scans still may read and retain large coordinate arrays, so reserve them for targeted files.

Patch.attrs contains non-coordinate metadata. Access coordinate bounds and steps from Patch.summary, PatchSummary.get_coord_summary(...), or the flattened summary returned by flat_dump().

Directory spools

dc.spool(directory) creates a lazy, indexed view of every supported file below that directory:

from dascore import examples

directory = examples.spool_to_directory(dc.get_example_spool("diverse_das"))
spool = (
    dc.spool(directory)
    .select(acquisition_key="DAS2.*")
    .select(time=(..., "2022-01-01"))
    .chunk(time=2, overlap=0.5)
)

Iteration is where patch arrays are loaded:

for patch in spool:
    processed = patch.detrend("time")

The hidden .dascore_index.sqlite3 stores file metadata and enables selections before patch data is loaded. Opening a new directory creates the index; call spool.update() after files change. See the spool tutorial and index note for path attributes and index lifecycle.

update() scans new or modified files, removes rows for deleted files, and leaves unchanged rows alone. File size and modification time identify candidates for rescanning.

An optional .inventory companion describes the observing system and is not scanned as data.

The index and inventory are companions to the archive rather than data sources themselves, so both use reserved hidden names and are excluded from formatter discovery.

External formats

Patch.io also converts patches to structures used by Pandas, Xarray, and ObsPy. See the external conversion recipe.