Working with Files

The following highlights some DASCore features for working with IO.

Note

If you are working with remote UPath resources, especially HTTP or object store backends, see the Working with Remote Patches tutorial for examples, cache settings, and metadata-vs-read behavior.

Using UPath Resources

DASCore accepts both pathlib.Path and UPath inputs for many file-backed workflows. This is useful when your data lives on a non-local backend supported by fsspec.

The example below uses memory:// so it can run without any external network access.

from upath import UPath
import dascore as dc

source = dc.get_example_patch()
remote_path = UPath("memory://dascore/tutorial/example_patch.h5")

dc.write(source, remote_path, "DASDAE")

summary = dc.scan(remote_path)[0]
loaded = dc.read(remote_path)[0]

print(summary.source_path)
print(loaded.dims)

memory://dascore/tutorial/example_patch.h5
('distance', 'time')

Writing Patches to Disk

Patches can be written to disk using the io namespace. The following shows how to write a Patch to disk in the DASDAE format

from pathlib import Path
import dascore as dc

write_path = Path("output_file.h5")
patch = dc.get_example_patch()

patch.io.write(write_path, "dasdae")

PosixPath('output_file.h5')

Remote-style UPath destinations also work for supported formats.

from upath import UPath
import dascore as dc

patch = dc.get_example_patch()
remote_write_path = UPath("memory://dascore/tutorial/output_file.pkl")

patch.io.write(remote_write_path, "pickle")

round_trip = dc.read(remote_write_path)[0]
print(round_trip.dims)

('distance', 'time')

Scan Metadata Without Loading Data

dascore.scan returns PatchSummary objects. These results expose patch metadata, coordinate-summary envelopes, and source information without loading the data array into memory.

import dascore as dc
from dascore.utils.downloader import fetch

path = fetch("terra15_das_1_trimmed.hdf5")
scan_patch = dc.scan(path)[0]

print(scan_patch.source_path)
print(scan_patch.get_coord_summary("time").min)

/home/runner/work/dascore/dascore/.test_data_cache/0.0.0/terra15_das_1_trimmed.hdf5
2021-10-11T22:43:37.380047104

Scan results are useful for listing contents, building dataframes, and deciding which patches to load.

loaded_patch = dc.read(
    scan_patch.source_path,
    file_format=scan_patch.source_format,
    file_version=scan_patch.source_version,
    source_patch_id=scan_patch.source_patch_id or None,
)[0]
print(loaded_patch.data.shape)

(3620, 550)

When a coordinate-summary envelope is insufficient, dascore.scan_payloads(...) exposes each formatter’s full CoordManager without loading the patch data array. Pass snap=False to request the exact stored coordinate values from formats that store per-sample arrays.

payload = dc.scan_payloads(path, snap=False)[0]
time_coord = payload["coords"].get_coord("time")

print(payload["source_path"])
print(type(time_coord).__name__)
print(time_coord.get_discontinuities("gaps"))

/home/runner/work/dascore/dascore/.test_data_cache/0.0.0/terra15_das_1_trimmed.hdf5
CoordMonotonicArray
Empty DataFrame
Columns: [index, before, after, delta, excess]
Index: []

scan_payloads may read and retain large coordinate arrays, so prefer a specific file over an entire directory and discard each result promptly. Use dc.scan or dc.scan_to_df for compact directory indexing.

Note

Patch.attrs stores non-coordinate metadata only. Coordinate summaries such as time_min, time_max, and distance_step are accessed through PatchSummary.get_coord_summary(...) or via patch.summary.get_coord_summary(...).

Directory spools

A spool over a directory of dascore-readable files is created with the dascore.spool function. It is the same class as any other spool; the only difference is how it was constructed.

For example:

import dascore
from dascore import examples as ex

# Get a directory with several files
diverse_spool = dascore.get_example_spool('diverse_das')
path = ex.spool_to_directory(diverse_spool)

# Create a spool for interacting with the files in the directory.
spool = (
  dascore.spool(path)
  .select(acquisition_key='DAS2.*')  # sub-select one data source
  .select(time=(..., '2022-01-01'))  # unselect anything after 2022
  .chunk(time=2, overlap=0.5)  # change the chunking of the patches
)

# Iterate each patch and do something with it
for patch in spool:
  ...

Converting Patches to Other Library Formats

The Patch.io namespace also includes functionality for converting Patch instances to datastructures used by other libraries including Pandas, Xarray, and ObsPy. See the external conversion recipe for examples.

Directory Indexer

The DBDirectoryIndexer tracks the contents of a directory which contains fiber data. It creates a small, hidden SQLite index named .dascore_index.sqlite3 at the top of the directory. Directory spools use this index internally and push metadata selections into SQLite before loading patch data. See the spool index note for the schema and lifecycle.

.dascore_index.sqlite3 is one of two hidden names DASCore gives a meaning at the top of a data directory. The other is .inventory, which is where a directory may keep the inventory describing the observing system its data was recorded through. Both are companions the directory keeps rather than content it holds, which is why both are hidden and neither is scanned as data.

import dascore
from dascore.io.index.indexer import DBDirectoryIndexer
from dascore import examples as ex

# Get a directory with several files
diverse_spool = dascore.get_example_spool('diverse_das')
path = ex.spool_to_directory(diverse_spool)

# Create an indexer and update the index. This scans new or changed files
# (detected by per-file modification time and size), removes entries of
# deleted files, and creates the index if one does not yet exist.
DBDirectoryIndexer(path).update()

# The index is queried through a spool. Opening one indexes the directory
# if it has not been indexed yet, but does not rescan afterwards: call
# spool.update() (or the indexer's, above) when the files change.
df = dascore.spool(path).get_contents()

# This dataframe can be used to ascertain data availability, detect gaps, etc.

	source_path	source_format	source_version	_path_attrs	_patch_id	source_patch_id	dims	_dtype	time_min	time_max	...	distance_max	distance_step	_distance_def_key	distance_units	_time_def_key	time_units	tag	category	acquisition_key
0	DAS_____wayout__1989_05_04__1989_05_04T00_00_0...	DASDAE	1	None	10	DAS_____wayout__1989_05_04__1989_05_04T00_00_07	distance,time	float64	1989-05-04	1989-05-04 00:00:07.996	...	299.0	1.0	fp:413c28e8532bf78951ccb94d15e663f7\|m	m	fp:c73c562d0a7dda8a397829089c328a15\|s	s	wayout	DAS
1	DAS___DAS2.R2D1..RAW__random__2020_01_03__2020...	DASDAE	1	None	12	DAS___DAS2.R2D1..RAW__random__2020_01_03__2020...	distance,time	float64	2020-01-03	2020-01-03 00:00:07.996	...	299.0	1.0	fp:413c28e8532bf78951ccb94d15e663f7\|m	m	fp:41c3dce8c62437b01c4f98ffcc349703\|s	s	random	DAS	DAS2.R2D1..RAW
2	DAS_____big_gaps__2020_01_03__2020_01_03T00_00...	DASDAE	1	None	14	DAS_____big_gaps__2020_01_03__2020_01_03T00_00_07	distance,time	float64	2020-01-03	2020-01-03 00:00:07.996	...	299.0	1.0	fp:413c28e8532bf78951ccb94d15e663f7\|m	m	fp:41c3dce8c62437b01c4f98ffcc349703\|s	s	big_gaps	DAS
3	DAS_____overlaps__2020_01_03__2020_01_03T00_00...	DASDAE	1	None	11	DAS_____overlaps__2020_01_03__2020_01_03T00_00_07	distance,time	float64	2020-01-03	2020-01-03 00:00:07.996	...	299.0	1.0	fp:413c28e8532bf78951ccb94d15e663f7\|m	m	fp:41c3dce8c62437b01c4f98ffcc349703\|s	s	overlaps	DAS
4	DAS_____random__2020_01_03__2020_01_03T00_00_0...	DASDAE	1	None	13	DAS_____random__2020_01_03__2020_01_03T00_00_07	distance,time	float64	2020-01-03	2020-01-03 00:00:07.996	...	299.0	1.0	fp:413c28e8532bf78951ccb94d15e663f7\|m	m	fp:41c3dce8c62437b01c4f98ffcc349703\|s	s	random	DAS

5 rows × 21 columns