import h5py
import numpy as np
import xarray as xr
import dascore as dc
with h5py.File("original_foreshock.h5", "r") as source:
dataset = source["data"]
values = dataset[:, :1000]
start = dc.to_datetime64(dataset.attrs["begin_time"])
interval = dc.to_timedelta64(float(dataset.attrs["dt_s"]))
spacing = float(dataset.attrs["dx_m"])
array = xr.DataArray(
values,
dims=("distance", "time"),
coords={
"time": start + np.arange(values.shape[1]) * interval,
"distance": ("distance", np.arange(values.shape[0]) * spacing, {"units": "m"}),
},
attrs={"data_category": "DAS"},
)
patch = dc.io.xarray_to_patch(array)Patch Conversions
DASCore converts patches to common library objects and back.
Pandas
import dascore as dc
patch = dc.get_example_patch()
df = patch.io.to_dataframe()
patch_from_df = dc.io.dataframe_to_patch(df)Xarray
The xarray tutorial covers converting patches and spools, lazy_coords, and the dc accessor. This section adds index alignment, dask, and laziness details.
Combining lazy and ordinary indexes
A lazy index (lazy_coords=True) answers selections as a materialized index does, including partial datetime strings, method="nearest", and tolerance.
A lazy array combines with an array whose index holds its labels (built from numpy, loaded from netCDF, or converted with the default) only when the labels are equal and xarray is recent enough to compare them, which reads every lazy label. Otherwise, as when labels differ or when reindexing to new labels, xarray raises an AlignmentError. To combine them, give the lazy array ordinary indexes: drop_indexes removes the lazy index but keeps the coordinate’s labels, and set_xindex builds an ordinary index from them, reading every label into memory (8 bytes a sample; a day at 1 kHz is about 0.7 GB). The data stays as it is. Select first where you can, so only the selection’s labels are read:
import dascore as dc
lazy = dc.get_example_patch().io.to_xarray(lazy_coords=True)
eager = lazy.drop_indexes(["time", "distance"]).set_xindex("time").set_xindex("distance")Where the labels differ, or xarray cannot compare them, the reverse is often cheaper: give the ordinary array the lazy index type. Its labels are already in memory, and where the two are the same coordinate (one a selection of the other, say) they align without reading the lazy labels; equal labels stated another way, or labels that differ, are joined as ordinary indexes would be, which reads both.
from dascore.xarray.index import CoordIndex
aligned = eager.drop_indexes("time").set_xindex("time", CoordIndex)Spool DataTrees
A spool’s tree is partitioned exactly as chunk partitions it. The coordinate of the dimension that segments are merged along is lazy, so a segment spanning years costs nothing to label; other dimensions keep ordinary indexes, so per-channel arrays such as gains combine with a segment directly. Along the merged dimension, use the drop_indexes/set_xindex swap above. A local file that changed since the spool was indexed is refused when the tree is built, or when a later read finds it the wrong shape; run spool.update() and convert again.
A selection reads only the samples it names. block_size bounds a bulk read instead: computing a whole segment requests one block at a time, so a source patch larger than a block is read in several windows. Unset, it takes the configured xarray_block_size. A format that cannot return a window is read whole regardless, since splitting would read the file once per block.
The accessor with dask and lazy coordinates
An accessor method returns a non-patch result, such as an array or a spool, unchanged, and a method taking another patch takes another DataArray.
A dask-backed DataArray is passed through as it is, so a method that works on the array’s own backend leaves it lazy, and one that cannot converts it, as it would a patch. When such a method announces that conversion and the array is larger than free memory, the accessor raises instead; a method that converts silently is not caught and fails on memory as it would anywhere else.
A lazy coordinate stays lazy in what a method returns, so labelling a long acquisition costs no more after a call than before. Each DataArray in the call, arguments included, contributes how its coordinates arrived: a coordinate that arrived materialized anywhere is materialized in the result, since xarray aligns on materialized indexes. Laziness follows the name, so renaming a lazy coordinate materializes its labels under the new name.
ObsPy
import dascore as dc
patch = dc.get_example_patch()
stream = patch.io.to_obspy()
patch_from_stream = dc.io.obspy_to_patch(stream)
Note
obspy_to_patch requires every trace to have the same data length and the requested dimension in stats (distance by default).
Research HDF5 arrays
An HDF5 array with timing and spacing attributes may need an explicit mapping when it lacks the units required by a reader. For example, the Taiwan MiDAS release includes original_foreshock.h5, with a channel-by-time data array and begin_time, dt_s, and dx_m attributes, but no data units. After downloading that file, this example reads its first 1,000 time samples and converts them through xarray:
The distance origin here is relative to the first released channel. Data units remain unknown; the conversion does not assign strain or strain-rate units without source metadata. Choose the mapping from the release documentation rather than applying this layout to arbitrary HDF5 files.
A reduced AP Sensing release, such as the Cape Muroto tremor files, keeps the DAQ, Metadata and DAS groups but drops the version attributes, interrogator, time zone and data rate the AP Sensing reader needs, so it is not read automatically. The file alone does not settle its sampling: DAQ/PositionEnd over the 1,079 channels gives about 51 m, ten times the stated SpatialSampling of 5.1 m, and nothing says whether DAQ/RepetitionFrequency (100) is the output rate. Read such a file with h5py as above once the release’s documentation confirms the time step and channel spacing.