Xarray

Open in marimo

DASCore supports users who prefer to work with xarray. Spools become lazy DataTrees, patches convert to DataArrays and back, patch methods run directly on DataArrays, and patches index, compute, and write the way xarray users expect. Xarray is optional; install it with DASCore’s extras.

Spools as lazy DataTrees

Spool.io.to_xarray turns a whole spool into a dask-backed DataTree, with one node per group of related patches and one child node per merged segment. Building the tree reads no data; computing a selection reads only the patches it touches.

import numpy as np

import dascore as dc

spool = dc.get_example_spool("diverse_das")
tree = spool.io.to_xarray()

first = next(node for node in tree.subtree if "data" in node.dataset)
small = first.dataset["data"].isel(time=slice(0, 100)).compute()
small.shape

(300, 100)

Patches and DataArrays

Patch.io.to_xarray converts a patch to a DataArray, and dc.io.xarray_to_patch converts it back without loss:

patch = dc.get_example_patch()
dar = patch.io.to_xarray()
assert dc.io.xarray_to_patch(dar) == patch
dar

<xarray.DataArray (distance: 300, time: 2000)> Size: 5MB
array([[0.77770241, 0.23754122, 0.82427853, ..., 0.36950848, 0.07650396,
        0.23197621],
       [0.49689594, 0.44224037, 0.70329426, ..., 0.12617754, 0.11760625,
        0.78003741],
       [0.20681917, 0.19516906, 0.17434521, ..., 0.84933595, 0.36479426,
        0.80740811],
       ...,
       [0.61877586, 0.1053084 , 0.66896335, ..., 0.621027  , 0.43559346,
        0.49975826],
       [0.75717115, 0.25935121, 0.09051709, ..., 0.36099578, 0.9365496 ,
        0.10351814],
       [0.15780837, 0.29487104, 0.58475197, ..., 0.22898748, 0.23950251,
        0.49439913]], shape=(300, 2000))
Coordinates:
  * distance  (distance) int64 2kB 0 1 2 3 4 5 6 ... 293 294 295 296 297 298 299
  * time      (time) datetime64[ns] 16kB 2017-09-18 ... 2017-09-18T00:00:07.9...
Attributes:
    data_type:        
    data_category:    
    acquisition_key:  
    tag:              random
    history:          ()
    origin_id:        d66ac5c4ca1c4bb8a06c111213217627
    data_id:          d66ac5c4ca1c4bb8a06c111213217627
    category:         DAS

xarray.DataArray

• distance: 300
• time: 2000

• 0.7777 0.2375 0.8243 0.9657 0.9726 ... 0.303 0.229 0.2395 0.4944

  array([[0.77770241, 0.23754122, 0.82427853, ..., 0.36950848, 0.07650396,
          0.23197621],
         [0.49689594, 0.44224037, 0.70329426, ..., 0.12617754, 0.11760625,
          0.78003741],
         [0.20681917, 0.19516906, 0.17434521, ..., 0.84933595, 0.36479426,
          0.80740811],
         ...,
         [0.61877586, 0.1053084 , 0.66896335, ..., 0.621027  , 0.43559346,
          0.49975826],
         [0.75717115, 0.25935121, 0.09051709, ..., 0.36099578, 0.9365496 ,
          0.10351814],
         [0.15780837, 0.29487104, 0.58475197, ..., 0.22898748, 0.23950251,
          0.49439913]], shape=(300, 2000))

• Coordinates: (2)
  • distance
    (distance)
    int64
    0 1 2 3 4 5 ... 295 296 297 298 299

    units :

    m

    array([  0,   1,   2, ..., 297, 298, 299], shape=(300,))

  • time
    (time)
    datetime64[ns]
    2017-09-18 ... 2017-09-18T00:00:...

    array(['2017-09-18T00:00:00.000000000', '2017-09-18T00:00:00.004000000',
           '2017-09-18T00:00:00.008000000', ..., '2017-09-18T00:00:07.988000000',
           '2017-09-18T00:00:07.992000000', '2017-09-18T00:00:07.996000000'],
          shape=(2000,), dtype='datetime64[ns]')

• Attributes: (8)

  data_type :

  data_category :

  acquisition_key :

  tag :

  random

  history :

  ()

  origin_id :

  d66ac5c4ca1c4bb8a06c111213217627

  data_id :

  d66ac5c4ca1c4bb8a06c111213217627

  category :

  DAS

lazy_coords=True keeps DASCore’s coordinates as the index, computing labels only for the samples selected, so converting back is exact even for fractional sampling rates and gaps. Use it to keep the sampling step of a dimension with one sample, which materialized labels cannot record:

lazy = patch.io.to_xarray(lazy_coords=True)
type(lazy.xindexes["time"]).__name__

'CoordIndex'

The dc accessor

DataArrays from DASCore have a dc accessor with every Patch method; methods that return a patch return a DataArray. For other DataArrays, import dascore.xarray.accessor first.

filtered = dar.dc.pass_filter(time=(1, 10))

Xarray-style indexing

Patch.sel indexes by label and Patch.isel by position, as in xarray:

label_window = patch.sel(distance=slice(10, 20))
sample_window = patch.isel(time=slice(0, 100, 2))
nearest_channel = patch.sel(distance=10.2, method="nearest", tolerance=0.5)
channel = patch.isel(distance=3)  # drops the distance dimension

Their docstrings give the exact rules and what is unsupported. Patches have no .loc or .iloc. For DASCore’s range tuples and relative selections, use Patch.select.

Familiar methods

Many patch methods share xarray’s names: transpose, squeeze, where, fillna, dropna, pipe, rolling, idxmax, and the reductions (mean, max, std, …). Reductions keep the reduced dimension with length one unless given dim_reduce="squeeze". NumPy functions also return patches.

positive = patch.where(patch > 0, 0)
channel_mean = patch.mean("time", dim_reduce="squeeze")
differenced = np.diff(patch, axis=1)

Writing with encodings

The NetCDF, Zarr, and DASDAE writers accept xarray’s encoding, and their files open in xarray:

import tempfile
from pathlib import Path

import xarray as xr

path = Path(tempfile.mkdtemp()) / "patch.nc"
encoding = {"data": {"zlib": True, "complevel": 4, "chunksizes": (50, 1000)}}
patch.io.write(path, "NETCDF_CF", encoding=encoding)
xr.open_dataset(path)["data"].encoding["chunksizes"]

(50, 1000)

See File formats in depth for Zarr and other options, and the external conversion recipe for index alignment and dask details.