Windows and Tiles

Patch.window declares windows over a patch, and what is done in them decides what comes back:

you want use what comes back
a value per sample, from the samples around it a reduction, such as patch.window(time=0.1).mean() a patch shaped like the input
a transform of each window, kept as windows Patch.stft, or apply with mode="stack" a patch with a window axis
a patch operation run in each window, and the windows blended back the operation, such as patch.window(time=1).detrend("time") a patch shaped like the input, or shorter where the operation keeps every n-th sample
a function of your own run in each window, and the windows blended back apply a patch shaped like the input

Window lengths use coordinate units or samples (samples=True). overlap accepts a count, duration, percentage, or per-dimension mapping, and step says how far each window advances instead. A window with neither overlaps the next by half when it is blended, and advances one sample when it is reduced.

A value for each window

A reduction gives one value for each window, labelled by one of the input’s samples: the window’s middle sample by default, or its first or last with align. With a step of one sample the result has the input’s shape. A window which reaches past either end of the data is incomplete: ends="pad", the default, gives it NaN, and ends="drop" leaves it out.

import numpy as np

import dascore as dc

patch = dc.get_example_patch()

smooth = patch.window(time=0.01).mean()
assert smooth.shape == patch.shape

# Only the complete windows, a tenth of a second long and half that apart.
envelope = patch.window(time=0.1, step=0.05, ends="drop").max()
assert not np.isnan(envelope.data).any()

# Any function which reduces the last axis.
upper = patch.window(time=0.1, step=0.05).reduce(np.nanpercentile, 90)

An operation in each window

A patch operation called on a window runs in each window, and the windows are blended back under complementary tapers. The result is a different one from the operation run on the whole patch: this takes a trend out of each 20 ms, which follows a drift a single line through the whole record cannot.

import numpy as np

import dascore as dc

patch = dc.get_example_patch("example_event_2")

whole = patch.detrend("time")
local = patch.window(time=0.02).detrend("time")
assert local.shape == patch.shape
assert not np.allclose(local.data, whole.data)
print(local.attrs.history[-1])
window(time='0.02').detrend(dim='time',type='linear')

The window covers the one operation called on it, and what comes back is an ordinary patch, so a chain goes on from there: patch.window(time=0.02).detrend("time").pass_filter(time=(1, 300)) detrends in windows and then filters the blended result.

One plan is worked out for the operation, from the first window, and run in every window, so only an operation which acts relative to its window can be run this way. DASCore’s detrend, taper, filters, normalizations, derivatives and the like say that they do. line_mute and taper_range do when their values are measured from the start of a coordinate (relative=True, which is line_mute’s default and not taper_range’s) or given for a dimension which is not windowed; absolute values along a windowed dimension name a place only some windows hold, and are refused. The operation must also give back a window with the same coordinates, or every n-th sample of them as described below, and take no other patch, so select is refused, and the window must fit inside its dimension. All of this is decided before any data are touched.

An operation may keep every n-th sample of a windowed dimension, as decimate and fbe with a step do, when the windows advance by a multiple of n samples; the result is then n times shorter. An operation built on a rolling window, such as fbe or stalta, has no value for the first samples of each window. Those are discarded and the rest blended, so the windows must overlap by at least the rolling window:

# Band energy every 0.5 ms, each 20 ms filtered on its own. The rolling
# window is 2 ms, so the windows overlap by more than that.
options = {"time": (500, 2000), "window": 0.002, "step": 0.0005}
energy = patch.window(time=0.02, overlap=0.005).fbe(**options)
whole = patch.fbe(**options)
assert energy.coords == whole.coords
# Empty only where the plain energy is: the first samples of the patch.
assert np.array_equal(np.isnan(energy.data), np.isnan(whole.data))

For a patch operation, data on an array backend other than NumPy stay where they are. A dask array is computed a piece at a time, each from the windows which overlap it, so nothing is computed when the operation is called. On any other backend the windows are cut, run and blended there, which is several array operations for each window and so slower than NumPy; an operation with no implementation for the backend is converted through NumPy as it is outside a window.

A recursive filter in pieces

pass_filter and notch_filter reach across their whole axis, so on a lazy patch they need it whole. Their response to one sample decays, though, and Patch.impulse_window measures how far it reaches before what is left is under a tolerance. A lazy or dask patch is then filtered a chunk at a time, each read with that much around it, and the result is within about the tolerance of the exact one. Data in memory are filtered whole, exactly.

patch = dc.get_example_patch()
exact = patch.pass_filter(time=(1, 20))
cut = patch.impulse_window(time=1e-6).pass_filter(time=(1, 20))
assert np.array_equal(cut.data, exact.data)
print(cut.attrs.history[-1])
impulse_window(time='1e-06').pass_filter(corners='4',time='(1,20)',zerophase='True')

The tolerance is the fraction of the summed absolute response which may lie beyond the reach. A causal filter (zerophase=False) reads only the samples before each output sample, so it reaches one way.

Tiles, blended

apply passes an [n_tiles, *window] array to a function and blends its same-shaped result with complementary tapers. An identity function reconstructs the input exactly. This example applies windowed automatic gain control:

import numpy as np
import matplotlib.pyplot as plt

import dascore as dc
from dascore.units import percent

patch = dc.get_example_patch("example_event_2").pass_filter(time=(1, 300))


def agc(tiles):
    """Scale every tile to unit RMS."""
    rms = np.sqrt(np.mean(tiles**2, axis=(1, 2), keepdims=True))
    return tiles / np.where(rms > 0, rms, 1)


normalized = patch.window(time=0.02, distance=40).apply(agc)

fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=True)
for ax, pa, title in zip(axes, [patch, normalized], ["band-passed", "AGC, 20 ms by 40 m"]):
    scale = np.percentile(np.abs(pa.data), 99)
    pa.viz.waterfall(ax=ax, scale=scale, scale_type="absolute", cmap="bwr")
    ax.set_title(title)
fig.tight_layout()

The adaptive spectral filter uses the same machinery with spectral weighting.

Tiles, kept

With mode="stack", tiles remain separate. Windowed dimensions become tile-center axes, {dim}_start and {dim}_stop record physical cell edges in the dimension’s units, and tile samples use {dim}_offset. Process the stack as a patch, then reassemble it. Paired physical bounds require monotonic row order; explicitly drop the public bounds before an arbitrary tile permutation. Private reconstruction indices follow the rows and still allow reassembly.

tiles = patch.window(time=0.02, distance=40).apply(lambda x: x, mode="stack")
print(tiles.dims, tiles.shape)

# Drop every tile whose energy is in the lowest quarter, then blend back.
# The mean keeps the offset axes at length one, so a per-tile flag
# broadcasts back over every sample of its tile.
energy = (tiles**2).mean(dim=("distance_offset", "time_offset"))
quiet = energy.data < np.percentile(energy.data, 25)
kept = tiles.new(data=np.where(quiet, 0, tiles.data))
denoised = kept.reassemble()
assert denoised.shape == patch.shape
('distance', 'time', 'distance_offset', 'time_offset') (32, 12, 40, 200)

An unchanged stack reassembles exactly. stft transforms the offset axes; istft inverts them and reassembles.

A compiled function, one tile at a time

A Numba-compiled function receives one tile at a time in parallel. This supports any number of dimensions and mode="stack", with a one-time compilation cost per process.

import numba


@numba.njit
def agc_tile(tile):
    rms = np.sqrt(np.mean(tile * tile))
    return tile / rms if rms > 0 else tile


normalized = patch.window(time=0.02, distance=40).apply(agc_tile)

A window on the way in

By default only blending is tapered. Set analysis to window each tile before the function; blending uses the window’s dual so identity remains exact and overlap may exceed half the window.

def keep_strongest(tiles):
    """Keep the largest quarter of each tile's f-k coefficients."""
    spectra = np.fft.rfft2(tiles)
    cutoff = np.percentile(np.abs(spectra), 75, axis=(1, 2), keepdims=True)
    return np.fft.irfft2(np.where(np.abs(spectra) < cutoff, 0, spectra), s=tiles.shape[1:])


window = patch.window(time=0.02, distance=40, analysis="hann", overlap=75 * percent)
local_fk = window.apply(keep_strongest)
assert local_fk.shape == patch.shape

What a tile sees

Tiles extend before the data with zero padding so every contributing tile exists. Overlap defaults to half the window and cannot exceed half with blend-only tapers; analysis windows may overlap as far as SciPy can invert them. get_window supplies supported taper shapes.