Coordinates and Coordinate Managers

Warning

This page covers advanced DASCore features. Most users will be fine with only the coordinate material presented in the patch tutorial.

In order to manage coordinate labels and array manipulations, DASCore implements two classes, BaseCoord, which has several associated subclasses corresponding to different types of coordinates, and CoordManager which manages a group of coordinates. Much like the Patch, instances of both of these classes are immutable (to the extent possible), so they cannot be modified in place but have methods which return new instances.

Coordinates

Coordinates usually keep track of labels along an associated dimension of an array, but they can also be independent of array data. They provide methods for slicing, re-ordering, filtering etc. and are used internally by DASCore for such operations.

Coordinates are a collection of classes which implement a common interface.

Note

Coordinates are very similar (in concept) to Pandas’ indices, with some significant differences in implementation.

Coordinate Creation

Get Coord

get_coord returns an instance of a subclass of BaseCoord appropriate for the input values. Here are a few examples:

import numpy as np

from dascore.core import get_coord

# Get monotonic, evenly sampled coords from start, stop, step.
range_coord = get_coord(start=0, stop=10, step=1)

# Do the same as above but using an evenly sampled sorted array.
array = np.arange(0, 10, step=1)
array_coord = get_coord(data=array)

# Assert the type and values of the resulting coordinates are the same.
assert range_coord == array_coord

# the Arrays don't have to be evenly sampled, or sorted.
sorted_array = np.sort(np.random.rand(10))
sorted_coord = get_coord(data=sorted_array)

random_array = np.random.rand(10) 
random_coord = get_coord(data=random_array)

String arrays create string-backed coordinates.

import numpy as np

from dascore.core import get_coord

coord = get_coord(data=np.array(["sensor_001", "sensor_002", "sensor_003"]))
print(type(coord).__name__)
print(coord.values)

CoordString
['sensor_001' 'sensor_002' 'sensor_003']

Segmented Coordinates

CoordSegmented preserves monotonic coordinate blocks and the discontinuities between them. It is useful when a logical coordinate contains dropped samples, acquisition gaps, or a sampling-rate change but the corresponding data should remain in one in-memory patch.

Use concat_coords(...) when the blocks are already known:

from dascore.core.coords import concat_coords, get_coord

first = get_coord(start=0.0, stop=5.0, step=1.0, units="m")
second = get_coord(start=8.0, stop=13.0, step=1.0, units="m")
segmented = concat_coords(first, second)

print(type(segmented).__name__)
print(segmented.get_discontinuities("gaps"))

CoordSegmented
   index  before  after  delta  excess
0      5     4.0    8.0    4.0     3.0

Use CoordSegmented.from_array(...) for an exact, strictly monotonic stored array. The constructor preserves every value and returns the simplest truthful coordinate, so an exactly uniform input becomes a normal CoordRange and a wholly irregular input may become a CoordMonotonicArray.

import numpy as np

from dascore.core.coords import CoordSegmented

values = np.array([0.0, 1.0, 2.0, 7.0, 8.0, 9.0])
exact_coord = CoordSegmented.from_array(values, tolerance=0, units="m")

assert np.array_equal(exact_coord.values, values)
assert len(exact_coord.get_discontinuities("gaps")) == 1

Segmented coordinates have step=None because one step cannot describe the entire coordinate. coord.simplify(tolerance=...) may replace segments with a simpler fit while bounding how far values move; coord.snap() always forces one evenly sampled range and can move interior values without a bound. Use patch.split_gaps() to turn a patch with segmented dimensional coordinates into contiguous patches before writing it to a format that cannot store gaps.

Update

Update uses the existing coordinate as a template and returns a coordinate with some part modified.

import numpy as np

import dascore as dc
from dascore.core import get_coord

# Create example coordinate.
start, stop = np.datetime64("2023-01-01"), np.datetime64("2023-01-01T01")
step = np.timedelta64(60, 's')
coord = get_coord(start=start, stop=stop, step=step)

# Update entire array, adding 10s to each element.
new_data = coord.values + np.timedelta64(10, "s")
coord_new_data = coord.update(data=new_data)

# Update step, keeping length and start but shortening end time.
coord_new_step = coord.update(step=np.timedelta64(1800, 's'))

# Change maximum value, keeping length the same and changing step.
coord_new_max = coord.update(max=stop + 10 * step)

Coordinate Attributes

The following tables shows some of the commonly used coordinate attributes:

Coordinate attributes
Attribute	Description
`sorted`	`True` if the coordinate is sorted in ascending order.
`reverse_sorted`	`True` if the coordinate is sorted in descending order.
`evenly_sampled`	`True` if the coordinate has uniform step sizes.
`dtype`	The numpy data type of the coordinate.
`data`	Return an array of coordinate values.
`units`	Coordinate units.
`degenerate`	`True` if the coordinate has a zero length dimension.
`min()`	Return the minimum value in the coordinate.
`max()`	Return the maximum value in the coordinate.

Coordinate Methods

This section highlights some of the coordinate methods. The methods which would cause changes to a data array return a new coordinate and an object that can be used for indexing an array. This can either be a slice instance or another array which uses numpy’s advanced indexing features for sorting or selection.

Sort

sort sorts the values of the coordinate.

import numpy as np

from dascore.core import get_coord

random_array = np.random.rand(10) 
random_coord = get_coord(data=random_array)

# Returns a new array and indexer that can be used to apply the sorting
# operation to a dimension of an array.
sorted_coord, indexer = random_coord.sort()

# The array could then be updated as follows.
data = np.random.rand(10, 20)
sorted_data = data[indexer, :]

Snap

snap replaces coordinate values with an evenly sampled range spanning the same minimum and maximum. This intentionally loses interior precision and should be used only when that idealization is acceptable. Unlike sort, it returns only a new coordinate and no indexer, because it replaces the values rather than permuting them, so for an unsorted coordinate the snapped values no longer label the samples they used to. When data alignment matters, use CoordManager.snap, which sorts the coordinate and its associated array together before snapping, or Patch.snap_coords(...), which does the same across the selected dimensions of a patch.

import numpy as np

from dascore.core import get_coord

values = np.array([0.0, 1.0, 2.2, 3.0])
irregular_coord = get_coord(data=values)
snapped_coord = irregular_coord.snap()

assert snapped_coord.evenly_sampled
assert snapped_coord.min() == irregular_coord.min()
assert snapped_coord.max() == irregular_coord.max()

Use simplify(tolerance=...) instead when the maximum allowed coordinate movement must be explicit. Most coordinate types are already in their simplest form; this distinction is most useful for segmented coordinates.

Select

select is used for slicing/sub-selecting.

import numpy as np

from dascore.core import get_coord

coord = get_coord(start=0, stop=21, step=1)

new_coord, indexer = coord.select((3, 14))

data = np.random.rand(10, 20)
selected_data = data[:, indexer]

Most coordinate methods also support units.

import numpy as np

from dascore.core import get_coord
from dascore.units import ft

coord = get_coord(start=0, stop=21, step=1, units='m')

new_coord, indexer = coord.select((14*ft, 50 * ft))
print(new_coord)

CoordRange( min: 5 max: 15 step: 1 shape: (11,) dtype: int64 units: m )

String Coordinates

String coordinates support exact matching, wildcard matching with * and ?, compiled regular expressions, boolean-mask selection, and sorting. They are useful for channel names, sensor identifiers, and other categorical labels that do not have meaningful numeric spacing.

import numpy as np
import re

from dascore.core import get_coord

coord = get_coord(data=np.array(["ch_a", "ch_b", "aux_1", "aux_2"]))

selected, indexer = coord.select("ch_b")
print(selected.values)
print(indexer)

matched, matched_indexer = coord.select("ch_*")
print(matched.values)
print(matched_indexer)

regex_matched, regex_indexer = coord.select(re.compile(r"^aux_\d$"))
print(regex_matched.values)
print(regex_indexer)

sorted_coord, sort_indexer = coord.sort()
print(sorted_coord.values)
print(sort_indexer)

['ch_b']
[False  True False False]
['ch_a' 'ch_b']
[ True  True False False]
['aux_1' 'aux_2']
[False False  True  True]
['aux_1' 'aux_2' 'ch_a' 'ch_b']
[2 3 0 1]

Unlike numeric, datetime, or timedelta coordinates, string coordinates do not support:

units
range-style selection such as (start, stop)
relative selection
range calculations such as coord_range()

Selection for string coordinates should use exact values, wildcard patterns, compiled regular expressions, or boolean masks. Strings containing * or ? are treated as wildcard patterns, so avoid those characters in labels you intend to select literally.

Units

convert_units and set_units are used to change/set the units associated with a coordinate.

import numpy as np

from dascore.core import get_coord
from dascore.units import ft

coord = get_coord(start=0, stop=21, step=1, units='m')

# Convert coords to ft.
coord_ft_converted = coord.convert_units("ft")

# Simply change unit label (values remain the same).
coord_ft_set = coord.set_units("ft")

# Create coord with silly units.
silly_units = "10*PI*m/ft * furlongs * fortnight"
coord_silly_units = get_coord(start=10, stop=21, step=1, units=silly_units)

# Simplify the coordinates and modify coordinate values accordingly.
simple_coord = coord_silly_units.simplify_units()
print(f"Simplified units are: {simple_coord.units}")
print(f"New coord lims are: {simple_coord.limits}")

Simplified units are: 1 m * s
New coord lims are: (np.float64(250805152879.93182), np.float64(501610305759.8637))

Get Next Index

get_next_index returns the index value (an integer) for where a value would be inserted into the coordinate. It can only be used on a sorted coordinate.

from dascore.core import get_coord
coord = get_coord(start=0, stop=10, step=1)
# Find the index for a value contained by the coordinate.
assert coord.get_next_index(1) == 1
# The next (not closest) index is return for value not in coord.
assert coord.get_next_index(2.000001) == 3

CoordManager

The CoordManager handles a group of coordinates and provides methods for updating managed data arrays.

Coordinate Manager Creation

CoordManager instances can be created from a dictionary of coordinates via the get_coord_manager function.

from dascore.core import get_coord, get_coord_manager

coord_dict = {
    "dim1": get_coord(start=1, stop=10, step=1),
    "dim2": get_coord(start=0.001, stop=1, step=.1),
}

cm = get_coord_manager(coords=coord_dict, dims=("dim1", "dim2"))

# dims are the dimension names (in order).
print(f"dimensions are {cm.dims}")

# coord_map is a mapping of {coord_name: coordinate}.
print(dict(cm.coord_map))

# dim_map is a mapping of {coord_name: (associated_dimensions...)}.
print(dict(cm.dim_map))

dimensions are ('dim1', 'dim2')
{'dim1': CoordRange( min: 1 max: 9 step: 1 shape: (9,) dtype: int64 ), 'dim2': CoordRange( min: 0.001 max: 0.901 step: 0.100 shape: (10,) dtype: float64 )}
{'dim1': ('dim1',), 'dim2': ('dim2',)}

CoordManagers can have non-dimensional coordinates which may or may not be associated with a coordinate dimension.

from dascore.core import get_coord, get_coord_manager

coord_dict = {
    "dim1": get_coord(start=0, stop=10, step=1),
    "dim2": get_coord(start=0.001, stop=1, step=.1),
    # "dim_coord" is a non-dimensional coordinate associated with
    # dim1, so it must have the same shape. Notice how the associated
    # dimension and coordinate values can be specified in a tuple. 
    "dim_coord": ("dim1", get_coord(start=10, stop=20, step=1)),
    # Non-dimensional coordinates are not associated with a dimension
    # and must use None as the first argument in the tuple.
    "non_dim_coord": (None, get_coord(start=1, stop=100, step=1)),
}

cm_many_coords = get_coord_manager(coords=coord_dict, dims=("dim1", "dim2"))
print(cm_many_coords)

➤ Coordinates (dim1: 10, dim2: 10)
    *dim1: CoordRange( min: 0 max: 9 step: 1 shape: (10,) dtype: int64 )
    *dim2: CoordRange( min: 0.001 max: 0.901 step: 0.100 shape: (10,) dtype: float64 )
    dim_coord ('dim1',): CoordRange( min: 10 max: 19 step: 1 shape: (10,) dtype: int64 )
    non_dim_coord (): CoordRange( min: 1 max: 99 step: 1 shape: (99,) dtype: int64 )

Update

update uses an existing CoordinateManager as a template and updates some aspect in the returned coordinate.

import dascore as dc

# Get coordinate manager from default patch.
patch = dc.get_example_patch()
cm = patch.coords

# Add 10 to each distance values create new coord.
dist_coord = cm.get_coord("distance") 
new_dist_array = dist_coord.data + 10
new_dist_coord = dist_coord.update(data=new_dist_array)

# Create new coordinate manager with new distance coord.
new_cm = cm.update(distance=new_dist_coord)

It can also be used to add new coordinates,

distance_length = len(cm.get_coord("distance"))

# Create a new coordinate. 
new_coord = get_coord(data=np.random.rand(distance_length))

# Add it to the coord manager associated with distance dimension.
new_cm_1 = cm.update(new_coord=("distance", new_coord))

# Add the coordinate but don't associate it with any dimension.
new_cm_2 = cm.update(new_coord=(None, new_coord))

and drop or disassociate coordinates.

# Disassociate "new_coord" from dimension distance.
new_cm_3 = new_cm_1.update(new_coord=(None, new_coord))

# Drop coordinate "new_coord". Note this must be done on a coord manager
# which actually has "new_coord"; dropping a missing coord is a no-op.
new_cm_4 = new_cm_1.update(new_coord=None)
assert "new_coord" not in new_cm_4.coord_map

# Drop dimension "time".
new_cm_5 = cm.update(time=None)
assert "time" not in new_cm_5.dims

Coordinate Manager Methods

Much like BaseCoord, the CoordinateManager class implements a variety of methods for filtering, sorting, modifying units, etc. However, there are some difference. Unlike coordinates, when an operation would change the data array associated with the coordinates, the CoordManager method accepts the array as an argument and returns a new array. Like the Patch methods, CoordManager methods use keyword arguments to specify coordinates by name.

Select

select trims the coordinate manager and, optionally, an associated array.

import dascore as dc

patch = dc.get_example_patch()
cm, data = patch.coords, patch.data

new_cm, new_data = cm.select(data=data, distance=(..., 100))

Sort

sort sorts along one or more axes.

import dascore as dc

patch = dc.get_example_patch()
cm, data = patch.coords, patch.data

# Sort along both dimensions in descending order.
new_cm, new_data = cm.sort("time", "distance", reverse=True)

Rename Coord

rename_coord renames a coordinate or dimension.

import dascore as dc

patch = dc.get_example_patch()
cm = patch.coords

# Rename time to money.
renamed_cm = cm.rename_coord(time="money")
print(renamed_cm)

➤ Coordinates (distance: 300, money: 2000)
    *distance: CoordRange( min: 0 max: 299 step: 1 shape: (300,) dtype: int64 units: m )
    *money: CoordRange( min: 2017-09-18 max: 2017-09-18T00:00:07.996 step: 0.004s shape: (2000,) dtype: datetime64[ns] units: s )