import dascore as dc
patch = dc.get_example_patch()
filtered = patch.pass_filter(time=(1, 10))
# Where the data came from survives processing; which array it is does not.
assert filtered.attrs.origin_id == patch.attrs.origin_id
assert filtered.attrs.data_id != patch.attrs.data_id
# The same route gives the same array id.
assert filtered.attrs.data_id == patch.pass_filter(time=(1, 10)).attrs.data_idKeys, IDs and Rows
DASCore names things three ways, and the suffix says which.
| Suffix | What it is | Meaningful | Examples |
|---|---|---|---|
_id |
An identity: 32 hex characters, hashed or random | Everywhere | origin_id, data_id, operation_id |
_key |
A native name inside a resource, given by a reader | Within that resource | source_patch_key, ArraySource.key |
_row |
A row number in a spool index | Within that database | patch_row, source_row, coord_row |
Two indexes over the same files agree on every _id and _key, and on no _row. A key may be hashed into an ID; a row never is.
Three names sit outside the table. Inventory resource_id values are UUID labels for shareable objects, and acquisition_key is the inventory identity which joins a patch to them. The frame columns output_id (chunk plans) and group_id (gap reports) predate the rule: they are ordinal numbers, not identities.
The three identities
Every ID comes from one function, H, which hashes the canonical JSON of a domain and a payload. Domains keep equal payloads of different kinds apart.
| ID | Says | Carried by |
|---|---|---|
origin_id |
Which whole stored thing this came from | patches, ArraySource |
data_id |
Which array this is | patches, ArraySource, coordinates |
operation_id |
Which call: name, version and parameters | patch functions, PatchProcessor |
Development versions briefly called the two patch IDs patch_id and processing_id. Attributes read under those names are taken as origin_id and data_id; only the current names are written.
Weak and strong data_id
A data_id is the best name available for an object, and there are two grades.
A strong ID is a hash of content. Coordinates always have one: a range hashes its start, stop, step and units, never its materialized values, and an array coordinate hashes the values it holds in memory. A strong ID needs nothing beside it.
first = dc.get_coord(start=0, stop=10, step=1, units="m")
second = dc.get_coord(start=0, stop=10, step=1, units="m")
assert first.data_id == second.data_idA weak ID is derived without reading data, so it is only as good as what it was derived from, and it always has an origin_id beside it:
- a stored patch takes its origin’s ID, which is derived from the format, version, path, key, size and modification time unless the file stores one;
- a window of a stored array is derived from the whole array’s ID and the absolute sample windows, so slicing twice and slicing once agree;
- an operation’s result is derived from the
data_idof each input, in order, and theoperation_id.
Equal IDs mean the same array under either grade. Unequal IDs promise nothing: the same bytes reached two ways can carry two IDs. A strong ID only makes equal IDs more common, which is what lets identical coordinate arrays in thousands of files share one entry.
An ID is never knowingly false. When an operation’s parameters cannot be described faithfully (a closure, a lambda, an object of an unknown type), or an input carries no ID, the call still succeeds and its result receives a random data_id with a DASCoreWarning.
Operations
operation_id is H("operation", {name, version, params}). Patches among the arguments are inputs rather than parameters, numbered in one canonical walk, so which argument a patch filled is part of the operation. An argument which restates its default is left out, so adding a defaulted parameter changes no ID; changing what a default does requires raising the operation’s version.
Rows
Row numbers exist so tables can refer to each other: patches.source_row points at sources.source_row, and patch_coords joins patch_row to coord_row. They are assigned at ingest, differ between databases, and never enter an ID. A spool frame carries the patch’s row as the private column _patch_row.
See Patch identity for the user-facing rules and Spool Index for the tables.