Working with Remote Patches

Open in JupyterLite

DASCore accepts remote resources through UPath and tests backends including memory://, s3://, and http://.

Operation	Default behavior
scan, get_format	Avoid downloading a complete file
scan_payloads	Same cache policy, but may transfer full coordinate arrays
read, spool indexing/iteration	Perform the heavier I/O needed to load data

Basic example

memory:// demonstrates the remote interface without network access:

from upath import UPath

import dascore as dc

remote_path = UPath("memory://dascore/tutorial/remote_patch.h5")
dc.write(dc.get_example_patch(), remote_path, "DASDAE")

summary = dc.scan(remote_path)[0]
payload = dc.scan_payloads(remote_path, snap=False)[0]
loaded = dc.read(remote_path)[0]
spool = dc.spool(remote_path)

Backends that support direct access may require no local cached copy.

The summary contains compact coordinate envelopes and source fields. The payload includes the formatter’s coordinate manager, and indexing the spool loads the patch array.

Cache policy

Remote behavior is controlled through dascore.config:

Setting	Purpose
remote_cache_dir	Directory for materialized remote files
allow_remote_cache	Permit caching for data reads
allow_remote_cache_for_metadata	Permit caching for metadata operations
warn_on_remote_cache	Warn on the first download into the cache

from pathlib import Path

from dascore.config import config_context

with config_context(
    remote_cache_dir=Path("/tmp/dascore-remote-cache"),
    allow_remote_cache=True,
    allow_remote_cache_for_metadata=False,
):
    patch = dc.read(remote_path)[0]

When a metadata query would require a full download and metadata caching is disabled, DASCore raises RemoteCacheError. Opt in only for a known operation:

http_path = UPath("https://raw.githubusercontent.com/dasdae/test_data/master/das/example_dasdae_event_1.h5")

with config_context(allow_remote_cache_for_metadata=True):
    summary = dc.scan(http_path)[0]

scan_payloads avoids the data array but may still transfer and retain exact coordinate arrays. Prefer scan or scan_to_df for broad directory listings.

Warnings occur on the first materialization when warn_on_remote_cache=True, making an unexpected local download visible without repeating the message for every file.

Reads and spools

dc.read(...) may materialize a remote file when allow_remote_cache=True. For a spool, get_contents() uses metadata while indexing and iteration load patch data. If a backend cannot provide spool metadata directly, enable metadata caching explicitly.

remote_spool = dc.spool(http_path)
metadata = remote_spool.get_contents()
with config_context(allow_remote_cache=True):
    patch = remote_spool[0]

Remote HDF5 and garbage collection

Remote HDF5 file-like reads can deadlock if cyclic garbage collection runs on fsspec’s event-loop thread while h5py holds its global lock, as described in h5py’s file-like object documentation. DASCore therefore pauses cyclic garbage collection while remote HDF5 handles are open, restores it after the last handle closes, and warns once per process. Reference counting and local-file reads are unaffected.

Only cyclic garbage accumulates during the pause, and only until the final remote HDF5 handle closes. Objects reclaimed through normal reference counting are still released immediately.

with config_context(warn_on_gc_pause=False):
    patch = dc.read(http_path)[0]

Transfer tuning

Two settings control the remote HDF5 block cache:

Setting	Default	Trade-off
remote_hdf5_block_size	5 MiB	Larger blocks reduce requests but may over-fetch
remote_hdf5_max_blocks	8	More blocks retain more data per open handle

Their product approximates memory retained per handle: about 40 MiB by default. For metadata scans, reduce both values; for sequential patch reads, use larger blocks and fewer of them. Concurrent handles multiply this memory. When reading complete files, local materialization is usually more efficient than repeated range requests.

Small blocks reduce over-fetch during metadata scans but increase network round trips. Large blocks favor contiguous reads. Lower remote_hdf5_max_blocks first when many handles are open concurrently.

with config_context(remote_hdf5_block_size=262_144, remote_hdf5_max_blocks=4):
    summaries = dc.scan_to_df([http_path])

HTTP and HTTPS use both settings. S3 uses its own read-ahead cache and takes the block size only.