Skip to content

Zarr And Product Overview

The dashboard depends on a mix of numeric Zarr stores, camera media products, and operations-monitor products.

The individual product pages describe schemas and may include dated example sizes or time ranges. Treat those examples as historical documentation, not as live health checks. Use the Operations Dashboard for current stream freshness and the deployed runtime configuration for current paths.

Numeric instrument Zarrs

WXcam products

AURORACam products

Operations products

Design pattern

The general rule is:

  • raw mirrored inputs live under /project/aurora/raw
  • dashboard-facing products live under /data/aurora/products

For most numeric instruments, that means a fixed-schema Zarr plus generated PNG quicklooks. The fixed-summary 1D instruments can also write prewarmed latest interactive Plotly JSON under /data/aurora/products/dashboard/prewarm/. WXcam adds media/catalog products on top of that pattern. AURORACam adds a raw JPEG browser plus a small metadata Zarr. Operations Dashboard adds raw JSONL snapshots, a monitoring Zarr, archived PNG quicklooks, live trend cards, and observe-only health reports.

This storage split is not itself the backup policy. Production infrastructure copies raw data and selected products additively to both GWS and object storage. Derived products never substitute for exact raw-retention evidence, and some mutable/rebuildable products such as wxcam.zarr are deliberately excluded. See Archive health.

Numeric appenders use a common safe-append policy: incoming files are sorted, deduplicated, filtered to genuinely new timestamps, and materialized before the Zarr append. That keeps the deployed stores monotonic and avoids partial chunk writes that can otherwise appear as false data gaps or all-NaN stripes.