Skip to content

Operations Dashboard

Operations Dashboard is the live operational status tab for the whole Aurora stack.

What it shows

  • source-host probe health
  • local and remote storage pressure
  • Aurora Power Supply battery voltage from DCInverterVolts, scored green above 52 V, amber from 50-52 V, and red below 50 V
  • Aurora Power Supply battery state of charge from BatterySOC, scored green at or above 50 %, amber above 40 % and below 50 %, and red at or below the 40 % operational minimum
  • Aurora Power Supply estimated time until depleted, calculated from BatterySOC, a 26 kWh battery capacity, and BatteryWatts; negative BatteryWatts is treated as discharge, while positive power is reported as charging
  • Aurora Power Supply internal temperature from InternalTemperature, scored green from 10-40 C, amber from 5-10 C or 40-45 C, and red below 5 C or at 45 C or above
  • Aurora Power Supply internal dew-point margin, calculated only when a true APS internal InternalHumidity signal is present; ambient/site humidity is not used as a substitute, and missing APS humidity is shown as unavailable
  • source sync and processing health
  • HATPRO source, local mirror, GWS mirror, Zarr-build, and quicklook health alongside the other science streams
  • dashboard performance-log freshness, including whether browser activity is still being written to /data/aurora/products/dashboard/dashboard_perf.jsonl
  • dashboard HTTP endpoint health and response time
  • public primary and standby dashboard endpoint health, including a full document check that catches a blank Bokeh shell
  • dashboard and infrastructure git cleanliness and local ahead/behind counts
  • recent dashboard render-performance statistics, including p50, p95, slowest timed event, and live-session counts
  • root-cause grouping for source computers, network/source sync, local processing, GWS transfer, and dashboard/render behavior
  • seven-day trend cards for worst storage pressure, battery SOC, battery voltage, worst source lag, and worst GWS lag
  • GWS transfer status
  • mirror verification and prune-readiness indicators
  • per-stream archive state, including WXcam backfill progress

Performance-log freshness and render-performance statistics are diagnostic signals only. They stay visible on the Operations Dashboard and in the health reports, but they do not drive the top-level Overall action state.

Most source streams are marked stale after 1.5 h without new source files. HATPRO is deliberately marked stale only after 3 h because it lands in hourly batches; this requires two missed batches before the Operations Dashboard calls the source stale.

The storage cards are intentionally broken out as:

  • CL61 root and CL61 data
  • ASS data and ASS root
  • APS data and APS root
  • AURORA Cloud product and AURORA Cloud root
  • JASMIN GWS

Each card subtitle uses the resolved pwd -P path that was actually probed for filesystem usage.

The trend cards read the operations Zarr directly and are cached briefly in the dashboard process. They are intended as quick context, not as a replacement for the archived HK_Operations plots.

Display model

This tab reads the latest operations snapshot directly rather than waiting for an archived quicklook to exist. That means a fresh deployment can show the live Operations tab before the archived operations PNGs have accumulated enough samples to plot. Archive traffic lights are based on settled mirror health, so a stream stays green when the settled GWS and object-store evidence has no missing or mismatched files even if the newest just-arrived source file has not yet landed in the next transfer batch. A fresh derived file that is not yet in object storage stays in a separate pending-upload state for 30 hours; pending is not a red archive failure. A settled missing or mismatched file is red and remains so until exact repair and a new complete inventory succeed.

The inventory progress heartbeat lets the dashboard distinguish a slow shard-by-shard scan from a stall. A running inventory is healthy while that heartbeat is no more than five minutes old. Stable object parity requires two distinct complete clean reports; prune_ready for one stream is only one input to the independently enforced retention gate.

The Public endpoints section is snapshot-backed. It probes:

  • https://data.gamb2le.co.uk/app
  • https://data-ocean.gamb2le.co.uk/app

Each probe requires a successful HTTP response and the full AURORA Data Viewer document. A small Bokeh shell with title Bokeh Application is treated as unhealthy because it usually means the Python app handler crashed before rendering the dashboard. During the July 2026 JASMIN shutdown window, data.gamb2le.co.uk identifies the JASMIN endpoint and data-ocean.gamb2le.co.uk identifies the droplet endpoint, unless a full DNS failover is explicitly initiated.

Archived products

The archived operations products live under:

  • /data/aurora/products/quicklooks/ops_monitor
  • /data/aurora/products/ops_monitor/health

These include:

  • summary quicklooks
  • HK_Operations
  • observe-only health JSON and daily Markdown reports

Email Alerts

Operations alert email is handled by send_ops_alerts.py, normally from aurora-ops-monitor-alerts.timer. It evaluates the same latest snapshot used by the dashboard and emails gamb2le@ncas.ac.uk for storage pressure at 80 %, battery SOC at or below the 40 % operational minimum, APS internal temperature below 5 C or at or above 45 C, APS internal dew-point margin at or below 0 C when internal humidity is available, battery voltage below 50 V, and stream-health problems that persist for 3 h.

The service keeps state under /data/aurora/products/ops_monitor/alerts so it can send initial, repeat, and recovery messages without spamming every timer tick. The deployed delivery path is intended to be mailx backed by msmtp or another sendmail-compatible outbound relay.

Detailed product documentation: