Skip to content

Core Files

This repo is a mix of application code, appenders, quicklook generators, and operations tooling. The most important files are grouped below by role.

Dashboard application

  • app.py - main Panel application and UI wiring
  • grouped_timeseries.py - shared summary-layout, labeling, static quicklook generation, and Plotly trace-reduction helpers for the curated 1D instruments
  • quicklook_time_axis.py - shared UTC time-axis formatting for static science and housekeeping quicklooks

Numeric data appenders

  • append_new_cloud_radar_to_zarr.py
  • append_new_vaisalamet_to_zarr.py
  • append_new_asfs_logger_to_zarr.py
  • append_new_asfs_fast_sonic_to_zarr.py
  • append_new_power_to_zarr.py
  • append_new_netcdf_to_zarr.py

These scripts ingest raw mirrored files and append them into the deployed Zarr stores. The appenders sort and deduplicate time coordinates, filter to samples that are genuinely newer than the existing store, materialize that filtered block, and then append. That policy avoids partial chunk writes that can show up as false all-NaN stripes in range-resolved products.

Quicklooks and latest plots

  • generate_cloud_radar_quicklooks.py
  • generate_vaisalamet_quicklooks.py
  • generate_asfs_logger_quicklooks.py
  • generate_asfs_fast_sonic_quicklooks.py
  • generate_power_quicklooks.py
  • generate_power_display_summary.py - builds the compact derived Power display-summary Zarr used by the APS interactive and quicklook summary panels
  • generate_power_display_energy.py - builds the compact derived Power display-energy Zarr used as a cumulative-panel compatibility product
  • generate_ops_monitor_quicklooks.py
  • plot_*_last24h.py

These scripts generate the archived PNG products and the latest-view assets used by the dashboard. The Power, Meteorology, and Radiation quicklook generators also write prewarmed latest interactive Plotly JSON under /data/aurora/products/dashboard/prewarm/ so those first interactive views can paint without rebuilding the whole figure.

On the deployed host, run these generators through their systemd services or source /etc/aurora-dashboard.env before manual runs. Without that environment they use the repo-local quicklooks/ fallback, which is useful for development but is not the live dashboard product tree.

WXcam tooling

  • wxcam_catalog.py - shared catalog helpers
  • index_wxcam_catalog.py - builds or refreshes the SQLite catalog
  • build_wxcam_daily_videos.py - builds daily MP4s, latest.mp4, and hourly thumbnails
  • append_new_wxcam_to_zarr.py - appends HDR JPG image data to the WXcam Zarr

Operations monitoring

  • collect_operations_snapshot.py - collects source-host, storage, mirror, systemd, dashboard endpoint, and git health into raw JSONL snapshots plus observe-only health JSON and Markdown reports
  • send_ops_alerts.py - evaluates the latest operations snapshot and sends threshold email alerts with stateful repeat and recovery handling
  • append_new_ops_monitor_to_zarr.py - appends or rebuilds the monitoring Zarr
  • extra_housekeeping.py - extra housekeeping quicklook helpers, including the Ceilometer, Cloud Radar, and WXcam HK products

Support and maintenance

  • summarize_dashboard_perf.py - summarizes JSONL performance timing logs
  • consolidate_zarr_metadata.py - metadata consolidation helper