Skip to content

WXcam Products

WXcam uses a richer product model than the purely numeric instruments.

Local raw mirror

The deployed raw mirror retains only HDR JPG and MP4 files for the FISH and PANO streams:

  • /project/aurora/raw/wxcam/FISH
  • /project/aurora/raw/wxcam/PANO

AUTO/LONG/SHORT files remain on the camera host and are not cataloged or archived from this VM.

These in-scope HDR JPG/MP4 files are the durable WXcam source archive and are copied additively to both GWS and object storage. They, rather than the mutable pixel Zarr, provide raw-retention evidence. The catalog, daily videos, and hourly thumbnails are archived separately as products after their 30-hour verification settle window.

Catalog

The WXcam catalog lives at:

  • /data/aurora/products/wxcam/wxcam_catalog.sqlite

It indexes FISH HDR and PANO HDR JPG/MP4 files. Timestamps are derived from filenames and stored as UTC.

The active table is images. Key fields include:

  • image_type - fish_hdr or pano_hdr
  • media_kind - image or video
  • time_utc, time_epoch_ns, day_utc
  • raw_path, relative_path, filename
  • width, height, size_bytes, mtime_ns, indexed_at

Catalog queries should order records by time_epoch_ns and then raw_path. When checked on 2026-05-21, the catalog had no null timestamp rows, no duplicate raw_path rows, all cataloged images were HDR files, and FISH/PANO image and video records were ordered cleanly by timestamp. At that check it contained 48414 FISH HDR image records, 48413 PANO HDR image records, and 1343 MP4 records for each stream. The cataloged image span was 2026-01-12 02:25:00 to 2026-05-21 20:10:39; cataloged videos ran through 2026-05-21 19:00:00.

Read-only standby access

The development host receives this catalog by replication and serves it from data-ocean.gamb2le.co.uk without running WXcam writer timers. Dashboard readers must therefore treat the catalog as read-only:

  • use open_catalog(path, readonly=True) for viewer queries
  • do not call ensure_schema() from viewer/query helpers
  • keep schema creation and migration in the catalog writer/indexer path

The read-only helper first tries SQLite mode=ro, then falls back to mode=ro&immutable=1 for replicated catalog files that would otherwise raise sqlite3.OperationalError: attempt to write a readonly database while reading. That failure mode caused a blank data-ocean.gamb2le.co.uk/app page on 2026-06-22, because WXcam panes are constructed at app startup even when the default visible instrument is Aurora Power Supply.

Daily videos and hourly thumbnails

  • daily MP4s: /data/aurora/products/wxcam/daily_videos/<image_type>/YYYYMMDD.mp4
  • rolling latest MP4s: /data/aurora/products/wxcam/daily_videos/<image_type>/latest.mp4
  • hourly thumbnails: /data/aurora/products/wxcam/hourly_thumbnails/<image_type>/YYYYMMDD/

Daily videos are stitched from the 24 hourly MP4 clips for that UTC day. latest.mp4 is stitched from the most recent 24 hourly clips across day boundaries.

The dashboard serves these MP4 files through the static route /wxcam-media/daily_videos/... with a file-mtime query string. This avoids sending tens of megabytes of video data through the Panel websocket and allows browser video controls to use HTTP range requests.

Dashboard usage

  • Interactive Data Browser uses the stitched MP4 products
  • Science Quicklooks uses the HDR JPG grid, selecting the image nearest the :30 mark in each UTC hour
  • House Keeping Quicklooks uses the catalog and coverage diagnostics

Related page