Skip to content

WXcam Source Sync

  • Source: aurora@100.124.55.22:/home/aurora/data/wxcam
  • Target raw directory: /project/aurora/raw/wxcam
  • Catalog: /data/aurora/products/wxcam/wxcam_catalog.sqlite
  • Daily videos: /data/aurora/products/wxcam/daily_videos
  • Hourly thumbnails: /data/aurora/products/wxcam/hourly_thumbnails
  • Local derived pixel Zarr: /data/aurora/products/wxcam/wxcam.zarr

The WXcam source contains nested FISH/ and PANO/ trees. The deployed sync retains HDR JPG and MP4 files from both streams locally. AUTO/LONG/SHORT files remain on the camera host and are not cataloged, Zarr-appended, or archived from this VM.

The operational sync scans only the current and previous UTC date folders. It transfers files newer than /var/lib/aurora-cloud/wxcam-sync.last, with a ten-minute overlap for late writes, and advances that checkpoint only after a successful rsync. A missing, invalid, or older-than-window checkpoint resumes at the live edge. Historical backfill is a separate manual operation and cannot block the two-minute live-data timer.

Dashboard behavior

  • Dashboard instrument name: WXcam
  • Interactive Data Browser: primary wxcam browser and player using stitched MP4 products
  • Science Quicklooks: hourly JPG thumbnail grid

The dashboard uses the SQLite catalog plus daily MP4 and hourly thumbnail products for browsing. The science-quicklook grid is driven by the selected HDR JPG stream, while the interactive browser uses stitched HDR MP4 products. The wxcam pixel Zarr is a local mutable working store and starts at 2026-07-04T00:00:00Z. It is reproducible from the archived HDR imagery and is deliberately excluded from the additive GWS and object-store product writers. Immutable raw HDR files, the catalog, daily videos, and hourly thumbnails are archived. The derived Zarr is never accepted as retention evidence.

The catalog, daily-video, and pixel-Zarr timers consume the incrementally updated raw mirror. Fresh in-flight media are deferred until they have settled.

Authentication

The sync uses Tailscale SSH over the tailnet. The rsync remote shell is regular ssh with identity keys disabled:

  • IdentitiesOnly=yes
  • IdentityFile=none
  • PubkeyAuthentication=no
  • StrictHostKeyChecking=accept-new

No private key is installed for this source.

Timers

  • aurora-wxcam-source-sync.timer
  • aurora-wxcam-catalog.timer
  • aurora-wxcam-daily-videos.timer
  • aurora-wxcam-append.timer

The sync script uses /var/lib/aurora-cloud/wxcam-sync.lock so a long-running rsync cannot overlap with the next timer tick.

Backup and retention

Archive and retention scope is deliberately limited to FISH/PANO HDR JPG and MP4 files. AUTO/LONG/SHORT files remain outside this cloud-managed backup and cannot be pruned by it. In-scope HDR media is archived additively to the raw prefixes on GWS and object storage and retained on ASS for at least seven days. The SQLite catalog, daily videos, and hourly thumbnails are product archives; the local mutable wxcam.zarr is excluded because it is reproducible from raw HDR imagery. A fresh product that is not yet archived is reported as pending_upload during its 30-hour settle window; it participates in stable parity only after that window. See Backups and Archive Services.