Cloud Radar Source Sync¶
Source And Destination¶
- Source:
aurora@100.124.55.22:/home/aurora/data/rpgfmcw94 - Target raw directory:
/project/aurora/raw/rpgfmcw94 - Target Zarr:
/data/aurora/products/rpgfmcw94/cloud_radar.zarr - Quicklooks:
/data/aurora/products/quicklooks/cloud_radar
The source stores hourly radar files under a recursive Yyyyy/Mmm/Ddd/ tree.
Together, the live and bounded backfill lanes preserve the complete native tree, including LV0/LV1 binaries,
LV0/LV1 NetCDF, and instrument JPG files. Relative paths are unchanged from
edge to cloud raw storage, GWS, and object storage:
/home/aurora/data/rpgfmcw94/Yyyyy/Mmm/Ddd/...
-> /project/aurora/raw/rpgfmcw94/Yyyyy/Mmm/Ddd/...
-> /gws/ssde/j25b/gamb2le/data/incoming/aurora-cloud/raw/rpgfmcw94/Yyyyy/Mmm/Ddd/...
-> s3://gamb2le-o/data/incoming/aurora-cloud/raw/rpgfmcw94/Yyyyy/Mmm/Ddd/...
Only *LV1.NC files feed the dashboard Zarr and its processing-watermark
gate. The other native file classes are archived and verified without being
passed to the LV1 processor.
Authentication¶
Radar source sync uses Tailscale SSH over the tailnet IP without private keys.
The script uses regular ssh for rsync compatibility, with identity-file and
public-key authentication disabled:
Maintained behavior¶
The source has two independent lanes:
aurora-radar-source-sync.serviceis the priority live lane. It polls every two minutes, uses a bounded bootstrap lookback on first start, overlaps its previous checkpoint by ten minutes, and ignores files modified in the last five minutes. Discovery quotes the remote wildcard before SSH invokes its shell. A live batch is limited to 8 GiB, with a mounted capacity check for all selected uncompressed bytes plus 8 GiB before copying. This lane selects only*.LV1.NCthroughradar_live_source_pattern; rsync compression reduces transport bytes without changing file contents. Its checkpoint is/var/lib/aurora-cloud/radar-live-sync.last.aurora-radar-backfill.serviceis the newest-first historical lane. It transfers a bounded batch in parallel and records progress in/var/lib/aurora-cloud/radar-backfill-status.json. It never advances the live checkpoint.
This separation prevents a historical LV0 archive catch-up from delaying new
radar data. Backfill keeps the independent radar_source_pattern: '*', so all
native classes remain in scope. A fresh dashboard is not proof that those
native files have arrived or reached both archives. Keep that outstanding
native backlog visible, and keep backfill paused during a capacity hold.
The backfill timer is enabled only on the authoritative writer
host; its temporary files live below .radar-partials and are excluded from
GWS replication.
For recovery, preserve the live checkpoint and copy one reviewed closed time window. Stop the live timer and verify its service is idle. The finite recovery mode applies the same capacity and batch limits and does not advance the normal checkpoint:
sudo systemctl stop aurora-radar-source-sync.timer
sudo -u aurora /usr/local/bin/aurora-radar-sync --recover-window <start-epoch> <end-epoch>
Use the source files' modification times for those bounds, and keep the end older than the five-minute grace period. Verify the selected source LV1 reaches cloud raw, the Zarr time coordinate and the public API before restoring the live timer. Leave bulk backfill paused until its independent seven-day headroom check passes. A successful empty source scan is not freshness proof.
Install only this wrapper and its shared capacity helper with
playbooks/radar_live_guard.yml. It requires
aurora_radar_live_expected_sha256 for the preserved installed wrapper,
requires an idle service and paused timer, validates Bash syntax and keeps
backups. Review --check --diff first. It does not change timer state, release
application code or touch scientific products.
Processing¶
The radar timers are enabled in Ansible:
aurora-radar-append.timeraurora-radar-quicklooks.timer
aurora-radar-source-sync.timer is enabled once source Tailscale SSH access is
authorized.
Backup and retention¶
The archive scope includes the complete native LV0/LV1 binary, NetCDF, and JPG
tree, not only the LV1 NetCDF files used by the dashboard. Object verification
partitions this multi-terabyte family into bounded year/month shards. ASS keeps
each radar file for at least seven days and the retention helper can remove it
only after exact cloud, GWS, and object-store proof. .radar-partials is local
interrupted-transfer state and is never archive or deletion evidence. See
Backups and Archive Services.
Capacity and interrupted transfers¶
Before backfill touches the raw directory, the guard requires the configured
/project mount. Before each compressed transfer it checks available bytes on
the destination filesystem. With four workers, the admission requirement is
8 GiB + four times the largest file selected in this batch. Every worker
uses the same upper bound, so simultaneous starts are accounted for. This is a
before-transfer reservation, not a quota or protection against unrelated
writers consuming the remaining space. Check the live lane and measured daily
growth separately before resuming either timer.
The helper is /usr/local/lib/aurora-radar-transfer-guard.py. A capacity hold
returns 75 before opening a new transfer. The batch does not report success,
and the live checkpoint stays unchanged. Increasing capacity or reducing the
reviewed worker/batch allocation is the recovery path; deleting raw files is not.
Retries use a unique .incoming.* partial, preserving previous interrupted
attempts. After a complete, size-validated transfer commits its destination,
the helper checks only partials for that exact file: the legacy name and at
most 16 attempts per worker. A partial must be over six hours old and match the
stable complete target byte-for-byte for its entire length. The target is
fsynced, and a SHA-256 audit record is fsynced before the partial is removed.
Mismatches, recent attempts, symlinks and changed files stay in place. This
bounded cleanup never removes a completed observation. Remaining old partials
need a separately reviewed recovery pass; this helper does not sweep the tree.
Deploy the helper and backfill wrapper together while backfill is stopped.
The focused playbooks/radar_backfill_guard.yml requires the current installed
wrapper's SHA-256 in aurora_radar_backfill_expected_sha256, checks the idle
worker and mounted reserve, and backs up both replaced files. Review it with
--check --diff before applying. It does not start or enable either radar timer.
Do not apply the broad source-sync role just to install this guard.
Validate the existing mount and the guard read-only before restoring its timer:
sudo -u aurora /usr/local/lib/aurora-radar-transfer-guard.py capacity \
--root /project/aurora/raw/rpgfmcw94 --required-mount /project \
--workers 4 --max-file-bytes <largest-selected-file-bytes> --reserve-bytes 8589934592
Record the checked reservation, free capacity, deployed source revision and the next scheduled result. Installing this guard is not evidence that the backfill has resumed or that the long-term capacity requirement is met.