Backups and Archive Services

aurora-cloud-infra is the authoritative repository for every service that pulls edge data to the cloud, mirrors it to the GWS or object store, verifies archive parity, publishes archive health, or coordinates retention.

The backup model in one minute

AURORA protects prune-managed raw observations in three stages, with four physical copies while a file remains on ASS:

1. the live file on ASS;
2. the operational raw mirror on the production cloud host; and
3. two independent long-term archive copies, one on the JASMIN GWS and one in the gamb2le-o object-store bucket.

The cloud mirror is used for processing and is also checked before deletion, but it is not sufficient on its own. An ASS file older than seven days becomes eligible only after path and size match across the cloud mirror, GWS, and object store; source/cloud/GWS verification also matches mtime, and checksums are compared wherever both inventories provide them. The edge helper rechecks the permitted file's exact path, size, and mtime immediately before deletion.

These other copies serve different purposes and do not replace the two raw archives:

Copy	Purpose	Counts as raw retention evidence?
Production cloud raw mirror	Live processing and the first checked copy	Yes, but only together with GWS and object storage
JASMIN GWS	Additive long-term archive	Yes
gamb2le-o object store	Independent additive long-term archive	Yes
Development/data-ocean mirror	Development availability and UI testing	No
Derived Zarrs, quicklooks, catalogs, and videos	Rebuildable or presentation products	No
Proxmox/PBS guest backups	Recovery of virtual machines and configuration	No; they are not science-data parity evidence

“Additive” means that writers copy new or changed files but never propagate a source deletion to either archive. GWS rsync has no delete option and the object-store writers use rclone copy, not sync.

Repository boundaries

Responsibility	Repository
ASS and APS host configuration and restricted delete helper	aurora-edge-infra
Edge-to-cloud source sync, GWS mirror, object-store mirror, verification, retention orchestration, archive monitoring	aurora-cloud-infra
Read-only presentation of archive health	aurora_cloud_dashboard

The dashboard must not install transfer, verification, or pruning services. It consumes /data/aurora/internal/archive_status/health-v1.json. Its generic operations collector may copy contract metrics into presentation snapshots for compatibility, but it must not SSH-probe the GWS, parse archive manifests, inspect archive writer units, or infer prune readiness.

Canonical stream catalog

inventory/group_vars/aurora_cloud.yml contains aurora_archive_streams. Each entry defines the edge source, cloud raw root, archive-relative path, retention duration, and whether pruning is allowed. Add a stream there before adding service-specific logic.

Radar (rpgfmcw94) uses the same seven-day ASS policy as the other verified raw streams after two consecutive clean GWS/object-store reports. APS power remains non-prunable pending a separate source and archive policy review.

The prune-managed ASS streams are CL61, radar, HATPRO, Vaisala MET, ASFS science, ASFS fast sonic, ASFS fast gas, PDU, WXcam HDR media, and AURORACam. APS Power is archived but explicitly excluded from edge pruning. Files under the bulk raw tree that are not in this catalog can still be copied by the raw writers, but they cannot be deleted from an edge host by the retention system.

Data flow

ASS / APS live files
        |
        | source-sync services
        v
production cloud raw mirror ----> derived products
        |                              |
        | additive GWS writers         | additive product writers
        | additive object writers      v
        v                         GWS + object store
 JASMIN GWS + object store
        |
        +---- independent verification evidence ----+
                                                     v
                                        health-v1.json
                                                     |
                                    raw-only exact signed permit
                                                     v
                              restricted ASS deletion helper

Writers never wait for verification and never delete destination data. In particular, the GWS rsync wrapper deliberately omits every --delete mode so source retention cannot propagate into the archive. Verification observes writers; it does not enable or disable them. A failed or slow verifier therefore cannot stop uploads, although it blocks pruning.

Fresh raw delivery has a dedicated exact-path lane. After a source-sync job successfully lands files on the cloud host, it records their path, size, and mtime in a durable SQLite queue. Every two minutes aurora-archive-dispatch.service selects a bounded newest-first batch and copies only those files to both GWS and object storage. Each destination is tracked independently, so a successful GWS copy is retained while a failed object-store copy retries, and vice versa. The full-tree writers remain enabled as an independent historical backstop.

Dispatch receipts prove that a fast-lane command completed; they are delivery telemetry, not retention evidence. Only complete independent inventories and the signed retention gate can authorize edge deletion.

What is copied and when it becomes evidence

Copy age and verification age are intentionally separate. A writer may upload a fresh file as soon as it has stopped changing, while parity waits longer so live products do not oscillate between “present” and “missing”.

Job	Content	Object-store writer minimum age	Verification age	Archive destination
raw	Complete production raw mirror, excluding temporary radar partials	15 min	6 h	GWS raw and object-store raw
products	Selected products and quicklooks, excluding the WXcam subtree	20 min	30 h	GWS products and object-store products
products-wxcam	WXcam catalog, daily videos, and thumbnails; not wxcam.zarr	30 min	30 h	GWS WXcam products and object-store WXcam products
model-evaluation	Campaign products, dereferencing approved symlinked inputs	1 h	6 h	GWS and object-store model-evaluation roots
manifests	GWS verification evidence, excluding operational logs	5 min	2 h	GWS internal and object-store manifest roots

The raw six-hour horizon is strict: every settled raw file must be present and matching in object storage. The independent GWS stream verifier also publishes all-age counts for visibility and seven-day age-bounded counts for retention. Only the age-bounded raw counts can authorize deletion.

For products, a file younger than 30 hours that is not yet present in object storage appears as pending_upload. Pending is normal delivery lag and does not count as a parity failure or reset the stable-parity streak. If a product is still absent or mismatched after 30 hours, it becomes a real archive gap, turns archive health red, and triggers the exact-path repair service. This is why a newly created quicklook no longer causes a permanent archive alert while an old missing product remains visible.

Common working material is outside verified scope: Git and virtualenv trees, caches, locks, partial or temporary files, SQLite WAL/SHM files, operational logs, and product directories named as backup/schema-backup Zarrs. Symlink pointers are excluded unless a job explicitly uses copy_links; in that case the target bytes are archived and verified as a regular restorable file.

Normal cadence

Schedules are UTC and may include a small randomized delay. A timer activation can be skipped or bounded while an earlier run holds its lock; backlog then continues on the next activation.

Activity	Configured cadence
GWS raw	Every 5 minutes
GWS core products	Every 10 minutes
GWS WXcam products	Every 30 minutes
GWS model evaluation	Every 30 minutes
GWS manifests	Every 10 minutes
GWS source/cloud/archive verifier	Every 10 minutes
Newest-first raw dispatch	Every 2 minutes after a 15-minute settle window
Object-store raw and core products	Every 30 minutes
Object-store WXcam products	Hourly
Object-store model evaluation	Daily at 17:00
Object-store manifests	Hourly
Complete object/GWS inventory	03:20, 09:20, 15:20, and 21:20
Archive-health publication	Every 2 minutes
ASS retention	Daily at 03:30, provided every gate passes

The dispatch queue avoids a recursive discovery scan in the critical path: source-sync jobs provide the exact files that just arrived. It sends at most 5,000 files or 20 GiB per run, ordered by newest mtime, and keeps independent GWS and object-store completion flags. A deployment bootstrap may seed a recent lookback, but normal operation is event-driven by successful ingress.

The full object writers reserve a bounded newest-first slice and then a bounded full-history slice. A large backlog therefore converges over multiple cycles without preventing new observations from receiving priority.

The full-history raw writer interleaves two bounded phases on every activation: the newest two days are copied first, then a full-history backfill slice runs. This prevents a multi-terabyte backlog from starving current observations. High-cardinality product, camera, and manifest writers use the same two-phase pattern: a short newest-first slice followed by a bounded full-history slice. Thus every timer cycle reconsiders newly published chunks while historical gaps continue to converge instead of falling permanently outside a lookback. Full product inventories list each source-present product family in smaller parallel shards; they never depend on one unbounded recursive object-store root listing. “Incremental” here means that families are verified as bounded, independent shards within a run. The complete report is published only after all required shards finish, so a partial run can never replace good evidence. Independent top-level jobs also run concurrently within the global process limit. Local inventory walks prune excluded directories before descending, so the verifier no longer scans the excluded 635-GB WXcam pixel Zarr or its own history tree. The manifest job excludes immutable history/ and operational logs/; those files are audit storage, not science parity evidence. Raw inventories use the same rule for every family, including the multi-terabyte radar archive. Families are scheduled independently, radar is listed as bounded year/month subtrees, and a global process semaphore limits nested listings to the configured object_store_inventory_process_limit (16 in production, matching the cloud host's CPU count). Each listing also has a 60-minute outer process guard; rclone's shorter inactivity timeout and bounded retries still detect dead connections without rejecting valid high-cardinality listings. Model-evaluation campaign data has independent additive writers to both GWS and object storage; it is not implicitly covered by the products job. Symlinked runtime inputs are dereferenced by both writers and verified as regular, restorable files under their campaign-relative paths.

WXCam's live wxcam.zarr is a mutable derived working store and is intentionally excluded from both product archives. Its immutable raw HDR imagery, catalog, daily videos, and hourly thumbnails remain covered. Because the Zarr is reproducible from archived imagery, it is never accepted as retention evidence. Production no longer appends to this redundant pixel cache. Its guarded cleanup tool requires fresh green strict archive evidence, stable object-store parity, zero raw gaps, zero GWS stream issues, and a disabled appender before it can remove the directory.

When an inventory publishes exact missing or mismatched paths, aurora-object-store-repair.path starts the catalog-driven repair service. It revalidates every settled source file, rejects paths outside the configured source root, follows symlinks only for jobs explicitly marked copy_links, orders candidates newest first, and performs only exact rclone copy --files-from-raw operations. It never deletes or broadly rewalks the archive to repair a known finite gap. An outer graceful GNU timeout enforces each wall-clock budget because the deployed legacy rclone can stop transfers yet continue scanning after its own --max-duration deadline.

Playbooks

• playbooks/archive_services.yml configures the complete archive stack.
• playbooks/gws_mirror.yml configures the additive GWS writer and verifier.
• playbooks/object_store_mirror.yml configures only non-destructive object storage and archive-health publication.
• playbooks/retention_coordinator.yml installs the fail-closed cloud coordinator while allowing its timer to remain explicitly disabled.
• playbooks/archive_dashboard_consumer.yml releases only the pinned dashboard consumer and points it at the read-only health contract. It does not install or reapply the contract producer, source-sync, archive writer, verifier, or retention roles.
• playbooks/dashboard_runtime_release.yml deliberately excludes every archive role. Use archive_services.yml for archive changes and site.yml for a complete host rebuild.

Always run --check --diff before applying a playbook.

Monitoring contract

aurora-archive-health.timer publishes schema health-v1 every two minutes. The contract contains:

• per-stream GWS missing and mismatch counts;
• per-stream coverage and retention readiness;
• per-job and aggregate object-store missing and mismatch counts;
• per-job pending_upload counts for files still inside their verification settle window;
• per-job and aggregate direct GWS missing and mismatch counts for raw, products, WXCam products, model evaluation, and manifests;
• every catalog stream's source-sync timer and service state, including the independent radar and AURORACam historical backfill lanes;
• raw, product, WXCam product, model-evaluation, and manifest GWS writer states;
• every GWS/object-store writer and verifier service, timer, repair path, and verification-gate path state;
• verification timestamps and the object-store clean streak.
• newest-first delivery queue depth and bytes, per-destination pending counts, oldest pending age, last result, and last successful delivery time;
• a human-readable operator_status with separate level, title, detail, and whether pruning is paused.

The inventory itself atomically updates /data/aurora/internal/object_store_manifests/progress.json with its state, current job, current phase, completed jobs, and total job count. A heartbeat refreshes updated_at every minute even while one historical object-store listing is still running. The health contract embeds that evidence and publishes a numeric running-state metric, so consumers can distinguish a slow healthy scan from a stalled one without inspecting processes or inferring progress from partial manifests. While the inventory service is running, a heartbeat older than five minutes is an authoritative health failure. Complete inventories are retained as immutable history, bounded by object_store_inventory_history_keep (12 reports in production). This preserves multiple independent proof runs without allowing large TSV evidence snapshots to grow without limit. Operational transfer logs are local diagnostics, not archive data. Both manifest writers explicitly exclude the root logs/ tree; any legacy additive copies are ignored by parity checks and are never retention evidence.

Missing or stale verification must be treated as unsafe for pruning, but it is not automatically evidence of lost data. When the previous complete report is clean, measured gaps remain zero, and a healthy inventory is running (or a remote listing timed out), the operator status is amber and says that verification is running or delayed and pruning is paused. Confirmed missing or mismatched settled files, a failed writer, or an unsafe last complete report is red. Neither state stops additive writers.

The status terms have precise meanings:

State	Meaning	Operator action
pending_upload	File is newer than the job's verification horizon and is not yet required for parity	Observe only; this is not an archive failure
dispatch queue pending	Exact recently landed raw files still need one or both archive copies	Observe progress; investigate if the oldest item keeps aging or the worker fails
missing or mismatch	A settled file is absent or differs at a destination	Let exact repair run, then verify again
inventory running with a recent heartbeat	A complete sharded scan is still progressing	Wait; do not infer a stall from report age alone
previous report clean, verification delayed	Current proof is unavailable but there is no measured gap	Amber; pruning is paused until a complete audit succeeds
inventory heartbeat older than five minutes while running	Verifier is stalled	Investigate the inventory service and its current shard
clean=true, streak 1	One complete clean report	Not yet stable parity
stable_parity=true	Two distinct complete clean reports	Global object-store stability gate is satisfied
stream prune_ready=true	That raw stream has exact age-bounded GWS/cloud candidates	Necessary but not sufficient for deletion

The health-v1 producer is the only code allowed to turn archive evidence into operator state. Browser, mobile API, reports, and notification code are read-only consumers of that result. This includes source-sync systemd health: dashboard collectors must not keep a second list of source-sync units or probe those units directly.

A fresh product can be pending without hiding raw archive health. A settled product gap remains a real global archive-health failure, so aggregate object parity cannot be green while an older non-raw job still has a gap.

Safe convergence runbook

Run these on the cloud host. None of them enables pruning:

systemctl status aurora-mirror-verify.service
systemctl status aurora-archive-dispatch.service
systemctl status aurora-archive-dispatch.timer
systemctl status aurora-object-store-inventory.service
systemctl status aurora-object-store-repair.path
systemctl status aurora-object-store-verification-gate.path
systemctl is-enabled aurora-ass-retention.timer
cat /data/aurora/internal/archive_dispatch/status.json

1. Leave every additive writer timer running.
2. Start one full inventory with sudo systemctl start --no-block aurora-object-store-inventory.service.
3. Wait for atomic publication at /data/aurora/internal/object_store_manifests/latest/comparison.json. The contract keeps the two archive layouts explicit: object storage preserves settled cloud-ingress relative paths, while raw data on GWS uses the canonical per-stream Y/M/D hierarchy. source_vs_s3 therefore proves cloud-to-object parity, and source_vs_gws is built from the independent canonical edge-source/GWS manifests for raw streams. Products, WXCam products, model evaluation, and manifests use an independent direct GWS inventory through a JASMIN transfer host. The service never compares the two deliberately different raw path layouts directly. The verifier publishes its complete latest/ tree by atomic directory replacement, so readers cannot combine manifests from different verification runs. Object parity excludes symlink pointers and re-stats every source file after the remote listing; anything changed during that window is deferred to a later run rather than reported as a destructive size mismatch. The parity snapshot trails live raw writers by six hours, derived products by 30 hours, model-evaluation data by six hours, and continuously changing verifier manifests by two hours. Fresh product files are published as pending_upload instead of gaps. These horizons remain well inside the seven-day edge-retention window. Raw GWS parity is read from one immutable verifier history directory and applies explicit age-bounded retention counters. verification_settle_age controls only this proof horizon. The independent settle_age used by additive copy writers remains 15 minutes for raw, 20 minutes for products, 30 minutes for WXCam products, one hour for model evaluation, and five minutes for manifests; verification must never slow delivery of fresh data.
4. The repair path unit copies only the exact reported missing or mismatched paths. It must finish successfully before another inventory is started.
5. Run a fresh inventory. A report is clean only when every settled job has zero gaps and mismatches against both GWS and object storage and all retention-age raw GWS counters are zero. Pending product uploads do not count as gaps. The clean report establishes clean streak one.
6. Run another independent full inventory. Only that distinct second clean report may establish stable parity.
7. Confirm health-v1.json is green and review both cloud and edge audit logs before any dry-run retention canary.

After deploying the fast lane, seed only recent arrivals and run one bounded batch; the historical writers continue to own older convergence:

sudo -u aurora /usr/local/bin/aurora-archive-dispatch scan --job raw --lookback-hours 48
sudo systemctl start aurora-archive-dispatch.service
sudo -u aurora /usr/local/bin/aurora-archive-dispatch status

The rebuildable WXcam pixel cache is a separate cloud-capacity operation. Run the command without --apply first. The tool refuses both modes unless the appender is disabled and strict archive evidence is green; apply mode writes an audit receipt before and after removal:

sudo /usr/local/sbin/aurora-cloud-cache-cleanup
sudo /usr/local/sbin/aurora-cloud-cache-cleanup --apply

The role defaults keep aurora-ass-retention.timer disabled and dry-run on. The committed production host overrides enable the reviewed live policy; ASS enables the restricted helper in live mode, while APS remains disabled and non-prunable. A failed, interrupted, or partial inventory never replaces the previous complete comparison and can never authorize deletion. Always check the live unit and both audit logs rather than treating committed configuration as proof of the current host state.

Retention permits are signed by the cloud coordinator using ass_retention_signing_private_key; edge helpers trust only the corresponding root-managed public key. Never commit either private-key material or an unencrypted key variable. Each permit is also built from the immutable GWS history directory named by its verification timestamp, never from a sequence of reads against the moving latest pointer. Missing signing, verification, or immutable-snapshot evidence fails closed.