Backup and restore

Hikyo backup and restore are host-only commands. They operate on the configured datastore and have no network API.

Add global -v, -vv or -vvv before the command to see phases, operational details or timings, for example hikyo -vvv backup export. Archive contents, unlock material and root keys are excluded from these diagnostics. See CLI verbosity for the shared output contract.

What an archive protectsCopy Anchor Link

One age-encrypted archive contains a consistent datastore snapshot and its wrapped key hierarchy. The export process does not read the Hikyo root key.

Decrypting the archive requires the backup identity. Reading restored values also requires the original root key. Store those two secrets separately.

1. Generate a backup identityCopy Anchor Link

hikyo backup keygen --output-file /secure-custody/hikyo-backup-identity

The private identity is delivered once. The command prints its public recipient, which is safe to place in server configuration.

Do not store the identity beside the root key. Two files in one password manager are one failure domain.

2. Configure automatic pre-migration exportsCopy Anchor Link

HIKYO_BACKUP_RECIPIENTS=age1...
HIKYO_BACKUP_DIR=/var/backups/hikyo

Before applying a pending migration, Hikyo writes an export when this policy is complete. A configured export that fails blocks the migration.

No configured policy causes a loud, durable skip but does not block migration. There are no down migrations, so the only downgrade path is restore.

2b. Schedule automatic exportsCopy Anchor Link

A complete export policy (recipients plus a destination) also enables an in-process daily export and a retention prune. Both run on the same scheduler as payload GC: once on startup catch-up and then hourly, doing work only when it is due.

HIKYO_BACKUP_INTERVAL=24h       # minimum 1h; export runs at most this often
HIKYO_BACKUP_RPO=26h            # at least the interval; health fails past it
HIKYO_BACKUP_RETAIN_COUNT=7     # protect newest archives within 180-day ceiling
HIKYO_BACKUP_RETAIN_DAYS=180    # earlier age threshold outside retained count
HIKYO_BACKUP_RTO_TARGET=30m     # the restore drill's pass/fail clock

A scheduled export that fails is loud and durable: it records a backup.export_failed event on the instance trail and updates the health row, and it never falls back to an unencrypted archive. The recovery point objective is a target gated on monitored success, not a promise: if no successful export is younger than HIKYO_BACKUP_RPO, health reports the breach and hikyo doctor returns an error.

The retention prune manages complete hikyo-<engine>-<timestamp>.age archives for this engine, including numeric collision suffixes. It leaves .partial staging files, other engines and unrecognized files alone. Scheduled, manual and pre-migration archives in HIKYO_BACKUP_DIR share one policy; there is no separate pre-migration count-three limit. Scheduled pruning scans only that directory.

Within an independent 180-day ceiling, the newest HIKYO_BACKUP_RETAIN_COUNT are protected from the earlier HIKYO_BACKUP_RETAIN_DAYS threshold. Other archives older than that configured threshold are deleted oldest-first. Count is a protection, not a maximum number of young archives. Archives older than 180 days are eligible even when fewer than the retained count exist. Exact equality does not trigger this ceiling, but the earlier configured age threshold can still apply outside retained count. HIKYO_BACKUP_RETAIN_DAYS accepts no value above 180 and has no unlimited option.

Recovery takes precedence in two visible exceptions. The last durably recorded successful archive remains protected until a replacement is durably recorded, even beyond 180 days or after a wall-clock rollback. A newer filename or a published but unrecorded file cannot replace that recovery point. If success state is missing/unreadable, the recorded artifact is absent or clock/custody is uncertain, pruning retains recovery candidates and reports the uncertainty. Successful export records bind a SHA-256 digest of the ciphertext. Legacy success records without a digest remain uncertain until a fresh export succeeds; filename and byte count alone cannot establish recovery custody. That can preserve more than one expired archive. Durable export success is not a substitute for the recurring readable-restore drill below.

Retained expired archives cause an immediate retention breach. Prune health records the attempt, bounded failure reason and retained-expired count separately from export health. A breach, failed deletion or uncertainty does not advance the successful-prune timestamp. Health also reports unverified or stale pruning; an old successful timestamp is not current compliance evidence. Fix export or storage failures and establish a new durable recovery point, then confirm a successful prune and cleared retention findings. Exceptions can persist indefinitely; investigate them instead of treating configured days as proof of erasure.

HIKYO_BACKUP_DIR must belong to one instance. Archive names carry no instance identity, so two instances of the same engine sharing a directory would prune each other’s archives. Give each instance its own directory (its own subdirectory of a shared mount is fine).

For personal-data expiry and deletion after restoration, follow personal data and retention.

Off-host storageCopy Anchor Link

Hikyo never reaches a remote object store itself: HIKYO_BACKUP_DIR is a local path. Make it a mount the operator manages (NFS, SMB, an rclone/s3fs FUSE mount) so the archives land in a different failure domain from the instance. The 180-day policy and its recovery exceptions only govern copies this instance’s pruner manages; any copy your tooling makes outside HIKYO_BACKUP_DIR extends the erasure window and is your responsibility to age out.

Under multi-node HA the scheduled jobs run only on the lease holder, so HIKYO_BACKUP_DIR must be a mount every node shares: otherwise a failover moves where archives land and the pruner on the new leader manages a different directory than the one the health row describes.

Final publication, success recording and pruning hold a directory advisory file lock across their filesystem verification/deletion and datastore outcome. The shared mount must support reliable cross-process advisory locking for all publication/record/prune writers, including writers on different HA nodes. A mount that merely supports creating and reading archives does not establish that requirement. The retention checks read ciphertext and public metadata without loading custody keys.

An absent managed backup schedule alone produces a not_scheduled warning, not a retention breach or doctor error. This applies both before any managed backup exists and after a fresh successful manual export/prune. Recorded prune evidence remains visible; a false breach flag does not certify an unchecked installation. Observed expired archives, failed/incomplete or uncertain prune attempts, and clock uncertainty remain errors even after scheduling is disabled. A configured schedule with missing or stale checks also remains unhealthy.

Custody separationCopy Anchor Link

Two secrets, two stores, two failure domains. The backup identity decrypts the archive container; the root key decrypts the values inside it. Export reads neither. Keep the age identity and the root key in separate custody stores; two files in one password manager is one failure domain wearing two names.

AlertingCopy Anchor Link

The operational /metrics endpoint exposes label-free gauges to alert on:

• hikyo_last_backup_export_success_timestamp_seconds (0 when never)
• hikyo_backup_rpo_exceeded (1 when the RPO is breached)
• hikyo_last_backup_prune_success_timestamp_seconds
• hikyo_backup_retention_breach (1 for breached or unverified retention)
• hikyo_backup_retained_expired_archives (last-observed expired archive count)
• hikyo_last_restore_drill_timestamp_seconds
• hikyo_restore_drill_ok

hikyo doctor folds the same state into a backup-rpo finding (error when exceeded) and a restore-drill finding (warning when the last successful drill is over 90 days old or has never run), so a cron-driven doctor check fails loudly. /healthz stays liveness-only on purpose: a stale backup is not a reason to restart the process.

Monitor backup-retention as well as RPO: a recent export does not prove that expired archives were deleted. Doctor reports severity error for breached or unverified retention. The backup object on GET /instance/retention-health exposes:

Alert on the breach gauge, pruning errors and absent/stale compliance evidence. The expired count is an observation, not a live directory scan; zero alone does not prove compliance. The successful-prune gauge above stays unchanged after failed checks or a run that preserved an expired recovery exception. Operational logs report the same retention outcome.

A prune records in_progress before hashing the protected ciphertext. During that work, even a healthy run reports retention_breach=true, sets the breach gauge to 1, and makes doctor refuse. A successful compliant run clears the signal. Hashing a large archive can take substantial time: choose an alert’s for: duration using measured normal hash-and-delete times, and investigate in_progress attempts that outlast that expectation using last_prune_attempt_at. A persistent incomplete attempt is not verified compliance; do not suppress its health signal.

3. Run a manual exportCopy Anchor Link

hikyo backup export

The command refuses to write an unencrypted archive. You may override the destination with --out and repeat --recipient for a complete replacement recipient set.

An export with --out OTHER_DIR can replace the recorded recovery point while placing the archive outside HIKYO_BACKUP_DIR. Scheduled pruning then reports recovery_uncertain and preserves candidates because it cannot verify that recovery point in the managed directory. Run hikyo backup export without --out to record a fresh recovery point there, then rerun pruning. Independently managed copies outside that directory need their own retention controls.

Verify the reported path, byte count, engine, and schema version. Move a copy to storage that does not share the instance’s failure domain.

An export reports success only after syncing the complete encrypted file and its destination directory ancestry. The destination filesystem must support these durability operations. If directory sync fails after publication, the command names the retained archive and reports backup artifact published but durability unconfirmed. Inspect that artifact and fix the storage error; the failed export does not advance the successful recovery point or permit a configured pre-migration export to continue. Retrying preserves the earlier artifact and repeats the durability checks.

4. Restore into an empty targetCopy Anchor Link

Stop the server. Configure HIKYO_DB for a new SQLite path or an empty PostgreSQL database, then run:

hikyo restore run \
  --from /var/backups/hikyo/<archive> \
  --identity-file /secure-custody/hikyo-backup-identity

Hikyo authenticates the complete archive before touching the target. It refuses engine mismatches, unsupported schema versions, and non-empty targets.

Use the original root key when starting the restored server. The backup identity alone cannot make restored values readable.

5. Reconcile identitiesCopy Anchor Link

A restore advances the credential epoch. Every restored password, factor, passkey, session, bearer credential, federation link, and single-use authority is inert.

hikyo restore status
hikyo restore reconcile --principal <principal-id>

Reconcile one human principal at a time after confirming it should regain its restored grants. Re-mint and redistribute machine bearer credentials; they are never reactivated.

6. Recurring restore drillCopy Anchor Link

Run a drill quarterly (the ops spec’s cadence; doctor warns past 90 days). The drill rehearses the whole recovery against an empty scratch target and never touches the live datastore except to record the result. It is a host-only operator verb, like restore run.

hikyo restore drill \
  --from /var/backups/hikyo/<archive> \
  --identity-file /secure-custody/hikyo-backup-identity \
  --root-key-file /secure-custody/hikyo-root-key \
  --principal <human-principal-id> \
  --project <org-id>/<project-id> \
  --target-sqlite /tmp/hikyo-drill.db \
  --cleanup -o json

For a passphrase-encrypted archive, pass --passphrase-file PATH instead of --identity-file (they are mutually exclusive, as for restore run). For a PostgreSQL scratch target, pass --target-postgres-dsn-file PATH (a file holding an empty database’s DSN) instead of --target-sqlite; --cleanup removes a SQLite scratch file after a successful drill but leaves a PostgreSQL scratch database for you to drop, since dropping a database is a DB-admin task, not something the server does through a datastore handle.

A passing drill proves all of these, in order:

1. The archive decrypts and restores into an empty target.
2. The original root key, supplied separately, boots the restored data.
3. One stored secret decrypts through the key hierarchy.
4. One approved human principal is reconciled.
5. A machine credential is minted, then revoked.
6. The whole sequence finished within HIKYO_BACKUP_RTO_TARGET.

The security admission check inventories executable SQLite schema objects even when an archive uses reserved sqlite_ names. Only exact engine bookkeeping definitions are excluded. PostgreSQL restore refuses counters in the upper half of their canonical sequence range, leaving substantial capacity for future audit events rather than accepting an exhausted archive. If a genuine installation has reached that range, migrate its counters through a supported operator procedure before restoring; do not edit the archive to bypass admission.

Restore destroys both active adapter credentials and pending origin-move credentials, clears pending destinations, and cancels the moves. Re-enter provider credentials and review routing before starting a new move. Reconciled grants alone cannot resume an archive-supplied activation using a retained PAT.

The drill records the archive identity and digest, binary and schema versions, elapsed time and RTO verdict on the live instance’s audit trail and health row. The root key and backup identity are read from files, used for the drill’s duration only, and never persisted or logged. The scratch target is left for inspection unless --cleanup was requested and the drill passed.

Include both readable restore and current root escrow verification in the quarterly runbook. A successful drill proves the archived hierarchy can be opened. To record verification against the current installation, recover a separate private root file from escrow and run this host-local command.

On a Linux systemd host that hikyo upgrade has set up, root runs it directly. The command finds the service’s environment files and root key through /etc/hikyo/upgrade.json, runs as the service user, and hands the recovered file to that user on a descriptor, so the copy can stay root-owned:

sudo hikyo escrow verify --root-key-file /root/recovered-root --assert-separate-custody

Elsewhere, supply the server’s datastore, public upgrade bundle and operator pin configuration yourself:

HIKYO_ROOT_KEY_FILE=/run/hikyo-root/root-key \
  hikyo escrow verify --root-key-file /separate-custody/recovered-root \
  --assert-separate-custody

HIKYO_ROOT_KEY_FILE identifies the actual configured server root file and is also supported by hikyo server; the server flag takes precedence. Verification refuses an unidentified or environment-only server root source. The command refuses the configured server root file, hardlink aliases, symlinks and files that are not private and owned by the operator or root. It verifies the complete current key hierarchy and atomically binds the public timestamp to the finalized root epoch, installation and recovery incarnation. Root rotation or restore makes the old record stale; dual-wrapped root rotation must finish before verification. Private key bytes never enter an API or audit payload. The flag records the operator’s separate-custody assertion: software cannot prove that a copy was physically offline. An export job alone is not a recovery test.

With --cleanup, SQLite scratch files are removed only after every drill step and its recovery-time check pass. A failed or refused drill retains the target for inspection, including any pre-existing database. PostgreSQL scratch cleanup remains an explicit database-administrator action.

Upgrade proof and restored admissionCopy Anchor Link

Ordinary exports and recurring drills do not issue upgrade permission. Use hikyo backup upgrade-export and the operator-held hikyo backup upgrade-drill to produce the public receipt and signed readable restore attestation required by a populated upgrade. Follow the exact commands in the manual upgrade runbook. The server receives the ciphertext and public proof, never the operator signing key, age identity or separate backup root escrow.

After supported restore, keep the current external installation state and operator pin. An old database backup must never replace that state. Restore advances the credential epoch and creates a new recovery incarnation; old attestations and pending upgrade authority remain invalid. Before starting the restored instance, export and drill again against that current incarnation and complete the same-release recovery gate with the exact signed source binary. restore status and individual principal reconciliation remain available through narrow local recovery commands while ordinary serving is blocked. The original root key alone does not reactivate restored serving or bearer credentials.

Managed Hikyo configurationCopy Anchor Link

For owner-local mail and release-notification settings, use Manage Hikyo with Hikyo: edit the protected project, publish an exact revision, reauthenticate and apply. That guide covers adoption, per-node convergence, independent remote projects and rollback.