Multi-node high availability

Multi-node HA lets one logical Hikyo instance run across several server nodes behind a single endpoint, so ordinary traffic survives the loss of a node and singleton background work runs under one fenced leader. A newly elected leader performs startup catch-up again; leadership does not promise exactly-once job execution across failover. It is application-tier HA, not cross-region disaster recovery: backups and restore remain the job of backup and restore.

HA is opt-in and Postgres-only. A single-node deployment (the default) is unchanged.

What HA requiresCopy Anchor Link

• An externally highly available PostgreSQL datastore. Hikyo coordinates through Postgres; it does not replicate the database itself. Running Postgres in an HA topology (a managed service, or Patroni, or similar) is the operator’s responsibility, exactly as for a single node. SQLite is single-writer and is refused under HA at boot.
• Stable, unique node identities. Each node sets HIKYO_NODE_ID. The Helm chart derives it from the pod name.
• One shared root-key authority. Every node mounts the same root key (rootKey.existingSecret). A development auto-generated root key is per-node and is refused under HA. A node whose root-key fingerprint disagrees with the installation refuses to serve.

Any of these missing or inconsistent is a boot refusal, never a silent single-node fallback.

TopologyCopy Anchor Link

              one endpoint (Service or load balancer)
                              |
        +---------------------+---------------------+
        |                     |                     |
     node A                node B                node C
        \                     |                     /
         +----- highly available PostgreSQL --------+

Sessions, CSRF tokens, and every authentication ceremony are stored in Postgres, so the load balancer needs no session stickiness: any node can serve any request. Point the load balancer’s health check at /readyz on the operational port. A node whose lease datastore is unreachable, or whose schema lags the database, reports not-ready and is taken out of rotation.

The MCP endpoint follows the same rule. It emits no session ID, keeps no connection-local state, and seals cursors with the shared root-key authority. Successive pages may reach different replicas. See the MCP operations runbook for the deployment and public checks.

Enable it in the chart:

ha:
  enabled: true
  replicaCount: 3
  minAvailable: 2

Coordination and failoverCopy Anchor Link

One node holds a fenced datastore lease and runs the singleton background work (retention sweeps, scheduled backups). The lease is renewed on a heartbeat; a node that cannot renew within the lease TTL drops leadership and cancels its in-flight singleton jobs, so a stale leader cannot keep running after losing authority. Another node then acquires the lease and continues.

• Lease TTL: 30 seconds. Heartbeat: 10 seconds.
• Declared RTO for singleton work = lease TTL + the load balancer probe period. Ordinary API, CLI, and browser traffic is served by every node and is not interrupted by a leader change; only the singleton work pauses for the RTO.

Installation-wide security state is shared across nodes: pre-authentication rate limits (per source IP, per account, per issuer) live in the datastore, so hopping between nodes cannot bypass a limit or a backoff. The per-node concurrency limit on password verification stays local to each node, because it bounds that node’s own memory.

Node clocks must be reasonably synchronised (NTP). Lease acquisition compares the acquiring node’s clock against the recorded expiry, so clock skew larger than the heartbeat is a split-brain risk.

UpgradesCopy Anchor Link

Rolling replacement is not an upgrade procedure. An HA upgrade is an operator-enforced full stop of every replica, one verified migrator, then an exact-target boot of the new image:

1. Scale the replicas to zero and confirm every old process has exited.
2. Run the verified migration once (hikyo migrate against the target release under the controlled migration gates).
3. Boot the replicas on the exact target release.

Never restart replicas on the old image after the migration lands, and never roll replicas while the schema is moving.

Node replacement and capacityCopy Anchor Link

• Replacing a node: start a replacement with the same shared root key and a fresh HIKYO_NODE_ID. It registers itself; the old node’s registry row is swept after it stops heartbeating. No manual datastore surgery is needed.
• Capacity floor: each node must meet the same per-node resource floor as a single-node deployment (see self-hosting). HA adds availability, not per-node headroom: size each replica as you would a single node, and keep at least minAvailable replicas schedulable.

Root-key rotation under HACopy Anchor Link

Rotating the root key changes the fingerprint every node checks. Because a mixed-fingerprint state is refused, rotate with a full stop: stop all replicas, rotate the root key, then start them again on the new key. Do not perform a rolling root-key rotation.

ObservabilityCopy Anchor Link

/metrics exposes three label-free gauges:

• hikyo_ha_is_leader (1 on the node holding the scheduler lease),
• hikyo_ha_nodes_seen (live nodes in the installation),
• hikyo_ha_lease_age_seconds (age of the current lease, sampled per tick).

Known boundsCopy Anchor Link

• Cross-node change advisories fall back to revision polling; a change made on one node is visible on another within the poll interval rather than instantly.
• Pre-authentication rate windows are fixed one-minute buckets, so an attacker can spend up to two windows’ allowance across a boundary. The concurrency limit still bounds the actual verification work.
• After a rotate-dek --instance, a node picks up the new instance key within one heartbeat.

Managed Hikyo configurationCopy Anchor Link

For owner-local mail and release-notification settings, use Manage Hikyo with Hikyo: edit the protected project, publish an exact revision, reauthenticate and apply. That guide covers adoption, per-node convergence, independent remote projects and rollback.