Self-hosting baseline

Production startup is fail-closed. Hikyo refuses to serve without an explicit datastore, root-key source, authenticated release bundle and independent installation custody. A non-loopback public listener requires native TLS or an explicit trusted-proxy boundary.

This page is the production checklist. Use the linked pages for the full configuration and recovery procedures.

For AWS, Azure, Google Cloud, DigitalOcean, or Hetzner, start with cloud deployment for provider-specific setup and infrastructure launch options.

For Kubernetes, use the complete Helm installation example.

For data handling, control ownership and evidence, read self-hosted compliance, the personal-data inventory and operator procedures.

Required decisionsCopy Anchor Link

Install a verified production releaseCopy Anchor Link

Use installation to verify a published artifact, then complete the fresh native installation runbook. Signed nightlies remain prereleases. Every production boot authenticates its exact release and public bundle; a direct source build lacks those release claims and is refused for production serving.

For development and inspection, a reviewed source checkout can build the UI and binary with:

npm install --global --ignore-scripts corepack@0.35.0
corepack enable
corepack install --global pnpm@11.24.0
pnpm --dir clients/ts install --frozen-lockfile
pnpm --dir web install --frozen-lockfile
pnpm --dir web build
go build -trimpath -tags ui -o ./hikyo ./cmd/hikyo
./hikyo version

Record the commit beside a source build and use it only for development. Do not substitute it for the signed executable in the production example below.

SQLite baselineCopy Anchor Link

First provision the signed binary, public bundle and operator pin at the paths below, and prepare persistent installation custody using the native installation runbook. The following settings are a runtime example, not a release installer.

install -d -m 0700 /var/lib/hikyo /etc/hikyo
openssl rand -hex 32 | install -m 0600 /dev/stdin /etc/hikyo/root.key

HIKYO_DB=sqlite:/var/lib/hikyo/hikyo.db \
HIKYO_EXTERNAL_ORIGIN=https://hikyo.example.com \
HIKYO_UPGRADE_BUNDLE=/srv/hikyo-public/bundle \
HIKYO_UPGRADE_OPERATOR_PUBLIC_KEY=/srv/hikyo-public/operator.pub \
HIKYO_UPGRADE_STATE_DIR=/var/lib/hikyo-installation \
./hikyo server \
  --listen 0.0.0.0:8443 \
  --operational-listen 127.0.0.1:8081 \
  --tls-cert-file /etc/hikyo/tls.crt \
  --tls-key-file /etc/hikyo/tls.key \
  --root-key-file /etc/hikyo/root.key

Install the TLS private key as mode 0400 or 0600. The bootstrap pair is imported once into the managed node configuration; files are not watched and SIGHUP does not reload them. Renew by publishing and applying the node certificate through managed configuration.

For reverse-proxy mode, omit the TLS flags, bind the public listener privately, set HIKYO_TRUSTED_PROXY_CIDRS to the exact proxy networks, and set HIKYO_EXTERNAL_ORIGIN explicitly. Do not expose that plaintext listener.

Setting an https external origin is also what turns HSTS on: Hikyo emits Strict-Transport-Security: max-age=31536000 whenever the configured public origin is https and its browser-visible host is not loopback, whether TLS is terminated natively or at a proxy whose Hikyo backend binds to loopback. Configure the proxy to pass that header through rather than adding its own. A proxy that appends a second Strict-Transport-Security, or overwrites it with a shorter max-age, gives browsers a policy nobody chose. If you want includeSubDomains or preload, own the header at the proxy and strip Hikyo’s; Hikyo does not send those directives because they commit hostnames it does not serve. See security response headers for the full carried set.

Check both probes through the private listener:

curl --fail http://127.0.0.1:8081/healthz
curl --fail http://127.0.0.1:8081/readyz

Readiness gives the complete datastore and optional HA check a two-second budget. An unavailable or stalled datastore returns HTTP 503; /healthz remains independent so a database outage removes traffic without triggering a process restart.

The same private listener serves /metrics in Prometheus text format (RED-style request counters and latency per surface class, in-flight and admission-pressure gauges, plus the retention and TLS gauges) with no external service dependency. It is unauthenticated but accepts only actual loopback peers; non-loopback scrapes return HTTP 403 even with a forwarded loopback address. Scrape from the host, or from a collector sharing the pod’s network namespace. Forward metrics through that collector’s separately authenticated transport if needed, and keep the operational bind private. See metrics for the full list.

Then check the public HTTPS origin from outside the host. A local probe does not prove DNS, TLS, or reverse-proxy routing.

PostgreSQL boundaryCopy Anchor Link

Remote PostgreSQL DSNs must name the host explicitly and use sslmode=verify-full for certificate and hostname verification. CA-only sslmode=verify-ca and plaintext connections to a non-loopback database host are refused.

Use a dedicated database and role. Back up the Hikyo archive through hikyo backup export; a generic PostgreSQL dump does not replace Hikyo’s encrypted export and restore reconciliation workflow.

Root-key custodyCopy Anchor Link

The root key is exactly 32 random bytes encoded as 64 hexadecimal characters. Hikyo refuses a file readable by group or other.

Prefer a service-manager credential mounted for the process. A fixed protected file is the next tier. HIKYO_ROOT_KEY is the weakest tier because the value remains in the process environment.

Before real secret materialCopy Anchor Link

1. Terminate TLS natively or name exact proxy CIDRs; keep the operational listener private. Set HIKYO_EXTERNAL_ORIGIN to the public https origin and do not double-set Strict-Transport-Security at the proxy.
2. Store the root key through your service manager’s credential mechanism.
3. Configure encrypted backups and perform a restore test.
4. Create the first administrator on the server host.
5. Review the security and support policies.

Continue with server configuration and backup and restore.

Operational checklistCopy Anchor Link

Run hikyo doctor --instance NAME -o json using an authenticated human operator session. The CLI and operator-only web banners include the existing backup, restore, retention, project storage and adapter checks, plus:

• Datastore volume: warning at 80% used and critical at 90%, based on space available on the actual datastore filesystem. SQLite measures locally. PostgreSQL reports unknown unless Kubernetes storage monitoring is configured as described below; otherwise configure alerts on the database host.
• Current root escrow verification, bound to the finalized root epoch and recovery incarnation. Follow the backup runbook to verify a separately recovered root copy. Root rotation and restore invalidate the previous claim.
• Expired pins and disjoint 1-day, 7-day and 30-day expiry tiers. Expiry removes retention protection; delivery continues only while the pinned payload exists.
• Dual root wrappers, key scopes with retiring versions and the timestamp of the last successfully completed re-encryption scope.
• The actual boot-validated database durability policy and Argon2id parameters.

Unknown findings remain explicit and produce a warning result. Critical findings produce doctor exit code 4. Diagnostics do not change admission or readiness. The existing public operational metrics expose bounded aggregate counts without identities or labels. hikyo_data_volume_capacity_known must be checked before using volume gauges; an unknown percentage is NaN, not zero. The same rule applies to hikyo_dynamic_gauges_known and hikyo_approval_gauges_known: when either is 0 the scrape could not measure the dynamic-secret lease gauges or the change-approval gauges, and those gauges are omitted from the scrape rather than reported as zero. Escrow timestamp and verification gauges expose public metadata only.

PostgreSQL capacity on KubernetesCopy Anchor Link

When PostgreSQL runs on Kubernetes, the Helm deployment can connect Hikyo to the database node’s kubelet statistics. No Prometheus installation or additional collector is required when the kubelet reports statistics for the database PVC. Enable the reader with an explicit mapping:

database:
  storageMonitoring:
    enabled: true
    kubeletURL: https://192.0.2.10:10250
    node: database-node
    namespace: database # Omit to use the Helm release namespace.
    pvc: data-postgres-0

Replace these example values with the node hosting PostgreSQL and its data PVC. The kubelet HTTPS serving certificate must validate against the cluster’s kube-root-ca.crt, including its hostname or IP address. The Hikyo pod must be able to reach that endpoint; the kubelet must support service-account token authentication and webhook authorization. There is no insecure TLS option.

The chart grants the server ServiceAccount only get on nodes/stats, restricted to the configured node, and projects a rotating token into the server container. It does not grant nodes/proxy, pod execution, or Secret access. Kubernetes authorizes statistics for the whole node; Hikyo selects only the exact configured node, namespace and PVC from that response.

For deployments outside Helm, the equivalent bootstrap variables are HIKYO_PG_STORAGE_KUBELET_URL, HIKYO_PG_STORAGE_NODE, HIKYO_PG_STORAGE_NAMESPACE, and HIKYO_PG_STORAGE_PVC. Supply all four, plus the cluster CA and service-account token at the standard Kubernetes mount path. These settings require PostgreSQL.

Incomplete or invalid settings and missing or invalid local credential files refuse startup. Once the reader is initialized, remote measurement failures are advisory and do not prevent serving requests.

Samples older than two minutes, more than five seconds in the future, missing capacity fields, conflicting samples, TLS/authentication failures and unreachable kubelets produce Unmeasured. Requests time out after three seconds. These diagnostics remain advisory and do not change readiness or database admission.

The reading describes the backing filesystem, which may be shared with other workloads. In particular, k3s local-path PVC requests do not establish a dedicated filesystem quota. Do not interpret a 10 GiB claim as 10 GiB of actual disk capacity. Update the mapping when moving PostgreSQL to another node, namespace or PVC, including when changing the configured database endpoint. This mapping is an operator assertion; Hikyo cannot infer the PostgreSQL-to-PVC relationship from the database DSN. For external databases or unsupported storage drivers, continue monitoring capacity at the database host.