Upgrades

Hikyo migrations move forward only. A downgrade means restoring a pre-upgrade archive with a compatible binary, not running a down migration.

Social sign-in upgrade notesCopy Anchor Link

When upgrading across migration 00057_social_signin, existing sessions survive the SQLite session-table rebuild and the equivalent PostgreSQL schema change. Every open disclosure reauthentication window closes on both engines. Users must complete fresh reauthentication before their next protected disclosure. In-flight OIDC transactions are cleared by migrations 57 and 58; restart an interrupted provider ceremony after the upgrade.

The earlier migration 00044_retire_oidc_provisioning removed the legacy OIDC JIT policy. Existing provisioned accounts and explicit identity links remain. Migration 00058_registration_switch does not fold that retired policy into registration, reread its old column or remove its audit history again. An operator must explicitly create and enable a registration policy to admit unknown identities. Existing linked identities continue through ordinary sign-in. Review these changes when your upgrade route crosses those migrations, even if a newer binary also contains later schema changes.

For cloud VMs, use the deployment-specific upgrade guide. For a fresh Docker nightly and its image replacement commands, use cloud nightlies.

One-command nightly upgradeCopy Anchor Link

For the explicit first-deployment Docker/Kubernetes opt-in, see unattended container upgrades. Its release status and custody requirements are separate from the systemd procedure below.

On a Linux systemd server with a local SQLite database, run:

sudo hikyo upgrade

The command verifies the signed nightly and its complete migration route, assembles release evidence, stops writers, exports an encrypted backup, restores it to scratch, proves key and credential recovery, and upgrades the service. Intermediate releases stay in maintenance. The command reports success only after the exact final executable and database state pass health checks.

The first run creates /etc/hikyo/upgrade.json for the standard hikyo.service installation and the encrypted operator keys, wrapped with a secret derived from the root key file, so no prompt is needed then or later and the command can run unattended. Custom deployments can supply a root-owned config with --config FILE; unsupported unit settings fail before the service stops. This command currently supports nightly releases on a single Linux systemd host with SQLite. Compose, Kubernetes, PostgreSQL and stable deployment upgrades use the manual procedure below.

Older nightlies, including the pre-ledger releases, do not contain this command. After a signed release containing it is published, bootstrap the new coordinator without replacing the working server first. The bootstrap needs curl, python3 and openssl on the server:

curl --fail --silent --show-error --proto '=https' https://hikyo.app/upgrade-nightly.sh | sudo sh

The bootstrap verifies the release before running the staged coordinator. Future upgrades use sudo hikyo upgrade. The first legacy transition still requires an exact recovery-signed bridge; the command finds and verifies it automatically.

Encrypted operator custody lives in /etc/hikyo/upgrade-keys/operator.age, owned by root and inaccessible to the runtime user. It includes the backup identity, independent attestation key and root escrow, wrapped with a secret derived from /etc/hikyo/root.key. Keys created by earlier releases under a passphrase are rewrapped the next time sudo hikyo upgrade runs from a terminal; it asks for the passphrase once more and never again. This explicitly chosen local custody model does not provide an off-host recovery copy. Preserve the encrypted vault together with the root key through your recovery process. After each completed upgrade the command keeps only the encrypted backup taken before the current schema under /var/lib/hikyo-upgrade; earlier backup-*, bundle-* and evidence-* directories there are removed. Copy any backup you want to keep elsewhere. No operator private key enters the service configuration or release workflow.

If an upgrade is interrupted, rerun the same command; if the installed binary still lacks hikyo upgrade, rerun the bootstrap command above. Its durable journal and systemd startup fence survive reboot. Proven pre-write attempts and known healthy or schema-applied targets can continue. An uncertain or failed schema change leaves the service stopped and requires explicit operator recovery; the command never automatically rolls back the database or starts an older binary.

Upgrade the CLI on macOS or WindowsCopy Anchor Link

macOS and Windows have no supported server host, so hikyo upgrade there upgrades the CLI itself. Run it without sudo:

hikyo upgrade

It checks this binary’s update channel, verifies the newest release exactly as the server command does, and atomically replaces the running executable. It refuses when run as root over an executable you own, and source builds (update channel off) must be rebuilt instead.

Progress and disk useCopy Anchor Link

Use repeatable global verbosity options before the command or its other options:

sudo hikyo -v upgrade       # phases, selected releases, route and cleanup
sudo hikyo -vv upgrade      # also artifact sizes, cache removals and HTTP status
sudo hikyo -vvv upgrade     # also operation and download timings

--verbose --verbose is equivalent to -vv. Verbosity follows the verified coordinator handoff. Diagnostic lines go to stderr, keeping stdout usable for command results. Request headers, bodies, URL credentials and query strings are not printed. Arguments after -- belong to the child command and remain untouched.

The directories have distinct ownership and lifetimes:

The similarly named hikyo-upgrade and hikyo-upgrader separate service-readable verification evidence from privileged coordinator state. They are not duplicate application data directories.

The coordinator prunes obsolete downloads before fetching a release, and again on completion, including an already-installed result. It retains the latest observed release for handoff and retry, plus an explicitly pinned target when needed. Installed executables and the service’s public verification bundle are independent copies. Superseded private route bundles are discarded during preparation; cache hits reverify existing bytes without writing a second full release copy. Anti-rollback trust state is never pruned.

After an interruption, exact downloaded releases and candidate executables stay available for the unfinished route. Disposable private bundles and abandoned assembly files are removed and rebuilt as needed. The journal, public recovery evidence, retained encrypted backup and operator keys remain protected. Cleanup errors are reported; cleanup after a completed upgrade does not stop the healthy service. A later invocation retries cleanup.

Full signed inventories still include artifacts for every published platform. A long route therefore needs temporary space for its releases, assembled bundle, candidate executables and backup drill. -v reports available space after initial cleanup; -vv shows individual artifact sizes. Do not delete these directories manually during an unfinished upgrade.

Release channelsCopy Anchor Link

stable follows signed production releases. nightly follows signed GitHub prereleases built from the newest green, changed main commit. A stable 1.0.0 makes the nightly line 1.0.0-nightly.<date>.<run>.g<sha>; after stable 1.1.0, it advances to 1.1.0-nightly.... Nightlies use the distinct recovery-authorized nightly/v1 keyless signing policy. They remain prereleases.

Nightlies may contain migrations added after their stable base. Update ordering therefore treats a nightly as newer than that base, unlike ordinary SemVer prerelease ordering. 1.1.0-nightly... cannot update to stable 1.1.0; only 1.1.1 or higher is eligible. Server transitions still require an authenticated forward migration route, a recovery-authorized bridge and backup/restore proof. Older nightlies published with next-minor names keep their original identities; the same conservative base-version rule applies to them.

Before any stable release exists, the first line is 0.0.1-nightly.<date>.<run>.g<sha>. Release discovery is paginated; a long nightly history cannot hide the latest published stable release.

Server-side and remote workspace checks are notification-only. Remote apply is disabled for every release channel because the legacy helper cannot prove safe rollback once a migration has started. The local operator command above and verified manual downloads remain available.

Release binaries also check their own CLI track before dispatching a command when stderr and a controlling terminal are available. Stable artifacts default to stable, nightly artifacts default to nightly, and binaries built directly from source default to off. A local hikyo update channel ... choice is persisted and overrides a published artifact’s default. Source builds remain off; install a published artifact to join a release channel.

At most once per 24 hours, Hikyo refreshes the selected track. When a newer release exists, the controlling terminal offers the appropriate download flow. Nightlies require an immutable release, authenticated policy, complete Sigstore bundle and the exact closed inventory of all published payloads. On Linux, accepting stages the verified download and assembled runtime bundle and prints their paths; it preserves the current executable, which may be a server’s. Use sudo hikyo upgrade to upgrade a supported server, including its installation-specific backup and restore proof. On macOS and Windows, accepting replaces the CLI executable after the same verification. Unsigned historical nightlies cannot pass this verification.

Stable updates download the exact archive for the running OS and architecture, require an immutable release and use Hikyo’s in-process, recovery-root-pinned signature verifier over the current trust metadata, signed manifest, candidate, and selected archive. No external shell, jq, or cosign executable participates in the self-update trust boundary.

After stable verification Hikyo extracts only the expected executable, takes a cross-process replacement lock, and writes and syncs a unique same-directory file. Unix atomically renames it over the resolved binary. Windows recoverably moves the mapped old image aside before publishing the replacement at the original path, then removes the backup on a later start after its last process exits. The current command stops before dispatch. Repeat the command to run the new version. An already running hikyo server process keeps its old in-memory image until its service is restarted.

The updater never elevates privileges. If the current binary is managed by a system package or is not writable by the current user, the replacement fails without changing it; use that installation’s package or service procedure. hikyo update check bypasses the 24-hour refresh interval and performs the same confirmation flow immediately.

Remote apply is disabledCopy Anchor Link

The legacy hikyo updater helper and its Compose, systemd, and Flux adapters are retired. They could automatically restore a database after schema writes had started; the Flux adapter also used ambient Git credentials. Remote platform application remains disabled. The operator-invoked systemd command and manual installation use the same mandatory signed-release gate as server startup.

HIKYO_UPDATER_SOCKET now prevents server startup. Remove that setting to start the server with release notifications. hikyo updater refuses before reading configuration, touching its socket or journal, or starting commands. The shipped platform adapters refuse every phase, including rollback. The API reports apply_supported: false with a migration-safety reason; authenticated apply requests are refused and never submitted to a helper. The WebUI also refuses legacy apply advertised by an older remote instance.

Before installing this release, stop and disable every existing updater helper service and check for already running backend commands. Replacing a binary on disk cannot stop an old process or a previously spawned privileged child. Preserve jobs.json and deployment logs as historical evidence; do not replay queued jobs or infer a safe rollback from a legacy journal’s phase. If an earlier attempt reached apply, investigate the datastore and use the operator-led recovery procedure below. Retirement does not mark existing jobs successful or restored.

Every connected workspace still checks release metadata directly. Administrators retain update notifications and links to release notes. An operator runs the local command above or the manual procedure below. The viewing instance holds no fleet deployment credentials.

For stable artifacts, use the installation and release-verification instructions to obtain and verify the signed bundle before the maintenance window.

Manual upgrade inputsCopy Anchor Link

Every production boot and hikyo migrate needs HIKYO_UPGRADE_BUNDLE, naming a complete authenticated offline release bundle, and the installation’s public operator pin and persistent custody state. A binary, image digest, ordinary release download directory or successful backup upload alone is insufficient.

The complete executable manual deployment runbook covers bundle layout, native commands, Compose and Helm PVC wiring. The release artifact ceremony produces the compatibility declaration and recovery-signed trust catalog. Official production bundle publication remains a release-owner obligation; unsigned evaluation artifacts cannot replace it.

For nightlies, follow the signed nightly runbook. It covers the one-time recovery authorization, automated publication and complete-download assembly. Existing unsigned pre-ledger databases additionally require an exact recovery-signed legacy bridge and fresh operator backup proof. The notification/download flow does not replace the server binary. The explicit sudo hikyo upgrade command handles the bridge and deployment on supported hosts.

Before the changeCopy Anchor Link

1. Verify all release manifests and exact binary/image artifacts for every route hop.
2. Stop all writers, including every pre-gate process on the first legacy hop.
3. Run hikyo backup upgrade-export to produce ciphertext and a bound public receipt.
4. Run hikyo backup upgrade-drill on the operator custody host against an empty scratch target.
5. Stage only its public receipt, signed attestation and encrypted archive for the target gate.

The private operator signing key, age identity and separate root escrow stay on the drill host. Missing recipients, failed decryption, wrong root custody, unsuccessful reconciliation or stale evidence cannot authorize a populated upgrade. A receipt and attestation bind the exact instance, source, epoch, recovery incarnation, route, target, generation and ciphertext bytes. Attestations last at most 24 hours and their nonces are consumed atomically.

Controlled migrationCopy Anchor Link

Configure the public inputs and retain durable installation state outside DB backups. After every writer has stopped, run one verified target migrator:

export HIKYO_UPGRADE_BUNDLE=/srv/hikyo-public/bundle
export HIKYO_UPGRADE_OPERATOR_PUBLIC_KEY=/srv/hikyo-public/operator.pub
export HIKYO_UPGRADE_STATE_DIR=/var/lib/hikyo-installation
export HIKYO_UPGRADE_EVIDENCE=/srv/hikyo-public/evidence
export HIKYO_UPGRADE_BACKUP=/srv/hikyo-public/backup.age
hikyo migrate
hikyo server --auto-migrate=false --root-key-file=/run/hikyo-root/root-key

For the first pre-gate hop, set HIKYO_UPGRADE_LEGACY_WRITERS_STOPPED=true only after verifying every old process exited. This assertion does not retroactively fence legacy binaries. HA uses full stop by default; ordinary rolling replacement is not a migration procedure.

migrate completes DDL at schema-applied and retains maintenance. Only the same verified target boot can complete candidate configuration/key checks and admit serving. A failed post-write health check requires explicit recovery. For multiple hops keep HIKYO_UPGRADE_TARGET_MANIFEST set to the final verified manifest SHA-256, execute each required binary in order, and retain maintenance between hops.

A genuinely fresh empty installation omits evidence and backup paths. Ordinary healthy same-release restarts need no new receipt; remove the evidence, backup and legacy-stop environment settings after the final healthy boot. Startup with automatic migrations enabled still uses this gate; there is no missing-backup skip for populated upgrades.

Verify the upgradeCopy Anchor Link

curl --fail http://127.0.0.1:8081/healthz
curl --fail http://127.0.0.1:8081/readyz
hikyo doctor --instance hikyo.example.com
hikyo whoami --instance hikyo.example.com

Also verify one authorized metadata read and one representative workload. A health response alone does not prove authentication, authorization, or delivery.

Roll backCopy Anchor Link

Stop the failed instance. Restore the pre-upgrade archive into an empty target with a binary that supports its schema, then follow the full reconciliation procedure.

Restoring invalidates all old authenticators. Include human recovery and machine credential redistribution in the rollback time estimate.

See backup and restore for the complete flow.

Privacy-control upgrade precautionsCopy Anchor Link

Versions introducing bounded audit retention activate 90-day access and 365-day security defaults, including for existing records. Choose justified retention configuration and preserve independently required evidence before starting the upgraded server. Keep current privacy action receipts separately from backups and use the restore reconciliation procedure before reopening a restored installation.

Social sign-in schema upgradeCopy Anchor Link

The version that adds the social sign-in schema (migration 00057) closes every open reauthentication window while it reshapes the sessions table. Sessions survive, so nobody is signed out, but a user who had just reauthenticated for a protected environment reauthenticates again. OIDC sign-ins in progress during the migration are cancelled and simply restart. The contact email an account had keeps its value, in canonical form (domain lowercased), when it is a valid plain address, and is cleared otherwise; kept addresses are marked unverified and are never used to sign in. Nothing else changes until registration is enabled.