Kubernetes operator

An admin-created, cluster-scoped, immutable HikyoInstance pins a credential-free HTTPS endpoint and trust configuration. A namespaced HikyoSecret selects that instance, its own credential and a Hikyo scope, then the operator creates and owns a native Kubernetes Secret. Workloads consume the Secret through envFrom, valueFrom, or a volume. Environment variables change only after a pod restart. Normal Secret volumes update eventually through the kubelet; subPath mounts never update.

To manage Hikyo’s own bootstrap sources, use the separate configuration rollout executor. It uses an administrator-authorized control path and does not grant the HikyoSecret operator access to the protected instance project.

K3s: encrypt native Secrets at rest

Synced values land in native Kubernetes Secrets. K3s stores them in plaintext at rest by default. Enable K3s --secrets-encryption. Select secretbox only on K3s ≥ v1.30.12+k3s1, v1.31.8+k3s1, v1.32.4+k3s1, or v1.33.0+k3s1. Use the AES-CBC provider on older releases.

Install the operatorCopy Anchor Link

Generate the root key and create both required server Secrets first:

openssl rand -hex 32 > root-key
kubectl create namespace hikyo
kubectl --namespace hikyo create secret generic hikyo-root-key \
  --from-file=root-key=./root-key
kubectl --namespace hikyo create secret generic hikyo-database \
  --from-literal=HIKYO_DB='postgres://hikyo:password@postgres.example.com/hikyo?sslmode=verify-full'
kubectl --namespace hikyo create secret tls hikyo-server-tls \
  --cert=./tls.crt \
  --key=./tls.key

Protect and back up root-key separately from the database. The chart supports PostgreSQL only: its container root filesystem is read-only. A private database CA can be mounted with database.tls.existingSecret; name its mounted path in the DSN as sslrootcert=/run/hikyo-database-ca/ca.crt.

For the default manual installation, provision two persistent claims first: one holding the verified public bundle and independently pinned operator.pub, and one holding installation custody. Follow the Kubernetes installation runbook for their directory layout, ownership and permissions. An empty claim alone does not provide the release evidence needed to boot. Keep installation state independent of database restores.

Create a values file for the Hikyo server and operator, replacing the claim names with those provisioned above:

database:
  existingSecret: hikyo-database
  tls:
    existingSecret: "" # Set for a private PostgreSQL CA.
    key: ca.crt

rootKey:
  existingSecret: hikyo-root-key
  key: root-key

upgrade:
  existingClaim: hikyo-upgrade-public
  stateExistingClaim: hikyo-upgrade-state

externalOrigin: https://hikyo.example.com

tls:
  existingSecret: hikyo-server-tls

network:
  trustedProxyCIDRs: []
  allowPlaintextOrigin: false

operator:
  enabled: true
  namespaces:
    - payments
  triggerRollouts: true
  designatedServiceAccounts:
    payments:
      - payments-api
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 128Mi

Install a packaged release chart. The package already pins its matching server image digest:

helm upgrade --install hikyo oci://ghcr.io/hikyo-org/charts/hikyo \
  --version <release-version> \
  --namespace hikyo \
  --values hikyo-values.yaml

When validating a source checkout instead, use ./chart/hikyo and pass the exact candidate image with --set image.repository=... and --set image.digest=sha256:<64-hex-digest>.

An empty operator.namespaces watches cluster-wide. A non-empty list creates namespace-scoped bindings and the watch list from the same value. Effective reach is the intersection of those watches and RBAC. Add every federated ServiceAccount to operator.designatedServiceAccounts under its namespace; the chart restricts TokenRequest creation to those names.

Helm refuses missing datastore/root-key Secret references, upgrade claims, external origin, or network boundary. For an empty first deployment, the explicit unattended opt-in replaces the manual public claim with verified local custody; it still requires persistent installation state and a separate PostgreSQL scratch database. externalOrigin must be an origin only: no path, query, fragment, credentials, or trailing slash. HTTP requires the explicit network.allowPlaintextOrigin development escape hatch.

The chart ships no Ingress, and an https externalOrigin makes the app emit Strict-Transport-Security itself, behind a TLS-terminating Ingress as much as with a native TLS Secret. Configure whatever fronts the Service to pass the app’s response headers through; an Ingress controller that adds its own HSTS sends browsers two policies with different lifetimes. See security response headers.

The operator runs as UID/GID 65532 with runAsNonRoot, a read-only root filesystem, all capabilities dropped, RuntimeDefault seccomp, and privilege escalation disabled. Its pod holds no Hikyo credential, Hikyo root key, or datastore. Each HikyoSecret supplies its own identity.

Rollout triggering costs get, list, watch, and patch on Deployments, StatefulSets, and DaemonSets in every bound namespace. Kubernetes RBAC cannot limit those patch rights by label or annotation. “Only opted-in workloads” is an operator-code rule, not an RBAC boundary; the namespace binding is the only real boundary. Set operator.triggerRollouts: false if you will not grant workload patch access. Secret delivery still works, but you own restarts.

operator.statusReporting (default true) reports each HikyoSecret’s conditions to its Hikyo server after every status write, and at least once per max(resyncInterval, 5m), capped at 24 hours. A report carries only the target’s identity, generation, lifecycle and each condition’s type, status and reason: never values, key names, mappings, cursors or condition messages. It uses the HikyoSecret’s own credential, which needs the report-delivery-status grant on the environment; without it the server refuses and the operator pauses reporting for that HikyoSecret for an hour. The operator reports only to a server that advertises the capability, and it reads the kube-system Namespace UID as the cluster id, so the chart grants get on that one Namespace. Set operator.statusReporting: false to send nothing and drop that grant.

Create a HikyoInstanceCopy Anchor Link

Only cluster admins should create instances. The complete spec is immutable. url must use HTTPS, caBundle is an optional base64-encoded PEM bundle, and audience is required when any HikyoSecret uses ServiceAccount federation.

apiVersion: hikyo.dev/v1alpha1
kind: HikyoInstance
metadata:
  name: primary
spec:
  url: https://hikyo.example.com
  caBundle: <base64-encoded-PEM-bundle>
  audience: https://hikyo.example.com/kubernetes/primary

There is no credential field and no insecure TLS option. Create a new HikyoInstance to change its endpoint, CA bundle, or audience.

Credentials and designationCopy Anchor Link

Bootstrap SecretCopy Anchor Link

Create a workload service account in Hikyo, grant its required scope, then mint the bearer credential with the shipped CLI verb:

hikyo sa create --name payments-api --kind workload
hikyo access grant add \
  --principal <service-account-id> \
  --capability <read-capability>
hikyo sa credential mint \
  --sa <service-account-id> \
  --lifetime 720h \
  --output-file ./hikyo-token

kubectl --namespace payments create secret generic hikyo-payments-credential \
  --from-file=hikyo-token=./hikyo-token
kubectl --namespace payments label secret hikyo-payments-credential \
  hikyo.dev/delivery="true" \
  hikyo.dev/instance=primary

The bootstrap Secret must be in the same namespace as the HikyoSecret. Its data key is hikyo-token and both designation labels are required.

The same three Hikyo-side steps are available in the browser: Machine access → Create service account, Add environment grant, and Mint credential (shown exactly once). See Operate Hikyo from the browser.

ServiceAccount federationCopy Anchor Link

Federation avoids a stored bearer credential. Designate the same-namespace ServiceAccount for one instance:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  labels:
    hikyo.dev/delivery: "true"
    hikyo.dev/instance: primary

List that name under operator.designatedServiceAccounts.payments. The operator requests a short-lived token whose audience exactly equals HikyoInstance.spec.audience; it keeps that token in memory only. Configure a trusted issuer in Hikyo from the cluster’s OIDC issuer and JWKS, then create an immutable byte-exact issuer/subject/audience binding. See machine identities for the issuer and binding workflow.

Naming either credential in a HikyoSecret is not authority to use it. The referenced Secret or ServiceAccount must carry both designation labels, and its hikyo.dev/instance value must match instanceRef.name.

Create a HikyoSecretCopy Anchor Link

This example uses the bootstrap Secret above with explicit key mappings:

apiVersion: hikyo.dev/v1alpha1
kind: HikyoSecret
metadata:
  name: payments-config
  namespace: payments
spec:
  instanceRef:
    name: primary
  auth:
    secretRef:
      name: hikyo-payments-credential
  scope:
    org: org_01900000-0000-7000-8000-000000000001
    project: proj_01900000-0000-7000-8000-000000000002
    environment: env_01900000-0000-7000-8000-000000000003
  mapping:
    - key: DATABASE_URL
      secretKey: DATABASE_URL
    - key: LOG_LEVEL
      secretKey: LOG_LEVEL
  target:
    name: payments-env
    creationPolicy: Owner
  projection: full
  acknowledgedLoaderKeys: []
  resyncInterval: 5m

Use auth.serviceAccountRef.name instead of auth.secretRef.name for federation. projection: config-only requests a server-authorized manifest that omits secret keys; it is not a client-side filter. creationPolicy is Owner or Orphan.

K3s: encrypt the managed Secret at rest

This sync writes payments-env as a native Kubernetes Secret. K3s stores it in plaintext at rest by default. Enable K3s --secrets-encryption. Select secretbox only on K3s ≥ v1.30.12+k3s1, v1.31.8+k3s1, v1.32.4+k3s1, or v1.33.0+k3s1. Use the AES-CBC provider on older releases.

Deliver all authorized keysCopy Anchor Link

Upgrade both the operator and its CRD before using spec.allKeys. To deliver new settings without adding a mapping for each one, replace the complete mapping field in the example above with:

spec:
  allKeys: true
  target:
    name: payments-env
    type: Opaque
    creationPolicy: Owner

This is a fragment: keep the instance, identity, scope, projection and resync fields from the full example. Choose either allKeys: true or a nonempty mapping, never both. All-keys delivery supports only Opaque targets and keeps each source key’s name. It does not grant additional access: the same identity, projection and reveal permissions govern every value. A presence-only secret refuses the whole delivery. Published additions and removals update the complete managed Secret, including removal of the last key.

Consume that Secret without repeating its keys in a Deployment:

metadata:
  annotations:
    hikyo.dev/secrets: payments-env
spec:
  template:
    spec:
      containers:
        - name: payments
          image: example.invalid/payments:1.0.0
          envFrom:
            - secretRef:
                name: payments-env
          env:
            - name: LOG_LEVEL
              value: info

The explicit LOG_LEVEL entry overrides a same-named Secret key. Other delivered keys become process environment variables. The application still validates its required settings and decides what recognized settings do. Kubernetes checks that the referenced Secret exists, but envFrom does not enforce individual required keys. With rollout triggering enabled, the workload annotation requests a restart when delivered content changes; unchanged content does not cause a restart.

Protected loader-control names, including PATH, NODE_OPTIONS, LD_* and GIT_*, remain blocked unless each exact name appears in acknowledgedLoaderKeys. In all-keys mode an acknowledgement may name a protected key that is currently absent, so later additions and removals need no manifest edit. Wildcards, duplicates and unprotected names are invalid acknowledgements. A newly published unacknowledged protected key refuses the whole replacement, retaining the previous Secret and clearing the cursor for a fresh fetch. Only acknowledge names the workload intentionally trusts.

K3s: encrypt the managed Secret at rest

All-keys delivery writes a native Kubernetes Secret. Enable K3s --secrets-encryption; the datastore is plaintext at rest by default. Select secretbox only on K3s ≥ v1.30.12+k3s1, v1.31.8+k3s1, v1.32.4+k3s1, or v1.33.0+k3s1. Use the AES-CBC provider on older releases.

The five-step journeyCopy Anchor Link

1. Mint a workload service-account credential or configure federation.
2. Grant that principal read on the target environment.
3. Create the HikyoSecret. A fresh workload principal receives config values and secret presence only, so an all-or-nothing sync refuses with Delivery=False, reason UndeliveredSecrets, naming the secret keys.
4. A Hikyo operator enables the per-project machine-reveal opt-in: hikyo project-settings machine-reveal set --enabled true (a project-settings and reveal act at project scope; second factor required). The Machine access page offers the same toggle.
5. Grant reveal to the workload principal on the environment: hikyo access grant add --principal <mch_...> --capability reveal --env <env-id>, or Members → New grant in the browser with the service account’s principal id. The grant runs the widening ceremony over the environments it reaches.

Until steps 4–5 are done, a workload principal delivers config values and secret presence only, and a HikyoSecret selecting any presence-only secret, through a mapping or allKeys, is refused with Delivery=False/UndeliveredSecrets. Withdrawing the opt-in later returns every workload in the project to that state on its next fetch, without touching the grants.

Rollout triggeringCopy Anchor Link

Consent lives on each workload. Add the managed Secret name to the workload’s metadata annotation:

metadata:
  annotations:
    hikyo.dev/secrets: "payments-env,shared-env"

After a changed delivery, the operator patches that workload’s pod template:

spec:
  template:
    metadata:
      annotations:
        stamp.hikyo.dev/payments-env: v1:<32-hex-characters>

The patch requests a rollout; it does not force one. A paused Deployment, an OnDelete StatefulSet, or stalled readiness can leave Rollout=False/Stalled. Each stamp covers one target, so the restart blast radius follows Secret consumption. It is a keyed, per-target stamp, never a bare content digest or the server change token.

Reloader is prior art, not part of this contract. Its default strategy injects a resource hash through an environment variable, exposing a bare digest of secret material in a workload-readable field. Its non-default annotations strategy avoids that shape but still needs the same workload patch authority.

Native Secret typesCopy Anchor Link

spec.target.type defaults to Opaque. Set a native type when a Kubernetes consumer requires one. First opt in with Helm --set operator.nativeSecretTypes=true. This enables the runtime capability and adds Secret delete within the configured namespace authority. Opaque-only installs omit that permission. Standalone operators set HIKYO_OPERATOR_NATIVE_SECRET_TYPES=true with matching delete RBAC. Disabling the capability refuses typed targets before fetching and retains existing targets; migrate or remove them explicitly before disabling it.

Map Hikyo’s uppercase source key to the native data key:

spec:
  target:
    name: registry-credentials
    type: kubernetes.io/dockerconfigjson
  mapping:
    - key: REGISTRY_DOCKERCONFIGJSON
      secretKey: .dockerconfigjson

This is a fragment of the HikyoSecret above; keep its instance, identity and scope fields. The mapped value is the JSON text itself, not base64 text. For K3s, enable --secrets-encryption before syncing credentials: native Secrets are stored in the cluster datastore, which is plaintext at rest by default. Reference registry-credentials in the consuming pod’s imagePullSecrets.

For an Ingress, map TLS_CERTIFICATE to tls.crt and TLS_PRIVATE_KEY to tls.key, set type: kubernetes.io/tls, then reference the target name from spec.tls[].secretName. Neither image-pull nor TLS data keys are environment variable names; consume them through the native reference, not envFrom.

Unknown types are rejected at admission. Missing required destination mappings report Delivery=False/KeysMissing before fetching. Invalid content reports Delivery=False/InvalidSecretData; both retain any existing target. No partial Secret or delivery cursor is written. Loader-control acknowledgement still applies to every mapped destination.

Changing the type of an existing target reports Conflict=True/TargetTypeImmutable. The operator never deletes and recreates a Secret to change its type. Create a new HikyoSecret with a new target name and migrate its consumers explicitly.

Authorization withdrawal differs from a type change. An authenticated 404, or a successful manifest that removes a mandatory mapped key, deletes an owned typed target because Kubernetes cannot represent the withdrawn state as an empty Secret of that type. This includes an accidental published unset: the operator cannot distinguish it from intentional withdrawal and must not continue serving a removed credential. In contrast, malformed content for a still-present key rejects its replacement and retains the last accepted target. TLS content is not parsed by Kubernetes or Hikyo; malformed certificate/key bytes are delivered and the consumer decides whether to keep its previous certificate. Publish coupled credential changes atomically and monitor delivery conditions.

Deletion uses UID and resource-version preconditions, followed by an uncached absence check. A replacement target is never deleted; pending deletion is reported as failure. The cursor is cleared and opted-in workloads receive the empty-content stamp. Repeated refusals keep the target absent; a later complete authorized delivery recreates it. creationPolicy: Orphan controls CR deletion only and does not suppress authorization withdrawal. With operator.nativeSecretTypes=true, Secret RBAC includes delete in bound namespaces, without granting list or watch. Pods may retain values already loaded into memory; removal from Kubernetes does not erase a consumer’s cache.

Lifecycle and conditionsCopy Anchor Link

Ready=True is the summary: Synced=True with no active refusal condition. The operator uses this closed reason set:

PinExpired emits a Warning event and does not make a successful delivery unready. Every successful response, including a conditional current response that leaves the Secret unchanged, refreshes the warning. A successful response with a fresh pin or no pin clears it. Fetch failures retain the last known pin condition.

The managed Secret has three distinct lifecycle outcomes:

1. Deleting the CR with creationPolicy: Owner lets Kubernetes garbage-collect the Secret. Orphan removes the owner reference during deletion and leaves the Secret standing unowned.
2. An unreachable server, 5xx, 429, or 401 retains the last-synced Secret and reports Synced=False/FetchFailed. Retry backoff is jittered from 1 second to 5 minutes. There is no staleness bound and no timeout scrub.
3. An authoritative refusal under a still-working credential converges the Secret to current authority. Lost environment access scrubs it; mapped keys missing from an authenticated manifest are removed.

Operational rule: revoke-the-token-and-done leaves delivered values standing. If the cluster must converge, remove grants while the credential still authenticates, or replace it and let the replacement identity reconcile. A scrub cannot remove values already copied into a running process environment; restart those pods.

Loader-control keysCopy Anchor Link

The baseline is LD_*, PATH, IFS, ENV, BASH_ENV, SHELLOPTS, NODE_OPTIONS, PYTHONSTARTUP, PYTHONPATH, PERL5OPT, PERL5LIB, RUBYOPT, RUBYLIB, JAVA_TOOL_OPTIONS, _JAVA_OPTIONS, JDK_JAVA_OPTIONS, CLASSPATH, GIT_*, SSL_CERT_FILE, SSL_CERT_DIR, CURL_CA_BUNDLE, REQUESTS_CA_BUNDLE, and NODE_EXTRA_CA_CERTS.

If mapped destination keys match that baseline, acknowledgedLoaderKeys must equal exactly the matching mapped set. Missing names and extra acknowledgements both refuse delivery. When the list is non-empty, the operator sends it as acknowledged_keys. When it is empty, the operator omits the parameter; the server records an empty list on every fetch.

This check is consumption-agnostic because the operator writes a Secret before it can know whether a later workload will use envFrom or a file mount. Inspecting current workloads would race with later changes. File-only consumers therefore still acknowledge a mapped key named PATH. Separately, Kubernetes envFrom silently skips Secret keys that are not valid environment names; Hikyo reports Delivery=True/EnvFromSkip as a warning while sync remains successful.

Beside the operatorCopy Anchor Link

For CI, the shipped export command is hikyo values export --reveal --dangerously-print. It writes revealed JSON to stdout; it does not render a Kubernetes Secret manifest, so direct render-and-apply is not shipped. A pipeline must supply its own reviewed JSON-to-Secret transformation (for example, jq plus kubectl create secret generic ... --from-file=... --dry-run=client -o yaml) before piping the manifest to kubectl apply -f -. Never commit the JSON, temporary value files, or rendered Secret to the GitOps repository. This is a values export operation under the normal authorization formula, so CI needs reveal explicitly to export secret values.

K3s: encrypt render-and-apply Secrets at rest

This path also creates a native Kubernetes Secret. Enable K3s --secrets-encryption. Select secretbox only on K3s ≥ v1.30.12+k3s1, v1.31.8+k3s1, v1.32.4+k3s1, or v1.33.0+k3s1. Use the AES-CBC provider on older releases.

For an operator-free pod, a user-authored init container can run hikyo values export ... into a shared emptyDir, with the application mounting that volume read-only. This reverses the operator’s availability trade-off: if Hikyo is unreachable at pod start, the init container fails and the application pod remains blocked.

An upstream External Secrets Operator provider is the named future direction after the Hikyo API freezes. It is not a v1 integration; ESO’s latest-minor support policy brings an ongoing monthly upstream release cost.

In-cluster Kubernetes issuersCopy Anchor Link

Discovery-mode federation can follow Kubernetes ServiceAccount signing-key rotation without copying a static JWKS document. Two prerequisites apply to private-CA clusters such as k3s:

1. The cluster must permit unauthenticated reads of its issuer discovery and JWKS endpoints. A cluster administrator can bind the built-in discovery role when that exposure is appropriate:

   Copy Text

   kubectl create clusterrolebinding hikyo-issuer-discovery \
     --clusterrole=system:service-account-issuer-discovery \
     --group=system:unauthenticated

   This exposes public discovery metadata and signing public keys, not workload tokens. Check the cluster’s anonymous-authentication policy too; an RBAC binding cannot enable an authentication mode the API server disables.

2. Configure the issuer with its cluster CA. Obtain ca.crt from a trusted administrator or the cluster’s kube-root-ca.crt ConfigMap:

   Copy Text

   kubectl --namespace default get configmap kube-root-ca.crt \
     -o jsonpath='{.data.ca\.crt}' > cluster-ca.pem
   hikyo instance-config federation-issuer add \
     --issuer https://kubernetes.default.svc.cluster.local \
     --type kubernetes --jwks discovery \
     --refuse-audience https://kubernetes.default.svc.cluster.local \
     --ca-bundle-file cluster-ca.pem

Use the cluster’s actual byte-exact issuer and default API-server audience; they need not be identical. The CA bundle contains PEM CA certificates only, up to 64 KiB. It replaces system roots for this issuer’s discovery and JWKS requests, including HTTPS redirects. Other issuers keep their own trust. Hostname verification remains mandatory, and HTTP redirects remain refused. The bundle is write-only; issuer listings report ca_bundle_configured.

TLS trust does not grant network reachability. The server’s OIDC egress policy must separately allow the private CIDRs for both the issuer origin and the origin in the discovered jwks_uri. For example, import an operator-owned file with HIKYO_OIDC_EGRESS_POLICY_FILE during initial setup:

{
  "https://kubernetes.default.svc.cluster.local": ["10.43.0.1/32"],
  "https://192.168.1.10:6443": ["192.168.1.10/32"]
}

Replace those addresses with the exact cluster endpoints. Existing managed instances change the node’s HIKYO_OIDC_EGRESS_POLICY_JSON through the explicit configuration publish and Apply flow.

Use federation-issuer update --id <issuer-id> --jwks discovery --refuse-audience <default-audience> --ca-bundle-file cluster-ca.pem to replace the CA bundle. Omitting the bundle preserves current discovery trust; --clear-ca-bundle restores system roots. Changing or clearing trust forces a fresh discovery/JWKS fetch before that configuration authenticates tokens. Switching to static mode clears the CA bundle, and --ca-bundle-file is refused with --jwks static. Static JWKS remains the offline option.