hikyo
Documentation

MCP operations

Connect native Codex and T3, authorize scoped MCP tools, and troubleshoot both transport profiles.

Hikyo exposes two opt-in, stateless JSON transport profiles: /mcp implements MCP 2026-07-28; /mcp/codex supports native Codex with MCP 2025-06-18. Both expose five bounded read tools, plus stage and validate when the write flag is enabled. Neither exposes resources, prompts, subscriptions, publish, Apply, secret plaintext reads, or secret draft material.

MCP authentication is a Hikyo-managed service-account bearer. Hikyo does not publish OAuth authorization-server metadata and does not perform automatic OAuth registration or token exchange. A client’s OAuth button is therefore not a substitute for provisioning a workload or automation credential in Hikyo.

Enable the endpoint

Set all three production facts explicitly:

HIKYO_EXTERNAL_ORIGIN=https://hikyo.example.com
HIKYO_MCP_ENABLED=true
HIKYO_MCP_ALLOWED_ORIGINS=

HIKYO_EXTERNAL_ORIGIN has no path or trailing slash. The endpoint appends /mcp. Leave the browser-origin list empty for Go, Inspector, Codex, Claude Code, and OpenAI API clients because they do not send an Origin header. Add only exact browser origins when a browser MCP client is required. Wildcards, null, paths, queries, fragments, credentials, and non-canonical origins stop startup.

The reverse proxy must:

  • terminate valid public HTTPS and preserve the canonical Host;
  • forward POST /mcp and POST /mcp/codex without rewriting either path;
  • pass Authorization, Mcp-Protocol-Version, Mcp-Method, and Mcp-Name;
  • allow request bodies up to 256 KiB and responses up to 256 KiB;
  • pass Hikyo security headers unchanged.

The transport returns ordinary JSON, not an SSE session. Disable response buffering only if the proxy imposes a smaller body limit. Do not configure cookie affinity or MCP session stickiness.

Enable the write surface

The endpoint is read-only by default. A second flag installs two more tools:

HIKYO_MCP_WRITE_ENABLED=true
ToolOperationGrantEffect
hikyo_stage_changevalue.stageedit on the environmentStages one set or unset as the caller’s own pending draft. Returns the version id and any secret-scanner findings.
hikyo_validate_changevalue.validateedit on the environmentReports what publish would refuse for a proposed set or unset, plus scanner findings. Stages nothing.

Both tools are audited with origin=mcp. Neither publishes or delivers: a staged draft is inert until a separate publish, and no MCP tool publishes. Whether the same identity may publish over the CLI or REST is decided by its grants there, unchanged by this flag. Secret reveal, publish, and every other mutation stay outside MCP. The flag requires HIKYO_MCP_ENABLED=true; with the flag off the write tools are not registered, so they appear in no tools/list and cannot be called.

Helm

The chart validates MCP settings and never accepts a client credential:

externalOrigin: https://hikyo.example.com
mcp:
  enabled: true
  allowedOrigins: []
ha:
  enabled: true
  replicaCount: 3
  minAvailable: 2

The chart ships no Ingress. Point the Ingress at service port 8080, use /readyz on operational port 8081 for readiness, and keep the operational port private. All replicas share PostgreSQL and the root-key authority. A load balancer may send every request to any ready replica.

Docker Compose

Use the executable fixture at deploy/mcp/compose.yaml with its exact deploy/mcp/nginx.conf reverse proxy. It pins the proxy image by digest, requires a digest-pinned Hikyo image, publishes only HTTPS publicly, preserves the exact /mcp and /mcp/codex routes and MCP headers, and configures no session affinity. Its one-shot staging service copies the root key into an owner-only file for the image’s nonroot process; Hikyo never consumes Compose’s group-readable file-backed secret directly. Keep the DSN, root key, and TLS key outside the Compose file:

export HIKYO_IMAGE='ghcr.io/hikyo-org/hikyo@sha256:<verified-release-digest>'
export HIKYO_DB="$(secret-store read hikyo-postgres-dsn)"
export HIKYO_EXTERNAL_ORIGIN=https://hikyo.example.com
export HIKYO_COMPOSE_SUBNET='<dedicated-compose-network-cidr>'
export HIKYO_ROOT_KEY_FILE=/run/credentials/hikyo-root-key
export HIKYO_TLS_CERT_FILE=/run/credentials/hikyo-tls-cert
export HIKYO_TLS_KEY_FILE=/run/credentials/hikyo-tls-key
export HIKYO_UPGRADE_PUBLIC_DIR=/srv/hikyo-public
export HIKYO_UPGRADE_INSTALLATION_DIR=/srv/hikyo-installation
./scripts/mcp-compose-preflight.sh
docker compose -f deploy/mcp/compose.yaml up -d root-key-stage
docker compose -f deploy/mcp/compose.yaml run --rm --no-deps hikyo migrate
docker compose -f deploy/mcp/compose.yaml up -d

Replace secret-store read with the deployment secret store’s non-logging read mechanism. Do not put its result in shell history or a committed env file.

Prepare the verified public bundle and operator.pub using the manual upgrade procedure. Provision the separate installation directory mode 0700, owned by runtime UID 65532; preserve it across database restores. Public artifacts mount read-only. The preflight checks paths and modes; the migration and boot gate authenticates signatures. Existing installations must stop every writer and provide fresh public restore evidence before the explicit migration. The server starts with automatic migration disabled. Never substitute fixture trust or development mode for production release artifacts.

Compose does not add replicas to one Hikyo installation safely by itself. Use the Helm HA shape or an equivalent orchestrator when multiple replicas are needed.

Provision least privilege

Use a dedicated automation identity for all five phase-1 tools. Its project-level read grant covers project listings and does not grant secret reveal. Workload identities retain environment-only grant scope; an environment grant does not authorize the project listing tools.

The write tools need a separate edit grant on each environment the identity may stage into. Keep it on its own identity so a read-only assistant never holds staging authority:

hikyo access grant add \
  --org <org-id> \
  --principal <machine-principal-id> \
  --capability edit \
  --project <project-id> \
  --env <environment-id>
hikyo sa create --org <org-id> --project <project-id> \
  --name mcp-read-prod --kind automation
hikyo access grant add \
  --org <org-id> \
  --principal <machine-principal-id> \
  --capability read \
  --project <project-id>
hikyo sa credential mint \
  --org <org-id> \
  --project <project-id> \
  --sa <service-account-id> \
  --lifetime 720h \
  --output-file /run/credentials/hikyo-mcp-token

Keep the returned principal ID, service-account ID, and credential ID in the deployment record. The bearer is shown once. Put it in the client’s secret store or an owner-only runtime file, never in Helm values, Compose YAML, .mcp.json, shell history, logs, traces, or metrics.

Client profiles

The interoperability gate pins these profiles. Configuration examples below describe credential channels; a listed profile is not a passing client claim.

ClientPinned profileCredential channel
Official Go SDKgithub.com/modelcontextprotocol/go-sdk v1.8.0HTTP transport injects an environment-loaded bearer
MCP Inspector@modelcontextprotocol/inspector 2.10.1Transient Authorization header, not a saved catalog
Codex CLI0.160.1, /mcp/codexbearer_token_env_var
Claude Code2.1.261.mcp.json environment expansion in headers
OpenAI Responses APIremote MCP toolrequest-time authorization, restricted allowed_tools

The 5 September 2026 rehearsal used the actual server image at source 9827af64962a36e2e31906b48906fd59c7a36c1c, two PostgreSQL-backed replicas, and verified local HTTPS. These observations are scoped to that image and endpoint:

ClientObserved production result
Official Go SDK v1.7.0All five production tools passed with a scoped synthetic identity.
Inspector 2.5.0, ModernAuthenticated, nonempty hikyo_list_definitions passed.
Codex CLI 0.153.2Unverified: isolated direct tool invocation failed in the client transport before reaching the local proxy. No protocol compatibility verdict.
Claude Code 2.1.261Its health check reached HTTPS but sent legacy initialize without the required mirrored protocol/method headers; the server refused it. This tested handshake is incompatible with the required modern profile. No production tool call passed.
OpenAI Responses APIUnverified: no authorized API credential or provider call.

The same local deployment passed cross-replica cursor continuation, wrong-scope refusal, and revocation on both replicas. Publicly trusted remote HTTPS remains unverified because the disposable ingress failed its network preflight. No model turns or paid provider requests were used. See the redacted rehearsal evidence for image identity, limitations, and the excluded invalid endpoint attempt. Those historical observations do not cover the compatibility profile added for #781. Use the explicit Codex route below; do not weaken TLS or global modern endpoint validation.

On 16 October 2026, actual Codex 0.160.1 passed local in-process HTTP tests on SQLite and PostgreSQL. The five-tool read-only profile exercised definitions and published configuration reads; the seven-tool profile exercised those two calls plus validation and staging. All seven tools were advertised, but only those four were invoked. The measured initialization and subsequent protocol headers were 2025-06-18. Typed config/secret checks and all-frame canary/token exclusion passed. This does not establish public HTTPS or final-candidate acceptance. Current Go SDK v1.8.0 and Inspector 2.10.1 are source pins, not new deployed passes. Current Claude and Responses remain unverified. See the current client support table and evidence.

Set the credential only in the launching process:

export HIKYO_MCP_URL=https://hikyo.example.com/mcp
export HIKYO_MCP_TOKEN="$(</run/credentials/hikyo-mcp-token)"

Codex and T3

Use /mcp/codex, not /mcp, for native Codex. A static header override cannot translate the initialization lifecycle. The profile pins 2025-06-18, accepts initialization and notification messages, and returns ordinary JSON. It creates no session, requires no affinity, and supports no server-initiated streams. Other protocol versions receive an actionable error. Tool calls retain the same bearer, grant, scope, redaction, audit and size checks as the modern route. Cancellation propagates when the HTTP request is cancelled; cancellation notifications are acknowledged without creating connection state.

[mcp_servers.hikyo]
url = "https://hikyo.example.com/mcp/codex"
bearer_token_env_var = "HIKYO_MCP_TOKEN"

The equivalent command is:

codex mcp add hikyo \
  --url "https://hikyo.example.com/mcp/codex" \
  --bearer-token-env-var HIKYO_MCP_TOKEN

Codex 0.160.1 was tested against disposable SQLite and PostgreSQL datastores using actual codex app-server, mcpServerStatus/list and mcpServer/tool/call, without a model turn or provider request. It discovered all seven tools and exercised four calls: definitions, published configuration, validation without staging, and staging exactly one inert draft. The separate read-only test discovers five tools and sends only the first two calls. Both verify config/secret boundaries; the write-profile test verifies origin=mcp audit rows and absence of publication. Reproduce with a private disposable HIKYO_TEST_POSTGRES_DSN to include PostgreSQL; without it that engine skips:

HIKYO_TEST_CODEX_NATIVE=1 go test ./internal/isolation \
  -run '^TestMCPCodexNative(ReadOnly)?$' -count=1 -v

For T3, configure the same file in the Codex home selected by its provider settings. Set HIKYO_MCP_TOKEN in T3’s provider environment, or inherit it from the process launching T3. A desktop launch does not automatically inherit an export from an unrelated terminal. T3’s Codex homePath overrides CODEX_HOME; editing a different home’s config will not affect that provider. Restart the provider session after changing its environment or MCP configuration. Do not put a bearer literal in TOML, tool arguments, prompts, or screenshots.

The separate T3 verification scope and exact source identity are recorded in the #781 evidence.

Claude Code

Commit only the variable references:

{
  "mcpServers": {
    "hikyo": {
      "type": "http",
      "url": "${HIKYO_MCP_URL}",
      "headers": {
        "Authorization": "Bearer ${HIKYO_MCP_TOKEN}"
      }
    }
  }
}

Claude Code warns and leaves a missing ${VAR} reference unexpanded; it does not refuse the server automatically. Treat either missing-variable warning as fatal, correct the launching environment, and approve the project-scoped server only after reviewing the expanded URL and tool catalog.

MCP Inspector

Inspector’s CLI header is a command-line value, so expanding a production credential there can expose it to process inspection. For a manual smoke, start Inspector 2.10.1, choose Streamable HTTP, set the URL from HIKYO_MCP_URL, and set Protocol Era to Modern. Paste the bearer into the transient Authorization field from the secret manager, call hikyo_list_definitions, then clear the field and do not save the server catalog. The default legacy/auto profile is not Hikyo’s 2026-07-28 wire contract.

CI runs Inspector only against a non-tenant fixture. The production public smoke below uses a Go process so credential values never enter command-line arguments.

OpenAI Responses API

The OpenAI API receives MCP tool results, so use it only after approving that recipient. Pass the Hikyo credential on every request; the Responses API does not return it in the response object:

import os
from openai import OpenAI

client = OpenAI()
response = client.responses.create(
    model=os.environ["OPENAI_MODEL"],
    input="List the key definitions in the configured Hikyo project.",
    tools=[{
        "type": "mcp",
        "server_label": "hikyo",
        "server_url": os.environ["HIKYO_MCP_URL"],
        "authorization": os.environ["HIKYO_MCP_TOKEN"],
        "allowed_tools": ["hikyo_list_definitions"],
        "require_approval": "never",
    }],
)
print(response.output_text)

require_approval: never is limited here to Hikyo’s closed read-only tool. Keep approval enabled for any future tool whose effects are not read-only.

Rotate, revoke, and prove the deployment

Mint a replacement credential, update every client secret store, and make one successful call with the replacement. Keep the predecessor live until the checker below has proved it works:

hikyo sa credential mint \
  --org <org-id> \
  --project <project-id> \
  --sa <service-account-id> \
  --lifetime 720h \
  --output-file /run/credentials/hikyo-mcp-token-next

Start the public checker with both credentials. It proves the predecessor is live, prints a safe prompt, and then polls for at most two minutes. In another terminal, revoke that exact credential. The first process proves the same credential changed from authorized to tenant-safe denial without ever writing the bearer, response body, or tool result:

export HIKYO_MCP_TOKEN="$(</run/credentials/hikyo-mcp-token-next)"
export HIKYO_MCP_ROTATING_TOKEN="$(</run/credentials/hikyo-mcp-token)"
go run ./scripts/mcp-public-smoke \
  --url "$HIKYO_MCP_URL" \
  --org <org-id> \
  --project <project-id>
unset HIKYO_MCP_TOKEN HIKYO_MCP_ROTATING_TOKEN
hikyo sa credential revoke \
  --org <org-id> \
  --project <project-id> \
  --sa <service-account-id> \
  --id <old-credential-id>

The check proves unauthenticated discovery, the closed tool catalog, one authorized safe read, invalid-token denial, immediate revoked-token denial, and absence of an MCP session ID. Invalid and revoked credentials must return the same uniform 401.

Conformance, replicas, and shutdown

CI locks @modelcontextprotocol/conformance@0.2.0-alpha.11, the first published suite profile carrying the 2026-07-28 server requirements used here, and all of its transitive packages. A conformance-only fixture exposes the upstream diagnostic tools, so no server-stateless MUST check is baselined. The committed baseline names only prompts/resources capabilities Hikyo deliberately does not advertise. Any new failure changes the baseline result and fails CI.

The isolation suite sends a continuation cursor between independently loaded keyrings, auth/admission services, domain services, registries, and handlers over the shared datastore and root authority. It also asserts that no Mcp-Session-Id is emitted and that completed calls leave no shared concurrency claim. Shutdown tests prove an in-flight MCP call may complete inside the five-second drain and that expiry cancels its request context. The chart and Compose grace periods both exceed that server drain budget.

Metrics and logs

MCP metrics use only closed labels. They never contain a raw path, ID, bearer, argument, result, or tenant name:

MetricLabels
hikyo_mcp_requests_totalmethod, tool, status
hikyo_mcp_requests_in_flightnone
hikyo_mcp_request_duration_secondsmethod, tool

The method labels are server/discover, tools/list, tools/call, and other. Tool labels are the five read names, hikyo_stage_change, hikyo_validate_change, none, and other. The two write labels remain in the closed label set even when write tools are disabled. Debug access logs use the same closed method/tool values plus status and duration. Trace and log exporters must not capture request headers or bodies.

Troubleshooting

SymptomCause and fix
404 at /mcpHIKYO_MCP_ENABLED is false, or the proxy rewrote the path. Enable it and forward exact POST /mcp.
401 with WWW-Authenticate: BearerThe tool call supplied no single bearer header, or the token is invalid, expired, or revoked. Repair the client’s runtime secret channel or rotate the credential.
Hikyo operation refusedThe token is live but is the wrong principal class or lacks scope. Check credential metadata and grants without logging the bearer.
unsupported protocol versionThe client did not negotiate 2026-07-28. Upgrade it to the pinned profile.
protocol mirror headers do not match requestA proxy or client removed or changed the modern MCP mirror headers. Preserve all three headers.

After any incident, revoke the exposed credential first, mint a replacement, redistribute it, and repeat the public checker. Do not widen the grant to make an authentication failure disappear.

Next: manage machine identities.