MCP operations
Connect native Codex and T3, authorize scoped MCP tools, and troubleshoot both transport profiles.
Hikyo exposes two opt-in, stateless JSON transport profiles: /mcp implements
MCP 2026-07-28; /mcp/codex supports native Codex with MCP 2025-06-18.
Both expose five bounded read tools, plus stage and validate when the write
flag is enabled. Neither exposes resources, prompts, subscriptions, publish,
Apply, secret plaintext reads, or secret draft material.
MCP authentication is a Hikyo-managed service-account bearer. Hikyo does not publish OAuth authorization-server metadata and does not perform automatic OAuth registration or token exchange. A client’s OAuth button is therefore not a substitute for provisioning a workload or automation credential in Hikyo.
Enable the endpoint
Set all three production facts explicitly:
HIKYO_EXTERNAL_ORIGIN=https://hikyo.example.com
HIKYO_MCP_ENABLED=true
HIKYO_MCP_ALLOWED_ORIGINS=HIKYO_EXTERNAL_ORIGIN has no path or trailing slash. The endpoint appends
/mcp. Leave the browser-origin list empty for Go, Inspector, Codex, Claude
Code, and OpenAI API clients because they do not send an Origin header. Add
only exact browser origins when a browser MCP client is required. Wildcards,
null, paths, queries, fragments, credentials, and non-canonical origins stop
startup.
The reverse proxy must:
- terminate valid public HTTPS and preserve the canonical
Host; - forward
POST /mcpandPOST /mcp/codexwithout rewriting either path; - pass
Authorization,Mcp-Protocol-Version,Mcp-Method, andMcp-Name; - allow request bodies up to 256 KiB and responses up to 256 KiB;
- pass Hikyo security headers unchanged.
The transport returns ordinary JSON, not an SSE session. Disable response buffering only if the proxy imposes a smaller body limit. Do not configure cookie affinity or MCP session stickiness.
Enable the write surface
The endpoint is read-only by default. A second flag installs two more tools:
HIKYO_MCP_WRITE_ENABLED=true| Tool | Operation | Grant | Effect |
|---|---|---|---|
hikyo_stage_change | value.stage | edit on the environment | Stages one set or unset as the caller’s own pending draft. Returns the version id and any secret-scanner findings. |
hikyo_validate_change | value.validate | edit on the environment | Reports what publish would refuse for a proposed set or unset, plus scanner findings. Stages nothing. |
Both tools are audited with origin=mcp. Neither publishes or delivers: a
staged draft is inert until a separate publish, and no MCP tool publishes.
Whether the same identity may publish over the CLI or REST is decided by its
grants there, unchanged by this flag. Secret reveal, publish, and every other
mutation stay outside MCP. The flag requires HIKYO_MCP_ENABLED=true; with the flag off the write
tools are not registered, so they appear in no tools/list and cannot be
called.
Helm
The chart validates MCP settings and never accepts a client credential:
externalOrigin: https://hikyo.example.com
mcp:
enabled: true
allowedOrigins: []
ha:
enabled: true
replicaCount: 3
minAvailable: 2The chart ships no Ingress. Point the Ingress at service port 8080, use
/readyz on operational port 8081 for readiness, and keep the operational
port private. All replicas share PostgreSQL and the root-key authority. A load
balancer may send every request to any ready replica.
Docker Compose
Use the executable fixture at deploy/mcp/compose.yaml with its exact
deploy/mcp/nginx.conf reverse proxy. It pins the proxy image by digest,
requires a digest-pinned Hikyo image, publishes only HTTPS publicly, preserves
the exact /mcp and /mcp/codex routes and MCP headers, and configures no session affinity.
Its one-shot staging service copies the root key into an owner-only file for
the image’s nonroot process; Hikyo never consumes Compose’s group-readable
file-backed secret directly. Keep the DSN, root key, and TLS key outside the
Compose file:
export HIKYO_IMAGE='ghcr.io/hikyo-org/hikyo@sha256:<verified-release-digest>'
export HIKYO_DB="$(secret-store read hikyo-postgres-dsn)"
export HIKYO_EXTERNAL_ORIGIN=https://hikyo.example.com
export HIKYO_COMPOSE_SUBNET='<dedicated-compose-network-cidr>'
export HIKYO_ROOT_KEY_FILE=/run/credentials/hikyo-root-key
export HIKYO_TLS_CERT_FILE=/run/credentials/hikyo-tls-cert
export HIKYO_TLS_KEY_FILE=/run/credentials/hikyo-tls-key
export HIKYO_UPGRADE_PUBLIC_DIR=/srv/hikyo-public
export HIKYO_UPGRADE_INSTALLATION_DIR=/srv/hikyo-installation
./scripts/mcp-compose-preflight.sh
docker compose -f deploy/mcp/compose.yaml up -d root-key-stage
docker compose -f deploy/mcp/compose.yaml run --rm --no-deps hikyo migrate
docker compose -f deploy/mcp/compose.yaml up -dReplace secret-store read with the deployment secret store’s non-logging
read mechanism. Do not put its result in shell history or a committed env file.
Prepare the verified public bundle and operator.pub using the
manual upgrade procedure. Provision the separate installation
directory mode 0700, owned by runtime UID 65532; preserve it across database
restores. Public artifacts mount read-only. The preflight checks paths and modes;
the migration and boot gate authenticates signatures. Existing installations
must stop every writer and provide fresh public restore evidence before the
explicit migration. The server starts with automatic migration disabled. Never
substitute fixture trust or development mode for production release artifacts.
Compose does not add replicas to one Hikyo installation safely by itself. Use the Helm HA shape or an equivalent orchestrator when multiple replicas are needed.
Provision least privilege
Use a dedicated automation identity for all five phase-1 tools. Its project-level
read grant covers project listings and does not grant secret reveal. Workload
identities retain environment-only grant scope; an environment grant does not
authorize the project listing tools.
The write tools need a separate edit grant on each environment the identity
may stage into. Keep it on its own identity so a read-only assistant never
holds staging authority:
hikyo access grant add \
--org <org-id> \
--principal <machine-principal-id> \
--capability edit \
--project <project-id> \
--env <environment-id>hikyo sa create --org <org-id> --project <project-id> \
--name mcp-read-prod --kind automation
hikyo access grant add \
--org <org-id> \
--principal <machine-principal-id> \
--capability read \
--project <project-id>
hikyo sa credential mint \
--org <org-id> \
--project <project-id> \
--sa <service-account-id> \
--lifetime 720h \
--output-file /run/credentials/hikyo-mcp-tokenKeep the returned principal ID, service-account ID, and credential ID in the
deployment record. The bearer is shown once. Put it in the client’s secret
store or an owner-only runtime file, never in Helm values, Compose YAML,
.mcp.json, shell history, logs, traces, or metrics.
Client profiles
The interoperability gate pins these profiles. Configuration examples below describe credential channels; a listed profile is not a passing client claim.
| Client | Pinned profile | Credential channel |
|---|---|---|
| Official Go SDK | github.com/modelcontextprotocol/go-sdk v1.8.0 | HTTP transport injects an environment-loaded bearer |
| MCP Inspector | @modelcontextprotocol/inspector 2.10.1 | Transient Authorization header, not a saved catalog |
| Codex CLI | 0.160.1, /mcp/codex | bearer_token_env_var |
| Claude Code | 2.1.261 | .mcp.json environment expansion in headers |
| OpenAI Responses API | remote MCP tool | request-time authorization, restricted allowed_tools |
The 5 September 2026 rehearsal used the actual server image at source
9827af64962a36e2e31906b48906fd59c7a36c1c, two PostgreSQL-backed replicas, and
verified local HTTPS. These observations are scoped to that image and endpoint:
| Client | Observed production result |
|---|---|
Official Go SDK v1.7.0 | All five production tools passed with a scoped synthetic identity. |
Inspector 2.5.0, Modern | Authenticated, nonempty hikyo_list_definitions passed. |
Codex CLI 0.153.2 | Unverified: isolated direct tool invocation failed in the client transport before reaching the local proxy. No protocol compatibility verdict. |
Claude Code 2.1.261 | Its health check reached HTTPS but sent legacy initialize without the required mirrored protocol/method headers; the server refused it. This tested handshake is incompatible with the required modern profile. No production tool call passed. |
| OpenAI Responses API | Unverified: no authorized API credential or provider call. |
The same local deployment passed cross-replica cursor continuation, wrong-scope refusal, and revocation on both replicas. Publicly trusted remote HTTPS remains unverified because the disposable ingress failed its network preflight. No model turns or paid provider requests were used. See the redacted rehearsal evidence for image identity, limitations, and the excluded invalid endpoint attempt. Those historical observations do not cover the compatibility profile added for #781. Use the explicit Codex route below; do not weaken TLS or global modern endpoint validation.
On 16 October 2026, actual Codex 0.160.1 passed local in-process HTTP tests
on SQLite and PostgreSQL. The five-tool read-only profile exercised definitions
and published configuration reads; the seven-tool profile exercised those two
calls plus validation and staging. All seven tools were advertised, but only
those four were invoked. The measured initialization and subsequent protocol
headers were 2025-06-18. Typed config/secret checks and all-frame canary/token
exclusion passed. This does not establish public HTTPS or final-candidate
acceptance. Current Go SDK v1.8.0 and Inspector 2.10.1 are source pins, not
new deployed passes. Current Claude and Responses remain unverified. See the
current client support table and evidence.
Set the credential only in the launching process:
export HIKYO_MCP_URL=https://hikyo.example.com/mcp
export HIKYO_MCP_TOKEN="$(</run/credentials/hikyo-mcp-token)"Codex and T3
Use /mcp/codex, not /mcp, for native Codex. A static header override cannot
translate the initialization lifecycle. The profile pins 2025-06-18, accepts
initialization and notification messages, and returns ordinary JSON. It creates
no session, requires no affinity, and supports no server-initiated streams.
Other protocol versions receive an actionable error. Tool calls retain the
same bearer, grant, scope, redaction, audit and size checks as the modern route.
Cancellation propagates when the HTTP request is cancelled; cancellation
notifications are acknowledged without creating connection state.
[mcp_servers.hikyo]
url = "https://hikyo.example.com/mcp/codex"
bearer_token_env_var = "HIKYO_MCP_TOKEN"The equivalent command is:
codex mcp add hikyo \
--url "https://hikyo.example.com/mcp/codex" \
--bearer-token-env-var HIKYO_MCP_TOKENCodex 0.160.1 was tested against disposable SQLite and PostgreSQL datastores using actual
codex app-server, mcpServerStatus/list and mcpServer/tool/call, without a
model turn or provider request. It discovered all seven tools and exercised
four calls: definitions, published configuration, validation without staging,
and staging exactly one inert draft. The separate read-only test discovers
five tools and sends only the first two calls. Both verify config/secret
boundaries; the write-profile test verifies origin=mcp audit rows and absence
of publication. Reproduce with a private disposable HIKYO_TEST_POSTGRES_DSN
to include PostgreSQL; without it that engine skips:
HIKYO_TEST_CODEX_NATIVE=1 go test ./internal/isolation \
-run '^TestMCPCodexNative(ReadOnly)?$' -count=1 -vFor T3, configure the same file in the Codex home selected by its provider
settings. Set HIKYO_MCP_TOKEN in T3’s provider environment, or inherit it from
the process launching T3. A desktop launch does not automatically inherit an
export from an unrelated terminal. T3’s Codex homePath overrides CODEX_HOME;
editing a different home’s config will not affect that provider. Restart the
provider session after changing its environment or MCP configuration. Do not
put a bearer literal in TOML, tool arguments, prompts, or screenshots.
The separate T3 verification scope and exact source identity are recorded in the #781 evidence.
Claude Code
Commit only the variable references:
{
"mcpServers": {
"hikyo": {
"type": "http",
"url": "${HIKYO_MCP_URL}",
"headers": {
"Authorization": "Bearer ${HIKYO_MCP_TOKEN}"
}
}
}
}Claude Code warns and leaves a missing ${VAR} reference unexpanded; it does
not refuse the server automatically. Treat either missing-variable warning as
fatal, correct the launching environment, and approve the project-scoped
server only after reviewing the expanded URL and tool catalog.
MCP Inspector
Inspector’s CLI header is a command-line value, so expanding a production
credential there can expose it to process inspection. For a manual smoke,
start Inspector 2.10.1, choose Streamable HTTP, set the URL from
HIKYO_MCP_URL, and set Protocol Era to Modern. Paste the bearer into the
transient Authorization field from the secret manager, call
hikyo_list_definitions, then clear the field and do not save the server
catalog. The default legacy/auto profile is not Hikyo’s 2026-07-28 wire
contract.
CI runs Inspector only against a non-tenant fixture. The production public smoke below uses a Go process so credential values never enter command-line arguments.
OpenAI Responses API
The OpenAI API receives MCP tool results, so use it only after approving that recipient. Pass the Hikyo credential on every request; the Responses API does not return it in the response object:
import os
from openai import OpenAI
client = OpenAI()
response = client.responses.create(
model=os.environ["OPENAI_MODEL"],
input="List the key definitions in the configured Hikyo project.",
tools=[{
"type": "mcp",
"server_label": "hikyo",
"server_url": os.environ["HIKYO_MCP_URL"],
"authorization": os.environ["HIKYO_MCP_TOKEN"],
"allowed_tools": ["hikyo_list_definitions"],
"require_approval": "never",
}],
)
print(response.output_text)require_approval: never is limited here to Hikyo’s closed read-only tool.
Keep approval enabled for any future tool whose effects are not read-only.
Rotate, revoke, and prove the deployment
Mint a replacement credential, update every client secret store, and make one successful call with the replacement. Keep the predecessor live until the checker below has proved it works:
hikyo sa credential mint \
--org <org-id> \
--project <project-id> \
--sa <service-account-id> \
--lifetime 720h \
--output-file /run/credentials/hikyo-mcp-token-nextStart the public checker with both credentials. It proves the predecessor is live, prints a safe prompt, and then polls for at most two minutes. In another terminal, revoke that exact credential. The first process proves the same credential changed from authorized to tenant-safe denial without ever writing the bearer, response body, or tool result:
export HIKYO_MCP_TOKEN="$(</run/credentials/hikyo-mcp-token-next)"
export HIKYO_MCP_ROTATING_TOKEN="$(</run/credentials/hikyo-mcp-token)"
go run ./scripts/mcp-public-smoke \
--url "$HIKYO_MCP_URL" \
--org <org-id> \
--project <project-id>
unset HIKYO_MCP_TOKEN HIKYO_MCP_ROTATING_TOKENhikyo sa credential revoke \
--org <org-id> \
--project <project-id> \
--sa <service-account-id> \
--id <old-credential-id>The check proves unauthenticated discovery, the closed tool catalog, one
authorized safe read, invalid-token denial, immediate revoked-token denial,
and absence of an MCP session ID. Invalid and revoked credentials must return
the same uniform 401.
Conformance, replicas, and shutdown
CI locks @modelcontextprotocol/conformance@0.2.0-alpha.11, the first published
suite profile carrying the 2026-07-28 server requirements used here, and all
of its transitive packages. A conformance-only fixture exposes the upstream
diagnostic tools, so no server-stateless MUST check is baselined. The committed
baseline names only prompts/resources capabilities Hikyo deliberately does not
advertise. Any new failure changes the baseline result and fails CI.
The isolation suite sends a continuation cursor between independently loaded
keyrings, auth/admission services, domain services, registries, and handlers
over the shared datastore and root authority. It also asserts that no
Mcp-Session-Id is emitted and that completed calls leave no shared concurrency
claim. Shutdown tests prove an in-flight MCP call may complete inside the
five-second drain and that expiry cancels its request context. The chart and
Compose grace periods both exceed that server drain budget.
Metrics and logs
MCP metrics use only closed labels. They never contain a raw path, ID, bearer, argument, result, or tenant name:
| Metric | Labels |
|---|---|
hikyo_mcp_requests_total | method, tool, status |
hikyo_mcp_requests_in_flight | none |
hikyo_mcp_request_duration_seconds | method, tool |
The method labels are server/discover, tools/list, tools/call, and
other. Tool labels are the five read names, hikyo_stage_change,
hikyo_validate_change, none, and other. The two write labels remain in
the closed label set even when write tools are disabled.
Debug access logs use the same closed method/tool values plus status and
duration. Trace and log exporters must not capture request headers or bodies.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
404 at /mcp | HIKYO_MCP_ENABLED is false, or the proxy rewrote the path. Enable it and forward exact POST /mcp. |
401 with WWW-Authenticate: Bearer | The tool call supplied no single bearer header, or the token is invalid, expired, or revoked. Repair the client’s runtime secret channel or rotate the credential. |
Hikyo operation refused | The token is live but is the wrong principal class or lacks scope. Check credential metadata and grants without logging the bearer. |
unsupported protocol version | The client did not negotiate 2026-07-28. Upgrade it to the pinned profile. |
protocol mirror headers do not match request | A proxy or client removed or changed the modern MCP mirror headers. Preserve all three headers. |
After any incident, revoke the exposed credential first, mint a replacement, redistribute it, and repeat the public checker. Do not widen the grant to make an authentication failure disappear.
Next: manage machine identities.
Private PKI
Issue short-lived X.509 service certificates from a self-hosted CA whose private keys never leave Hikyo, under closed profiles that can only narrow.
Local development and coding agents
Deliver development values to a dev server, tests, and non-interactive coding agents, and compose Hikyo with application-side loaders.