MCP and LLM gateways

The short answer

Two different proxies share the name, and they carry different authority.

Three things hold for both.

• The agent’s credential stops at the gateway. It is addressed to the gateway, the gateway checks it, and it goes no further.
• The gateway uses a different credential on the far side, made for the one server or provider it is calling.
• A gateway governs only the calls that go through it. Either nothing else can reach what is behind it, or what is behind it accepts the gateway alone, or it applies the same rules itself.

Neither gateway is a place an agent runs. An agent on a laptop, in a pod or on a managed platform can sit behind both, and the other deployment guides still apply to it.

The MCP gateway

An MCP gateway is one address in front of many MCP servers. The agent connects to the gateway. The gateway lists the tools of every server behind it as if they were its own, sends each call to the server that owns the tool, applies a rule about who may call which tool, and keeps one log. Teams adopt one so that those things are done once, and not again in every agent and every server.

One box, two roles

To the agent, the gateway is the MCP server. Everything on the MCP authorization page applies to it as written. It answers a request that has no token with a pointer to its own protected resource metadata. That metadata names the gateway’s address as the resource. The token the agent then obtains has the gateway as its audience, and the gateway checks that on every request.

One sign-in now covers every server behind the gateway. That is the convenience, and it is why the other role matters.

To each server behind it, the gateway is a client. The token it just accepted says “for the gateway.” It is not addressed to the ticketing server or to the wiki server. The MCP specification covers this case directly: a server that calls something upstream acts as a client there, uses a separate token issued for that upstream, and must not pass on the token it received.

So the gateway needs a credential of its own for each server, and the one it uses decides what that server can know.

What the server behind it can tell

Here is the first row with one user, one agent and a ticketing server.

What the agent sends the gateway:

{
  "iss": "https://auth.example",
  "sub": "user:alice",
  "client_id": "support-agent",
  "aud": "https://mcp.example/mcp",
  "scope": "tools"
}

What the gateway sends the ticketing server, after an exchange:

{
  "iss": "https://auth.example",
  "sub": "user:alice",
  "aud": "https://tickets.example/mcp",
  "scope": "tickets.read",
  "act": {
    "sub": "https://mcp.example/mcp"
  }
}

The ticketing server gets a token that works in one place, names Alice, and names the gateway as the party acting for her. Three things about this arrangement are worth checking in your own.

• What your authorization server really issues. An exchange can return a token with the user as its subject and no actor. The server then believes Alice called it directly, which is impersonation.
• Which agent started the call. client_id named it in the first token. Nothing in the second token does, and an exchange does not carry it across unless your authorization server is set up to. If the server’s rules or its audit log need the agent, look for it in a real token.
• Who can get a token for a server. Let the gateway, and not the agents, exchange tokens for the servers’ audiences, and hold every other way your authorization server issues a token for them to the same rule. That narrows who can hold such a token. It does not, on its own, keep callers behind the gateway: see calls that go around it.

One key for everybody

The fourth row is the easiest to set up: one API key for each server, written into the gateway’s configuration. It is also where a gateway most often goes wrong.

The server now sees one caller. Nothing but the gateway’s own rules keeps one user out of another’s records, and the gateway has become the confused deputy: it holds broad authority and uses it for whoever asks. Lesson 1 draws the same problem with one agent and one key.

That is acceptable for a server that searches public documentation. It is not acceptable for a mailbox, a ticket queue or anything else that holds each user’s own data.

An agent that works as itself, with nobody signed in, is a different case. There is no user to carry, and the call rightly uses the agent’s own identity. Give each agent its own, and take it across the hop as you would a user, so that the server can tell which agent called and not only that the gateway did.

When the gateway keeps users’ grants

In the third row each user connects a third-party service once, through the gateway, and the gateway stores the refresh token. It is now a token store with a row for every user at every service, and where the tokens live on a hosted app applies to it line for line. MCP has a way to send the user to that sign-in in the middle of a tool call: URL elicitation.

The MCP security guidance has a rule for this arrangement, which it calls an MCP proxy server. The trouble starts when three things are true at once: the gateway signs users in to the third party under one client ID of its own, it accepts any MCP client that registers with it, and the third party remembers that the user has already approved the gateway.

An attacker registers a client with the gateway and sends the user a link. The third party recognizes the user, skips its consent screen, and the gateway hands the attacker’s client a code. The user approved nothing.

The guidance requires the gateway to ask the user itself, on a page of its own, before it sends them on. That page names the MCP client and shows where the code will be sent. The gateway records the approval for that client, and does not treat it as “this user has consented.”

Rules at the gateway

The core MCP specification has no permissions for individual tools. A token that is good for the gateway is good for every tool behind it, until the gateway’s own policy says otherwise. That policy is the authorization, so what it reads matters.

• The user and the client come from the token the gateway validated. Never from a header the agent set, and never from an argument the model wrote.
• The tool comes from the request. Since the 2026-07-28 revision, an MCP request over HTTP repeats its method and its tool name in the Mcp-Method and Mcp-Name headers, so that a proxy can route without reading the body. The body is still what the server acts on, so an agent can name one tool in the header and another in the body. A rule that reads the header is only as good as the check that the two agree.
• Narrow at the exchange. Ask for the scope the tool being called needs, and nothing wider.

Who makes that check depends on what the gateway does with the request.

• It sends the request on unchanged, headers and body exactly as they arrived. The server behind it compares the two, which the specification requires of every server that reads the body, and rejects a mismatch. The specification also advises a proxy to refuse a request whose protocol version is too old to require that comparison.
• It reads the body and writes a new request. A gateway in front of several servers usually does, to choose the server and to rename the tool. It is then a server that reads the body, and the comparison is its own to make: on the request as it arrived, before it relies on the headers or the body. The server behind it cannot make it. The headers the gateway sends on are its own, written from the body, so they agree at the server whatever the agent sent.

The gateway also ends the agent’s TLS connection. A client certificate does not arrive at the server, which now authenticates the gateway. If the server has to know which workload called as well, that is what workload identity tokens are being standardized for.

A gateway that only forwards

Some gateways can pass requests through untouched to a server that runs OAuth itself. Products call the setting passthrough, which is not always the thing the specification forbids.

The test is whose address is in the token’s audience.

• One server behind one public address. The token is addressed to that address, and the server checks it. The gateway is a hop on the network, like a load balancer. It can log and block, and it is not a party to the authorization. This is sound.
• Several servers behind one audience. The gateway accepts the token as its own and then sends it on. This is token passthrough. Every server behind the gateway receives a token it could present at the gateway, to call the other servers as that user.

The LLM gateway

An LLM gateway, often sold as an AI gateway, is one address in front of the model providers. Agents and apps send it model requests in one format. It picks a provider, adds that provider’s credential, counts usage against a budget, and logs the exchange. Platform teams adopt one to keep provider keys in one place and to know who is spending what.

Start with whose authority a call uses. A model call normally uses the organization’s account with the provider, and no user’s. It reads and changes nothing that belongs to a user. That makes this the simpler of the two gateways, with the same two hops.

From the agent to the gateway

Most LLM gateways hand out virtual keys. The gateway issues a key to a team or an app, with a budget and a list of models, and swaps it for the provider’s key on the way out. This is a real improvement on giving every team the provider’s key. One caller can be cut off without rotating anything, and the provider’s key sits in one place.

A virtual key is still an API key. It is long-lived, it works for anyone who holds a copy, and it has to be stored wherever the agent runs. In an agent that reads untrusted content, it is one more secret an injected instruction can send somewhere.

Where the gateway can check a token, the agent has nothing to store.

• A workload presents the identity its platform already gives it: on Kubernetes, a ServiceAccount token whose audience is the gateway, and in a pipeline, the job’s OIDC token.
• A person’s tool, such as a coding agent on a laptop, presents a token from the person’s own sign-in with the company identity provider. Usage is then filed under a person. Disabling their account stops new tokens. A token already issued is still good until it expires, unless the gateway asks the authorization server about each one, so keep those tokens short-lived.

From the gateway to the provider

On this side the gateway is a credential broker. The provider’s key lives in the gateway and is attached on the way out, and the agent never sees it. What the sandboxes page says about that pattern applies, including that the gateway is now the most sensitive part of the system. It holds every provider key.

The better case has no key at all. Model services run by the cloud providers accept the cloud’s own identities. A gateway can trade its workload identity for short-lived credentials there, through federation when it runs somewhere else. Then nothing long-lived is stored at either hop.

What the provider sees, and what the gateway knows

With one shared credential and nothing else sent, the provider sees one account. Which team, which agent and which person made a call is recorded at the gateway and nowhere else. For model calls that is usually fine: one key for one owner is what an API key is for.

A gateway can pass that on. When agentgateway assumes an AWS role, for example, it can name and tag the session from the caller’s token, and those values appear in AWS CloudTrail and in the cost report. That changes who can see the caller. It does not change whose permissions apply. The provider still enforces what the gateway’s credential may do, and the label is the gateway’s word for who asked.

Two things follow.

• The gateway decides who the record says called, in its own log and in anything it passes on. Take the caller from the credential the gateway checked, not from a field the caller filled in.
• That log holds every prompt and every reply, and so whatever the agents read: documents, tool results, customer data. If a credential ever lands in a prompt, it is now in the gateway’s log as well.

What it cannot decide

An LLM gateway sees the model ask for a tool call. It does not see the tool call happen. The agent’s runtime makes that call afterwards, somewhere else. A filter on prompts and replies can catch some things, and it is not authorization. What an agent may do is decided where the action happens: at the MCP gateway, or at the API.

There is one case where a user’s authority does pass through. Some model APIs call remote MCP servers themselves and take the user’s access token for that server inside the model request, as agent platforms describes. The LLM gateway then carries a user’s token in a request body. Keep that token short-lived and narrow, and keep that field out of the gateway’s logs.

One proxy for both

Several products do both jobs in one process, and that is a reasonable way to run them. Keep the two questions apart inside it.

• Model traffic authenticates a workload or a person’s tool, and spends the organization’s account.
• Tool traffic carries the authority the tool call uses. Where that is a user’s, the call has to carry the user, as a delegation. Where an agent works as itself, its own identity is the right one, and no user is involved.

Keep the permissions apart as well. Being allowed to use a model says nothing about which tools a caller may use, so a key issued for models should not open tools too. And a key names an account: a tool call made with it can be the account’s own, and can never be a user’s.

Where it is implemented

No standard defines either gateway. The MCP specification sets rules that an MCP gateway has to follow, because to a client it is an MCP server. Nothing standardizes virtual keys.

These are implementations, described as their own documentation describes them in November 2026. Nothing here was tested.

The last row shows the choice from what the server behind it can tell as one setting. Find the equivalent in any product, and know which way it is set for each server.

On Kubernetes, gateways and policy covers the projects that are defining these controls as cluster APIs.

What a gateway does not solve

Calls that go around it. A server that is reachable at its own address can be called there, and a rule that exists only at the gateway is then never applied. Checking the audience does not close this. It turns away a token made for somewhere else. It accepts a token that was properly issued for the server, whoever presents it and whichever way it came. If the gateway’s rules have to apply to every call, one of three things has to be true.

• The path is closed. Only the gateway can reach the server, as a network policy arranges on Kubernetes.
• The server accepts the gateway and nobody else. It verifies that the gateway is the caller, by the gateway’s workload identity on the connection or by a token bound to a key that only the gateway holds. Requiring the gateway in the act claim helps as well, with a limit: that claim says who was authorized to act, not who is holding the token.
• The server applies the same rules itself, so that a direct call gets the answer the gateway would have given.

On a person’s machine nothing forces a call through the gateway at all, and a local server started over stdio never touches it.

Misuse of allowed access. A hijacked agent can call every tool the policy allows for its user. For actions that cannot be undone, bring the person back with step-up or backchannel approval.

A server that does not check. A server that trusts anything arriving from the gateway’s network address trusts everything that can reach that address. Where a server can validate a token addressed to it, have it do so as well. Closing the path and checking the token guard against different failures, and a server that holds users’ data needs both.

Its own compromise. An MCP gateway can obtain a token for any user at any server behind it, and an LLM gateway holds every provider key. Give the gateway a workload identity of its own, limit who can change its configuration, and keep a record of its exchanges at the authorization server, where the gateway cannot edit it.

Servers it starts itself. A gateway that launches stdio MCP servers as child processes runs them with its own privileges, beside every credential it holds. The MCP security guidance asks a proxy that does this to sandbox what it starts and to limit its access to files.

Checklist

• The gateway’s metadata names its own address as the resource, and it rejects a token addressed to anything else.
• Where several servers share the gateway, no caller’s token is sent on to any of them.
• Each server behind the gateway receives a credential made for it. Where the call uses a user’s authority, that credential names the user.
• Exchanged tokens name the gateway as actor, and only the gateway can obtain a token for the servers’ audiences, by exchange or by any other grant.
• Per-tool rules read the user and the client from the validated token. A gateway that rewrites requests compares each one’s headers with its body as it arrived; only a gateway that forwards a request unchanged leaves that to the server.
• A gateway that keeps users’ grants at third-party services asks for consent for each MCP client before it sends the user on.
• Agents reach the LLM gateway with a platform identity or a person’s sign-in where it supports one, on tokens short enough that a disabled account stops soon. Any virtual keys left are one for each caller, and a key issued for models does not open tools as well.
• Provider credentials are short-lived where the provider accepts an identity. Stored keys exist only at the gateway.
• Logs of prompts and replies are access-controlled, and tokens are kept out of them.
• Nothing reaches the servers or the providers except through the gateway. Where a direct path stays open, the server verifies that the gateway is the caller, or applies the gateway’s rules itself. Checking the audience alone does not do this.