AIGatewayRoute
aigateway.envoyproxy.io / v1beta1
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIGatewayRoute
metadata:
name: example
apiVersion
string
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec object
Spec defines the details of the AIGatewayRoute.
hostnames
[]string
Hostnames is a list of hostnames matched against the HTTP Host header to select an AIGatewayRoute
used to process the request. This is equivalent to the Hostnames field in the Gateway API HTTPRouteSpec.
When specified, the generated HTTPRoute will include these hostnames for hostname-based filtering.
See https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteSpec
for the details of the Hostnames field in the Gateway API.
maxItems:
16llmRequestCosts []object
LLMRequestCosts specifies how to capture the cost of the LLM-related request, notably the token usage.
The AI Gateway filter will capture each specified number and store it in the Envoy's dynamic
metadata per HTTP request. The namespaced key is "io.envoy.ai_gateway".
These route-level costs override any global defaults defined in GatewayConfig.Spec.GlobalLLMRequestCosts
for the same metadataKey. If a metadataKey is not defined in either place, no cost is calculated for it.
This allows you to define common cost formulas once at the gateway level (e.g., via GatewayConfig)
and only override them in specific routes when needed (e.g., premium routes with different pricing).
For example, let's say we have the following LLMRequestCosts configuration:
```yaml
llmRequestCosts:
- metadataKey: llm_input_token
type: InputToken
- metadataKey: llm_output_token
type: OutputToken
- metadataKey: llm_total_token
type: TotalToken
- metadataKey: llm_cached_input_token
type: CachedInputToken
- metadataKey: llm_cache_creation_input_token
type: CacheCreationInputToken
```
Then, with the following BackendTrafficPolicy of Envoy Gateway, you can have three
rate limit buckets for each unique x-tenant-id header value. One bucket is for the input token,
the other is for the output token, and the last one is for the total token.
Each bucket will be reduced by the corresponding token usage captured by the AI Gateway filter.
```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: some-example-token-rate-limit
namespace: default
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: usage-rate-limit
rateLimit:
type: Global
global:
rules:
- clientSelectors:
# Do the rate limiting based on the x-tenant-id header.
- headers:
- name: x-tenant-id
type: Distinct
limit:
# Configures the number of "tokens" allowed per hour.
requests: 10000
unit: Hour
cost:
request:
from: Number
# Setting the request cost to zero allows to only check the rate limit budget,
# and not consume the budget on the request path.
number: 0
# This specifies the cost of the response retrieved from the dynamic metadata set by the AI Gateway filter.
# The extracted value will be used to consume the rate limit budget, and subsequent requests will be rate limited
# if the budget is exhausted.
response:
from: Metadata
metadata:
namespace: io.envoy.ai_gateway
key: llm_input_token
- clientSelectors:
- headers:
- name: x-tenant-id
type: Distinct
limit:
requests: 10000
unit: Hour
cost:
request:
from: Number
number: 0
response:
from: Metadata
metadata:
namespace: io.envoy.ai_gateway
key: llm_output_token
- clientSelectors:
- headers:
- name: x-tenant-id
type: Distinct
limit:
requests: 10000
unit: Hour
cost:
request:
from: Number
number: 0
response:
from: Metadata
metadata:
namespace: io.envoy.ai_gateway
key: llm_total_token
```
Note that when multiple AIGatewayRoute resources are attached to the same Gateway, and
different costs are configured for the same metadata key, each route's rule is carried in
the filter configuration with the route identity; the data plane selects the matching rule
per request (by route), so each route can define its own cost for the same metadata key.
maxItems:
36
cel
string
CEL is the CEL expression to calculate the cost of the request.
The CEL expression must return a signed or unsigned integer. If the
return value is negative, it will be error.
The expression can use the following variables:
* model: the model name extracted from the request content. Type: string.
* backend: the backend name in the form of "name.namespace". Type: string.
* input_tokens: the number of input tokens. Type: unsigned integer.
* cached_input_tokens: the number of cached read input tokens. Type: unsigned integer.
* cache_creation_input_tokens: the number of cache creation input tokens. Type: unsigned integer.
* output_tokens: the number of output tokens. Type: unsigned integer.
* total_tokens: the total number of tokens. Type: unsigned integer.
* reasoning_tokens: the number of reasoning tokens. Type: unsigned integer.
For example, the following expressions are valid:
* "model == 'llama' ? input_tokens + output_token * 0.5 : total_tokens"
* "backend == 'foo.default' ? input_tokens + output_tokens : total_tokens"
* "backend == 'bar.default' ? (input_tokens - cached_input_tokens) + cached_input_tokens * 0.1 + cache_creation_input_tokens * 1.25 + output_tokens : total_tokens"
* "input_tokens + output_tokens + total_tokens"
* "input_tokens * output_tokens"
metadataKey
string required
MetadataKey is the key of the metadata to store this cost of the request.
type
string required
Type specifies the type of the request cost. The default is "OutputToken",
and it uses "output token" as the cost. The other types are "InputToken", "TotalToken",
"CachedInputToken", "CacheCreationInputToken", "ReasoningToken", and "CEL".
enum:
OutputToken, InputToken, CachedInputToken, CacheCreationInputToken, TotalToken, ReasoningToken, CELparentRefs []object
ParentRefs are the names of the Gateway resources this AIGatewayRoute is being attached to.
Currently, each reference's Kind must be Gateway.
maxItems:
16
group
string
Group is the group of the referent.
When unspecified, "gateway.networking.k8s.io" is inferred.
To set the core API group (such as for a "Service" kind referent),
Group must be explicitly set to "" (empty string).
Support: Core
pattern:
^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$maxLength:
253
kind
string
Kind is kind of the referent.
There are two kinds of parent resources with "Core" support:
* Gateway (Gateway conformance profile)
* Service (Mesh conformance profile, ClusterIP Services only)
Support for other resources is Implementation-Specific.
pattern:
^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$minLength:
1maxLength:
63
name
string required
Name is the name of the referent.
Support: Core
minLength:
1maxLength:
253
namespace
string
Namespace is the namespace of the referent. When unspecified, this refers
to the local namespace of the Route.
Note that there are specific rules for ParentRefs which cross namespace
boundaries. Cross-namespace references are only valid if they are explicitly
allowed by something in the namespace they are referring to. For example:
Gateway has the AllowedRoutes field, and ReferenceGrant provides a
generic way to enable any other kind of cross-namespace reference.
<gateway:experimental:description>
ParentRefs from a Route to a Service in the same namespace are "producer"
routes, which apply default routing rules to inbound connections from
any namespace to the Service.
ParentRefs from a Route to a Service in a different namespace are
"consumer" routes, and these routing rules are only applied to outbound
connections originating from the same namespace as the Route, for which
the intended destination of the connections are a Service targeted as a
ParentRef of the Route.
</gateway:experimental:description>
Support: Core
pattern:
^[a-z0-9]([-a-z0-9]*[a-z0-9])?$minLength:
1maxLength:
63
port
integer
Port is the network port this Route targets. It can be interpreted
differently based on the type of parent resource.
When the parent resource is a Gateway, this targets all listeners
listening on the specified port that also support this kind of Route(and
select this Route). It's not recommended to set `Port` unless the
networking behaviors specified in a Route must apply to a specific port
as opposed to a listener(s) whose port(s) may be changed. When both Port
and SectionName are specified, the name and port of the selected listener
must match both specified values.
<gateway:experimental:description>
When the parent resource is a Service, this targets a specific port in the
Service spec. When both Port (experimental) and SectionName are specified,
the name and port of the selected port must match both specified values.
</gateway:experimental:description>
Implementations MAY choose to support other parent resources.
Implementations supporting other types of parent resources MUST clearly
document how/if Port is interpreted.
For the purpose of status, an attachment is considered successful as
long as the parent resource accepts it partially. For example, Gateway
listeners can restrict which Routes can attach to them by Route kind,
namespace, or hostname. If 1 of 2 Gateway listeners accept attachment
from the referencing Route, the Route MUST be considered successfully
attached. If no Gateway listeners accept attachment from this Route,
the Route MUST be considered detached from the Gateway.
Support: Extended
format:
int32minimum:
1maximum:
65535
sectionName
string
SectionName is the name of a section within the target resource. In the
following resources, SectionName is interpreted as the following:
* Gateway: Listener name. When both Port (experimental) and SectionName
are specified, the name and port of the selected listener must match
both specified values.
* Service: Port name. When both Port (experimental) and SectionName
are specified, the name and port of the selected listener must match
both specified values.
Implementations MAY choose to support attaching Routes to other resources.
If that is the case, they MUST clearly document how SectionName is
interpreted.
When unspecified (empty string), this will reference the entire resource.
For the purpose of status, an attachment is considered successful if at
least one section in the parent resource accepts it. For example, Gateway
listeners can restrict which Routes can attach to them by Route kind,
namespace, or hostname. If 1 of 2 Gateway listeners accept attachment from
the referencing Route, the Route MUST be considered successfully
attached. If no Gateway listeners accept attachment from this Route, the
Route MUST be considered detached from the Gateway.
Support: Core
pattern:
^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$minLength:
1maxLength:
253rules []object required
Rules is the list of AIGatewayRouteRule that this AIGatewayRoute will match the traffic to.
Each rule is a subset of the HTTPRoute in the Gateway API (https://gateway-api.sigs.k8s.io/api-types/httproute/).
AI Gateway controller will generate a HTTPRoute based on the configuration given here with the additional
modifications to achieve the necessary jobs, notably inserting the AI Gateway filter responsible for
the transformation of the request and response, etc.
In the matching conditions in the AIGatewayRouteRule, `x-ai-eg-model` header is available
if we want to describe the routing behavior based on the model name. The model name is extracted
from the request content before the routing decision.
How multiple rules are matched is the same as the Gateway API. See for the details:
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRoute
At most 15 rules are allowed per AIGatewayRoute, corresponding to the Gateway API's limit on
HTTPRoute.spec.rules (one slot is reserved for a controller-injected catch-all rule). To
configure more rules on the same Gateway, split them across multiple AIGatewayRoute resources.
maxItems:
15backendRefs []object
BackendRefs is the list of backends that this rule will route the traffic to.
Each backend can have a weight that determines the traffic distribution.
The namespace of each backend defaults to the same namespace as the AIGatewayRoute when not specified.
Cross-namespace references are supported by specifying the namespace field.
When a namespace different than the AIGatewayRoute's namespace is specified,
a ReferenceGrant object is required in the referent namespace to allow that
namespace's owner to accept the reference.
BackendRefs can reference either AIServiceBackend resources (default) or InferencePool resources
from the Gateway API Inference Extension. When referencing InferencePool resources:
- Only one InferencePool backend is allowed per rule
- Cannot mix InferencePool with AIServiceBackend references in the same rule
- Fallback behavior is handled by the InferencePool's endpoint picker
For AIServiceBackend references, you can achieve fallback behavior by configuring multiple backends
combined with the BackendTrafficPolicy of Envoy Gateway.
Please refer to https://gateway.envoyproxy.io/docs/tasks/traffic/failover/ as well as
https://gateway.envoyproxy.io/docs/tasks/traffic/retry/.
maxItems:
128bodyMutation object
BodyMutation defines the request body mutation to be applied to this backend.
This allows modification of JSON fields in the request body before sending to the backend.
When both route-level and backend-level BodyMutation are defined,
route-level takes precedence over backend-level for conflicting operations.
This field is ignored when referencing InferencePool resources.
remove
[]string
Remove the given JSON field(s) from the HTTP request body before sending to the backend.
The value of Remove is a list of top-level field names to remove.
Input:
{
"model": "gpt-4",
"service_tier": "default",
"internal_flag": true
}
Config:
remove: ["service_tier", "internal_flag"]
Output:
{
"model": "gpt-4"
}
maxItems:
16set []object
Set overwrites/adds the request body with the given JSON field (name, value)
before sending to the backend. Only top-level fields are currently supported.
Input:
{
"model": "gpt-4",
"service_tier": "default"
}
Config:
set:
- path: "service_tier"
value: "scale"
Output:
{
"model": "gpt-4",
"service_tier": "scale"
}
maxItems:
16
path
string required
Path is the top-level field name to set in the request body.
Examples: "service_tier", "max_tokens", "temperature"
minLength:
1
value
string required
Value is the JSON value to set at the specified field. This can be any valid JSON value:
string, number, boolean, object, array, or null.
The value will be parsed as JSON and inserted at the specified field.
Examples:
- "\"scale\"" (string)
- "42" (number)
- "true" (boolean)
- "{\"key\": \"value\"}" (object)
- "[1, 2, 3]" (array)
- "null" (null)
group
string
Group is the group of the backend resource.
When not specified, defaults to aigateway.envoyproxy.io (AIServiceBackend).
Currently, only "inference.networking.k8s.io" is supported for InferencePool resources.
pattern:
^$|^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$maxLength:
253headerMutation object
HeaderMutation defines the request header mutation to be applied to this backend.
When both route-level and backend-level HeaderMutation are defined,
route-level takes precedence over backend-level for conflicting operations.
This field is ignored when referencing InferencePool resources.
remove
[]string
Remove the given header(s) from the HTTP request before the action. The
value of Remove is a list of HTTP header names. Note that the header
names are case-insensitive (see
https://datatracker.ietf.org/doc/html/rfc2616#section-4.2).
Input:
GET /foo HTTP/1.1
my-header1: foo
my-header2: bar
my-header3: baz
Config:
remove: ["my-header1", "my-header3"]
Output:
GET /foo HTTP/1.1
my-header2: bar
maxItems:
16set []object
Set overwrites/adds the request with the given header (name, value)
before the action.
Input:
GET /foo HTTP/1.1
my-header: foo
Config:
set:
- name: "my-header"
value: "bar"
Output:
GET /foo HTTP/1.1
my-header: bar
maxItems:
16
name
string required
Name is the name of the HTTP Header to be matched. Name matching MUST be
case-insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2).
If multiple entries specify equivalent header names, the first entry with
an equivalent name MUST be considered for a match. Subsequent entries
with an equivalent header name MUST be ignored. Due to the
case-insensitivity of header names, "foo" and "Foo" are considered
equivalent.
pattern:
^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$minLength:
1maxLength:
256
value
string required
Value is the value of HTTP Header to be matched.
<gateway:experimental:description>
Must consist of printable US-ASCII characters, optionally separated
by single tabs or spaces. See: https://tools.ietf.org/html/rfc7230#section-3.2
</gateway:experimental:description>
<gateway:experimental:validation:Pattern=`^[!-~]+([\t ]?[!-~]+)*$`>
minLength:
1maxLength:
4096
kind
string
Kind is the kind of the backend resource.
When not specified, defaults to AIServiceBackend.
Currently, only "InferencePool" is supported when Group is specified.
pattern:
^$|^[a-zA-Z]([-a-zA-Z0-9]*[a-zA-Z0-9])?$maxLength:
63
modelNameOverride
string
Name of the model in the backend. If provided this will override the name provided in the request.
This field is ignored when referencing InferencePool resources.
name
string required
Name is the name of the backend resource.
When Group and Kind are not specified, this refers to an AIServiceBackend.
When Group and Kind are specified, this refers to the resource of the specified type.
minLength:
1
namespace
string
Namespace is the namespace of the backend resource.
When unspecified (or empty string), this refers to the local namespace of the AIGatewayRoute.
Note that when a namespace different than the local namespace is specified,
a ReferenceGrant object is required in the referent namespace to allow that
namespace's owner to accept the reference. See the ReferenceGrant
documentation for details.
pattern:
^[a-z0-9]([-a-z0-9]*[a-z0-9])?$minLength:
1maxLength:
63
priority
integer
Priority is the priority of the backend. This sets the priority on the underlying endpoints.
See: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/priority
Note: This will override the `faillback` property of the underlying Envoy Gateway Backend
This field is ignored when referencing InferencePool resources.
Default is 0.
format:
int32minimum:
0
weight
integer
Weight is the weight of the backend. This is exactly the same as the weight in
the BackendRef in the Gateway API. See for the details:
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.BackendRef
Default is 1.
format:
int32minimum:
0matches []object
Matches is the list of AIGatewayRouteMatch that this rule will match the traffic to.
This is a subset of the HTTPRouteMatch in the Gateway API. See for the details:
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRouteMatch
maxItems:
128headers []object
Headers specifies HTTP request header matchers. See HeaderMatch in the Gateway API for the details:
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPHeaderMatch
maxItems:
16
name
string required
Name is the name of the HTTP Header to be matched. Name matching MUST be
case-insensitive. (See https://tools.ietf.org/html/rfc7230#section-3.2).
If multiple entries specify equivalent header names, only the first
entry with an equivalent name MUST be considered for a match. Subsequent
entries with an equivalent header name MUST be ignored. Due to the
case-insensitivity of header names, "foo" and "Foo" are considered
equivalent.
When a header is repeated in an HTTP request, it is
implementation-specific behavior as to how this is represented.
Generally, proxies should follow the guidance from the RFC:
https://www.rfc-editor.org/rfc/rfc7230.html#section-3.2.2 regarding
processing a repeated header, with special handling for "Set-Cookie".
pattern:
^[A-Za-z0-9!#$%&'*+\-.^_\x60|~]+$minLength:
1maxLength:
256
type
string
Type specifies how to match against the value of the header.
Support: Core (Exact)
Support: Implementation-specific (RegularExpression)
Since RegularExpression HeaderMatchType has implementation-specific
conformance, implementations can support POSIX, PCRE or any other dialects
of regular expressions. Please read the implementation's documentation to
determine the supported dialect.
enum:
Exact, RegularExpression
value
string required
Value is the value of HTTP Header to be matched.
<gateway:experimental:description>
Must consist of printable US-ASCII characters, optionally separated
by single tabs or spaces. See: https://tools.ietf.org/html/rfc7230#section-3.2
</gateway:experimental:description>
<gateway:experimental:validation:Pattern=`^[!-~]+([\t ]?[!-~]+)*$`>
minLength:
1maxLength:
4096
modelsCreatedAt
string
ModelsCreatedAt represents the creation timestamp of the running models serving by the backends,
which will be exported as the field of "Created" in openai-compatible API "/models".
It follows the format of RFC 3339, for example "2024-05-21T10:00:00Z".
This is used only when this rule contains "x-ai-eg-model" in its header matching
where the header value will be recognized as a "model" in "/models" endpoint.
All the matched models will share the same creation time.
Default to the creation timestamp of the AIGatewayRoute if not set.
format:
date-time
modelsOwnedBy
string
ModelsOwnedBy represents the owner of the running models serving by the backends,
which will be exported as the field of "OwnedBy" in openai-compatible API "/models".
This is used only when this rule contains "x-ai-eg-model" in its header matching
where the header value will be recognized as a "model" in "/models" endpoint.
All the matched models will share the same owner.
Default to "Envoy AI Gateway" if not set.
timeouts object
Timeouts defines the timeouts that can be configured for an HTTP request.
If this field is not set, or the timeout.requestTimeout is nil, Envoy AI Gateway defaults to
set 60s for the request timeout as opposed to 15s of the Envoy Gateway's default value.
For streaming responses (like chat completions with stream=true), consider setting
longer timeouts as the response may take time until the completion.
backendRequest
string
BackendRequest specifies a timeout for an individual request from the gateway
to a backend. This covers the time from when the request first starts being
sent from the gateway to when the full response has been received from the backend.
Setting a timeout to the zero duration (e.g. "0s") SHOULD disable the timeout
completely. Implementations that cannot completely disable the timeout MUST
instead interpret the zero duration as the longest possible value to which
the timeout can be set.
An entire client HTTP transaction with a gateway, covered by the Request timeout,
may result in more than one call from the gateway to the destination backend,
for example, if automatic retries are supported.
The value of BackendRequest must be a Gateway API Duration string as defined by
GEP-2257. When this field is unspecified, its behavior is implementation-specific;
when specified, the value of BackendRequest must be no more than the value of the
Request timeout (since the Request timeout encompasses the BackendRequest timeout).
Support: Extended
pattern:
^([0-9]{1,5}(h|m|s|ms)){1,4}$
request
string
Request specifies the maximum duration for a gateway to respond to an HTTP request.
If the gateway has not been able to respond before this deadline is met, the gateway
MUST return a timeout error.
For example, setting the `rules.timeouts.request` field to the value `10s` in an
`HTTPRoute` will cause a timeout if a client request is taking longer than 10 seconds
to complete.
Setting a timeout to the zero duration (e.g. "0s") SHOULD disable the timeout
completely. Implementations that cannot completely disable the timeout MUST
instead interpret the zero duration as the longest possible value to which
the timeout can be set.
This timeout is intended to cover as close to the whole request-response transaction
as possible although an implementation MAY choose to start the timeout after the entire
request stream has been received instead of immediately after the transaction is
initiated by the client.
The value of Request is a Gateway API Duration string as defined by GEP-2257. When this
field is unspecified, request timeout behavior is implementation-specific.
Support: Extended
pattern:
^([0-9]{1,5}(h|m|s|ms)){1,4}$status object
Status defines the status details of the AIGatewayRoute.
conditions []object
Conditions is the list of conditions by the reconciliation result.
Currently, at most one condition is set.
Known .status.conditions.type are: "Accepted", "NotAccepted".
lastTransitionTime
string required
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format:
date-time
message
string required
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength:
32768
observedGeneration
integer
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format:
int64minimum:
0
reason
string required
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
pattern:
^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$minLength:
1maxLength:
1024
status
string required
status of the condition, one of True, False, Unknown.
enum:
True, False, Unknown
type
string required
type of condition in CamelCase or in foo.example.com/CamelCase.
pattern:
^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$maxLength:
316No matches. Try .spec.hostnames for an exact path