このページは、まだ日本語ではご利用いただけません。翻訳中です。
AI Gateway
The AIGateway
CRD is an opinionated CRD to simplify getting started with Kong’s AI capabilities.
AIGateway
allows you to configure largeLanguageModels
and will translate the configuration in to Gateway
, HTTPRoute
and KongPlugin
resources automatically.
Prerequisites: Install the Kong Gateway Operator in your Kubernetes cluster with AI Gateway support enabled.
Prerequisites
The AIGateway
feature is an alpha release, and needs additional CRDs installed:
kubectl apply -f https://docs.jp.konghq.com/assets/gateway-operator/ai-gateway-crd.yaml --server-side
Install Kong Gateway Operator
Update the Helm repository:
helm repo add kong https://charts.konghq.com
helm repo update kong
Install Kong Gateway Operator with Helm:
helm upgrade --install kgo kong/gateway-operator -n kong-system --create-namespace --set image.tag=1.4 \
--set env.ENABLE_CONTROLLER_AIGATEWAY=true
You can wait for the operator to be ready using kubectl wait
:
kubectl -n kong-system wait --for=condition=Available=true --timeout=120s deployment/kgo-gateway-operator-controller-manager
Get Started
Before using AIGateway
, you need to provide API credentials for your AI providers. AIGateway
supports the following providers:
openai
azure
cohere
mistral
echo '
---
apiVersion: v1
kind: Secret
metadata:
name: acme-ai-cloud-providers
type: Opaque
stringData:
openai: "<INSERT TOKEN HERE>"
' | kubectl apply -f -
After providing authentication credentials, create a GatewayClass
and AIGateway
resource. The AIGateway
resource contains a list of largeLanguageModels
. Each of these will be exposed as individual HTTPRoute
s based on the identifier
field.
echo '
---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: kong-ai-gateways
spec:
controllerName: konghq.com/gateway-operator
---
apiVersion: gateway-operator.konghq.com/v1alpha1
kind: AIGateway
metadata:
name: kong-aigateway
spec:
gatewayClassName: kong-ai-gateways
largeLanguageModels:
cloudHosted:
- identifier: marketing-team-classic-chatgpt
model: gpt-3.5-turbo-instruct
promptType: completions
aiCloudProvider:
name: openai
- identifier: devteam-chatgpt
model: gpt-4
promptType: chat
defaultPrompts:
- role: system
content: "You are a helpful assistant who responds in the style of Sherlock Holmes."
defaultPromptParams:
maxTokens: 50 # shorter responses
aiCloudProvider:
name: openai
cloudProviderCredentials:
name: acme-ai-cloud-providers
' | kubectl apply -f -
Kong Gateway Operator converts the AIGateway
definition in to a Gateway
and multiple HTTPRoute
definitions. The creation of a Gateway
results in a ControlPlane
and a DataPlane
being deployed to handle traffic.
Call the API
Once the ControlPlane
and DataPlane
pods are running, you can call the API.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
dataplane-kong-aigateway-8w9v2-hb7dn-7c4bdf74d4-lsqsv 1/1 Running 0 12m
controlplane-kong-aigateway-4mtd8-dzlmz-589bfb8fbd-8lrgc 1/1 Running 0 12m
To call the API, fetch the PROXY_IP
for the Gateway
:
export PROXY_IP=$(kubectl get gateway kong-aigateway -o jsonpath='{.status.addresses[0].value}')
Finally, make a curl
request to one of the identifier
paths that you defined. If you used the above example, try devteam-chatgpt
:
curl $PROXY_IP/devteam-chatgpt -H 'Content-Type: application/json' -X POST -d '{
"messages": [
{
"role": "user",
"content": "What is the theory of relativity?"
}
]
}'
For more information about how to use the AI plugins, see the plugin hub.