このページは、まだ日本語ではご利用いただけません。翻訳中です。
旧バージョンのドキュメントを参照しています。 最新のドキュメントはこちらをご参照ください。
AI Gateway
The AIGateway
CRD is an opinionated CRD to simplify getting started with Kong’s AI capabilities.
AIGateway
allows you to configure largeLanguageModels
and will translate the configuration in to Gateway
, HTTPRoute
and KongPlugin
resources automatically.
Before you begin, ensure that you have installed the Kong Gateway Operator in your Kubernetes cluster with AI Gateway support enabled.
Before you begin, ensure that you have installed the Kong Gateway Operator in your Kubernetes cluster with AI Gateway support enabled.
Prerequisites
The AIGateway
feature is an alpha release, and needs additional CRDs installed:
kubectl apply -f https://docs.jp.konghq.com/assets/gateway-operator/ai-gateway-crd.yaml --server-side
Install Kong Gateway Operator
Update the Helm repository:
helm repo add kong https://charts.konghq.com
helm repo update kong
Install Kong Gateway Operator with Helm:
helm upgrade --install kgo kong/gateway-operator -n kong-system --create-namespace --set image.tag=1.2 \
--set env.ENABLE_CONTROLLER_AIGATEWAY=true
You can wait for the operator to be ready using kubectl wait
:
kubectl -n kong-system wait --for=condition=Available=true --timeout=120s deployment/kgo-gateway-operator-controller-manager
Get Started
Before using AIGateway
, you need to provide API credentials for your AI providers. AIGateway
supports the following providers:
openai
azure
cohere
mistral
echo '
---
apiVersion: v1
kind: Secret
metadata:
name: acme-ai-cloud-providers
type: Opaque
stringData:
openai: "<INSERT TOKEN HERE>"
' | kubectl apply -f -
After providing authentication credentials, create a GatewayClass
and AIGateway
resource. The AIGateway
resource contains a list of largeLanguageModels
. Each of these will be exposed as individual HTTPRoute
s based on the identifier
field.
echo '
---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: kong-ai-gateways
spec:
controllerName: konghq.com/gateway-operator
---
apiVersion: gateway-operator.konghq.com/v1alpha1
kind: AIGateway
metadata:
name: kong-aigateway
spec:
gatewayClassName: kong-ai-gateways
largeLanguageModels:
cloudHosted:
- identifier: marketing-team-classic-chatgpt
model: gpt-3.5-turbo-instruct
promptType: completions
aiCloudProvider:
name: openai
- identifier: devteam-chatgpt
model: gpt-4
promptType: chat
defaultPrompts:
- role: system
content: "You are a helpful assistant who responds in the style of Sherlock Holmes."
defaultPromptParams:
maxTokens: 50 # shorter responses
aiCloudProvider:
name: openai
cloudProviderCredentials:
name: acme-ai-cloud-providers
' | kubectl apply -f -
Kong Gateway Operator converts the AIGateway
definition in to a Gateway
and multiple HTTPRoute
definitions. The creation of a Gateway
results in a ControlPlane
and a DataPlane
being deployed to handle traffic.
Call the API
Once the ControlPlane
and DataPlane
pods are running, you can call the API.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
dataplane-kong-aigateway-8w9v2-hb7dn-7c4bdf74d4-lsqsv 1/1 Running 0 12m
controlplane-kong-aigateway-4mtd8-dzlmz-589bfb8fbd-8lrgc 1/1 Running 0 12m
To call the API, fetch the PROXY_IP
for the Gateway
:
export PROXY_IP=$(kubectl get gateway kong-aigateway -o jsonpath='{.status.addresses[0].value}')
Finally, make a curl
request to one of the identifier
paths that you defined. If you used the above example, try devteam-chatgpt
:
curl $PROXY_IP/devteam-chatgpt -H 'Content-Type: application/json' -X POST -d '{
"messages": [
{
"role": "user",
"content": "What is the theory of relativity?"
}
]
}'
For more information about how to use the AI plugins, see the plugin hub.