Set up AI Proxy Advanced with Hugging Face - Plugin

This guide walks you through setting up the AI Proxy plugin with Hugging Face.

For all providers, the Kong AI Proxy Advanced plugin attaches to route entities.

It can be installed into one route per operation, for example:

OpenAI chat route
Cohere chat route
Cohere completions route

Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

Prerequisites

Hugging Face account and subscription
You need a service to contain the route for the LLM provider. Create a service first:
```
curl -X POST http://localhost:8001/services \
  --data "name=ai-proxy-advanced" \
  --data "url=http://localhost:32000"
```
Remember that the upstream URL can point anywhere empty, as it won’t be used by the plugin.
Hugging Face access token with permissions to make calls to the Inference API
Text-generation model from Hugging Face

Provider configuration

Set up route and plugin

Create the route:

curl -X POST http://localhost:8001/services/ai-proxy-advanced/routes \
  --data "name=huggingface-chat" \
  --data "paths[]=~/huggingface-chat$"

Enable and configure the AI Proxy Advanced plugin for Hugging Face, replacing the <huggingface_token> with your own access token and <huggingface_model> with the name of the model to use.

routeで有効にする

Kong Admin API

Konnect API

Kubernetes

Declarative (YAML)

Konnect Terraform

次のリクエストを行います。

curl -X POST http://localhost:8001/routes/{routeName|Id}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
  "name": "ai-proxy-advanced",
  "config": {
    "targets": [
      {
        "route_type": "llm/v1/chat",
        "auth": {
          "header_name": "Authorization",
          "header_value": "Bearer <huggingface_token>"
        },
        "model": {
          "provider": "huggingface",
          "name": "<huggingface_model>",
          "options": {
            "max_tokens": 512,
            "temperature": 1.0,
            "top_p": 256,
            "top_k": 0.5
          }
        }
      }
    ]
  }
}
    '

ROUTE_NAME

IDを、このプラグイン構成が対象とするルートのid またはnameに置き換えてください。

独自のアクセストークン、リージョン、コントロールプレーン（CP）ID、ルートIDを代入して、次のリクエストをしてください。

curl -X POST \
https://{us|eu}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/routes/{routeId}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer TOKEN" \
    --data '{"name":"ai-proxy-advanced","config":{"targets":[{"route_type":"llm/v1/chat","auth":{"header_name":"Authorization","header_value":"Bearer <huggingface_token>"},"model":{"provider":"huggingface","name":"<huggingface_model>","options":{"max_tokens":512,"temperature":1.0,"top_p":256,"top_k":0.5}}}]}}'

地域固有のURLと個人アクセストークンの詳細については、 Konnect API referenceをご参照ください。

まず、KongPlugin リソースを作成します：

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-proxy-advanced-example
plugin: ai-proxy-advanced
config:
  targets:
  - route_type: llm/v1/chat
    auth:
      header_name: Authorization
      header_value: Bearer <huggingface_token>
    model:
      provider: huggingface
      name: "<huggingface_model>"
      options:
        max_tokens: 512
        temperature: 1.0
        top_p: 256
        top_k: 0.5
" | kubectl apply -f -

次に、次のようにingressに注釈を付けて、KongPluginリソースをイングレスに適用します。

kubectl annotate ingress INGRESS_NAME konghq.com/plugins=ai-proxy-advanced-example

INGRESS_NAMEを、このプラグイン構成がターゲットとするイングレス名に置き換えます。 kubectl get ingressを実行すると、利用可能なイングレスを確認できます。

注： KongPluginリソースは一度だけ定義するだけで、ネームスペース内の任意のサービス、コンシューマー、またはルートに適用できます。プラグインをクラスター全体で利用可能にしたい場合は、KongPluginの代わりにKongClusterPluginとしてリソースを作成してください。

このセクションを宣言型構成ファイルに追加します。

plugins:
- name: ai-proxy-advanced
  route: ROUTE_NAME|ID
  config:
    targets:
    - route_type: llm/v1/chat
      auth:
        header_name: Authorization
        header_value: Bearer <huggingface_token>
      model:
        provider: huggingface
        name: "<huggingface_model>"
        options:
          max_tokens: 512
          temperature: 1.0
          top_p: 256
          top_k: 0.5

ROUTE_NAME

IDを、このプラグイン構成が対象とするルートのid またはnameに置き換えてください。

前提条件： パーソナルアクセストークンの設定

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "kpat_YOUR_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Kong Konnectゲートウェイプラグインを作成するには、Terraform 構成に以下を追加します。

resource "konnect_gateway_plugin_ai_proxy_advanced" "my_ai_proxy_advanced" {
  enabled = true

  config = {
    targets = [
      {
        route_type = "llm/v1/chat"

        auth = {
          header_name = "Authorization"
          header_value = "Bearer <huggingface_token>"
        }

        model = {
          provider = "huggingface"
          name = "<huggingface_model>"
          options = {
            max_tokens = 512
            temperature = 1.0
            top_p = 256
            top_k = 0.5
          }
        }
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  route = {
    id = konnect_gateway_route.my_route.id
  }
}

Test the configuration

Make an llm/v1/chat type request to test your new endpoint:

curl -X POST http://localhost:8000/huggingface-chat \
  -H 'Content-Type: application/json' \
  --data-raw '{ "messages": [ { "role": "system", "content": "You are a mathematician" }, { "role": "user", "content": "What is 1+1?"} ] }'

前へ Set up AI Proxy Advanced with Gemini

次へ Set up AI Proxy Advanced with Llama2