Set up AI Proxy Advanced with Mistral

このページは、まだ日本語ではご利用いただけません。翻訳中です。

This guide walks you through setting up the AI Proxy Advanced plugin with Mistral.

Mistral is a self-hosted model. As such, it requires setting model option upstream_url to point to the absolute HTTP(S) endpoint for this model implementation.

There are a number of hosting/format options for running this LLM. Mistral offers a cloud-hosted service for consuming the LLM, available at Mistral.ai

Self-hosted options include:

• OLLAMA
• Hosting the GGUF model yourself, with an e.g. Python Web Server

Upstream Formats

The “upstream” request and response formats are different between various implementations of Mistral, and/or its accompanying web-server.

For this provider, the following should be used for the config.model.options.mistral_format parameter:

OpenAI Format

The openai format option follows the same upstream formats as the equivalent OpenAI route type operation (that is, llm/v1/chat or llm/v1/completions).

This format should be used when configuring the cloud-based https://mistral.ai/.

It will transform both llm/v1/chat and llm/v1/completions type route requests, into the same format.

Ollama Format

The ollama format option adheres to the chat and chat-completion request formats, as defined in its API documentation.

Using the plugin with Mistral

For all providers, the Kong AI Proxy Advanced plugin attaches to route entities.

It can be installed into one route per operation, for example:

• OpenAI chat route
• Cohere chat route
• Cohere completions route

Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

Prerequisites

You need a service to contain the route for the LLM provider. Create a service first:

curl -X POST http://localhost:8001/services \
  --data "name=ai-proxy-advanced" \
  --data "url=http://localhost:32000"

Remember that the upstream URL can point anywhere empty, as it won’t be used by the plugin.

Set up route and plugin

After installing and starting your Mistral instance, you can then create an AI Proxy Advanced route and plugin configuration.

Create the route:

curl -X POST http://localhost:8001/services/ai-proxy-advanced/routes \
  --data "name=mistral-chat" \
  --data "paths[]=~/mistral-chat$"

Enable and configure the AI Proxy Advanced plugin for Mistral (using openai format in this example).

curl -X POST http://localhost:8001/routes/{routeName|Id}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
  "name": "ai-proxy-advanced",
  "config": {
    "targets": [
      {
        "route_type": "llm/v1/chat",
        "auth": {
          "header_name": "Authorization",
          "header_value": "Bearer <MISTRAL_AI_KEY>"
        },
        "model": {
          "provider": "mistral",
          "name": "mistral-tiny",
          "options": {
            "mistral_format": "openai",
            "upstream_url": "https://api.mistral.ai/v1/chat/completions"
          }
        }
      }
    ]
  }
}
    '

curl -X POST \
https://{us|eu}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/routes/{routeId}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer TOKEN" \
    --data '{"name":"ai-proxy-advanced","config":{"targets":[{"route_type":"llm/v1/chat","auth":{"header_name":"Authorization","header_value":"Bearer <MISTRAL_AI_KEY>"},"model":{"provider":"mistral","name":"mistral-tiny","options":{"mistral_format":"openai","upstream_url":"https://api.mistral.ai/v1/chat/completions"}}}]}}'

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-proxy-advanced-example
plugin: ai-proxy-advanced
config:
  targets:
  - route_type: llm/v1/chat
    auth:
      header_name: Authorization
      header_value: Bearer <MISTRAL_AI_KEY>
    model:
      provider: mistral
      name: mistral-tiny
      options:
        mistral_format: openai
        upstream_url: https://api.mistral.ai/v1/chat/completions
" | kubectl apply -f -

kubectl annotate ingress INGRESS_NAME konghq.com/plugins=ai-proxy-advanced-example

plugins:
- name: ai-proxy-advanced
  route: ROUTE_NAME|ID
  config:
    targets:
    - route_type: llm/v1/chat
      auth:
        header_name: Authorization
        header_value: Bearer <MISTRAL_AI_KEY>
      model:
        provider: mistral
        name: mistral-tiny
        options:
          mistral_format: openai
          upstream_url: https://api.mistral.ai/v1/chat/completions

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "kpat_YOUR_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

resource "konnect_gateway_plugin_ai_proxy_advanced" "my_ai_proxy_advanced" {
  enabled = true

  config = {
    targets = [
      {
        route_type = "llm/v1/chat"

        auth = {
          header_name = "Authorization"
          header_value = "Bearer <MISTRAL_AI_KEY>"
        }

        model = {
          provider = "mistral"
          name = "mistral-tiny"
          options = {
            mistral_format = "openai"
            upstream_url = "https://api.mistral.ai/v1/chat/completions"
          }
        }
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  route = {
    id = konnect_gateway_route.my_route.id
  }
}

Test the configuration

Make an llm/v1/chat type request to test your new endpoint:

curl -X POST http://localhost:8000/mistral-chat \
  -H 'Content-Type: application/json' \
  --data-raw '{ "messages": [ { "role": "system", "content": "You are a mathematician" }, { "role": "user", "content": "What is 1+1?"} ] }'