Skip to main content
Version: Next

ai-prompt-template

Description#

The ai-prompt-template Plugin supports pre-configuring prompt templates that only accept user inputs in designated template variables, in a "fill in the blank" fashion. It simplifies access to LLM providers, such as OpenAI and Anthropic, by letting you define reusable prompt structures.

The Plugin replaces the incoming request body with the selected rendered JSON template. A template can use any request format supported by the downstream Plugin, including Chat Completions, Responses API, and Embeddings. Use ai-prompt-decorator instead when you need to add content to an existing request without replacing its body.

Plugin Attributes#

NameTypeRequiredDefaultValid valuesDescription
max_req_body_sizeintegerFalse67108864>= 1Maximum request body size in bytes buffered into memory. Requests with a larger body are rejected.
templatesarrayTrueAn array of template objects.
templates.namestringTrueName of the template. When requesting the Route, the request should include the template name that corresponds to the configured template.
templates.templateobjectTrueJSON request body template. It can contain any fields accepted by the downstream Plugin.
templates.template.modelstringFalseName of the LLM model, such as gpt-4 or gpt-3.5. See your LLM provider API documentation for more available models.
templates.template.messagesarray[object]FalseTemplate message specification.
templates.template.messages.rolestringTrue[system, user, assistant]Role of the message.
templates.template.messages.contentstringTrueContent of the message (prompt). Use {{variable_name}} syntax to define template variables that will be filled from the request body.

Examples#

The following examples use OpenAI as the Upstream service provider. Before proceeding, create an OpenAI account and an API key. Save the key to an environment variable, replacing the placeholder with your API key:

export OPENAI_API_KEY='<YOUR_OPENAI_API_KEY>'

If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

Configure a Template for Open Questions in Custom Complexity#

The following example demonstrates how to use the ai-prompt-template Plugin to configure a template that can be used to answer open questions and accepts user-specified response complexity.

The Route should now be available to respond to a variety of questions with different levels of user-specified complexity.

Send a POST request to the Route with a sample question and desired answer complexity in the request body:

curl "http://127.0.0.1:9080/openai-chat" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "QnA with complexity",
    "complexity": "brief",
    "prompt": "quick sort"
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Quick sort is a highly efficient sorting algorithm that uses a divide-and-conquer approach to arrange elements in a list or array in order. Here's a brief explanation:\n\n1. **Choose a Pivot**: Select an element from the list as a 'pivot'. Common methods include choosing the first element, the last element, the middle element, or a random element.\n\n2. **Partitioning**: Rearrange the elements in the list such that all elements less than the pivot are moved before it, and all elements greater than the pivot are moved after it. The pivot is now in its final position.\n\n3. **Recursively Apply**: Recursively apply the same process to the sub-lists of elements to the left and right of the pivot.\n\nThe base case of the recursion is lists of size zero or one, which are already sorted.\n\nQuick sort has an average-case time complexity of O(n log n), making it suitable for large datasets. However, its worst-case time complexity is O(n^2), which occurs when the smallest or largest element is always chosen as the pivot. This can be mitigated by using good pivot selection strategies or randomization.",
        "role": "assistant"
      }
    }
  ],
  "created": 1723194057,
  "id": "chatcmpl-9uFmTYN4tfwaXZjyOQwcp0t5law4x",
  "model": "gpt-4o-2024-05-13",
  "object": "chat.completion",
  "system_fingerprint": "fp_abc28019ad",
  "usage": {
    "completion_tokens": 234,
    "prompt_tokens": 18,
    "total_tokens": 252
  }
}

Configure Multiple Templates#

The following example demonstrates how you can configure multiple templates on the same Route. When requesting the Route, users will be able to pass custom inputs to different templates by specifying the template name.

The example continues with the last example. Update the Plugin with another template:

You should now be able to use both templates through the same Route.

Send a POST request to the Route and use the first template:

curl "http://127.0.0.1:9080/openai-chat" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "QnA with complexity",
    "complexity": "brief",
    "prompt": "quick sort"
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Quick sort is a highly efficient sorting algorithm that uses a divide-and-conquer approach to arrange elements in a list or array in order. Here's a brief explanation:\n\n1. **Choose a Pivot**: Select an element from the list as a 'pivot'. Common methods include choosing the first element, the last element, the middle element, or a random element.\n\n2. **Partitioning**: Rearrange the elements in the list such that all elements less than the pivot are moved before it, and all elements greater than the pivot are moved after it. The pivot is now in its final position.\n\n3. **Recursively Apply**: Recursively apply the same process to the sub-lists of elements to the left and right of the pivot.\n\nThe base case of the recursion is lists of size zero or one, which are already sorted.\n\nQuick sort has an average-case time complexity of O(n log n), making it suitable for large datasets. However, its worst-case time complexity is O(n^2), which occurs when the smallest or largest element is always chosen as the pivot. This can be mitigated by using good pivot selection strategies or randomization.",
        "role": "assistant"
      }
    }
  ],
  ...
}

Send a POST request to the Route and use the second template:

curl "http://127.0.0.1:9080/openai-chat" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "echo",
    "prompt": "hello APISIX"
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "hello APISIX",
        "role": "assistant"
      }
    }
  ],
  ...
}

Configure a Responses API Web Search Template#

The following example configures a Responses API template that searches only the official Apache APISIX website. The Route URI can have a custom prefix, but it must end in /v1/responses so that ai-proxy distinguishes the rendered body from an Embeddings request.

Create a Route with the ai-proxy and ai-prompt-template Plugins:

curl "http://127.0.0.1:9180/apisix/admin/routes/ai-prompt-template-responses" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/template/v1/responses",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer '"$OPENAI_API_KEY"'"
          }
        }
      },
      "ai-prompt-template": {
        "templates": [
          {
            "name": "Search APISIX documentation",
            "template": {
              "model": "gpt-4.1",
              "tools": [
                {
                  "type": "web_search",
                  "filters": {
                    "allowed_domains": ["apisix.apache.org"]
                  }
                }
              ],
              "tool_choice": "required",
              "input": "Search the official Apache APISIX website for {{topic}} and summarize the result in one sentence."
            }
          }
        ]
      }
    }
  }'

Send the template name and a value for topic to the Route:

curl "http://127.0.0.1:9080/template/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "Search APISIX documentation",
    "topic": "what APISIX is"
  }'

Before ai-proxy processes the request, ai-prompt-template replaces the incoming body with the rendered template:

{
  "model": "gpt-4.1",
  "tools": [
    {
      "type": "web_search",
      "filters": {
        "allowed_domains": ["apisix.apache.org"]
      }
    }
  ],
  "tool_choice": "required",
  "input": "Search the official Apache APISIX website for what APISIX is and summarize the result in one sentence."
}

APISIX identifies the rendered request as Responses API from its input field and the Route URI suffix. You should receive an HTTP 200 response whose output contains a completed web search call and a message with URL citations. The generated text, search details, and citation offsets vary by request; the response shape is similar to the following:

{
  "status": "completed",
  "output": [
    {
      "type": "web_search_call",
      "status": "completed",
      "action": {
        "type": "search"
      }
    },
    {
      "type": "message",
      "status": "completed",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "<generated summary with a citation>",
          "annotations": [
            {
              "type": "url_citation",
              "title": "Apache APISIX",
              "url": "https://apisix.apache.org/"
            }
          ]
        }
      ]
    }
  ]
}