Skip to main content
Version: 3.17

ai-request-rewrite

Description#

The ai-request-rewrite Plugin processes client requests by forwarding them to LLM services for transformation before relaying them to Upstream services. This enables LLM-powered modifications such as data redaction, content enrichment, or reformatting. The Plugin supports integration with OpenAI, DeepSeek, Gemini, Vertex AI, Anthropic, OpenRouter, and other OpenAI-compatible APIs.

Plugin Attributes#

NameTypeRequiredDefaultValid valuesDescription
promptstringTrueThe prompt to send to the LLM service for rewriting the client request.
providerstringTrue[openai, deepseek, azure-openai, aimlapi, gemini, vertex-ai, anthropic, openrouter, openai-compatible]LLM service provider. When set to aimlapi, the Plugin uses the OpenAI-compatible driver and proxies the request to https://api.aimlapi.com/v1/chat/completions. When set to openai-compatible, the Plugin proxies requests to the custom endpoint configured in override. When set to azure-openai, the Plugin also proxies requests to the custom endpoint configured in override and additionally omits the model parameter from the request body sent to Azure OpenAI.
authobjectTrueAuthentication configurations.
auth.headerobjectFalseAuthentication headers. Key must match pattern ^[a-zA-Z0-9._-]+$. At least one of header and query should be configured.
auth.queryobjectFalseAuthentication query parameters. Key must match pattern ^[a-zA-Z0-9._-]+$. At least one of header and query should be configured.
optionsobjectFalseModel configurations. In addition to model, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI, you can configure additional parameters such as temperature, top_p, and stream. See your LLM provider's API documentation for more available options.
options.modelstringFalseName of the LLM model, such as gpt-4 or gpt-3.5. See your LLM provider's API documentation for more available models.
overrideobjectFalseOverride setting.
override.endpointstringFalseLLM provider endpoint. Required when provider is openai-compatible.
timeoutintegerFalse300001 - 60000Request timeout in milliseconds when requesting the LLM service.
keepalivebooleanFalsetrueIf true, keep the connection alive when requesting the LLM service.
keepalive_timeoutintegerFalse60000≥ 1000Keepalive timeout in milliseconds for requests to the LLM service.
keepalive_poolintegerFalse30≥ 1Keepalive pool size for connections to the LLM service.
ssl_verifybooleanFalsetrueIf true, verify the LLM service's SSL certificate.

How It Works#

Examples#

The examples below demonstrate how you can configure ai-request-rewrite for different scenarios.

The examples use OpenAI as the LLM service. To follow along, obtain an OpenAI API key and save it to an environment variable:

export OPENAI_API_KEY=<your-api-key>
note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

Redact Sensitive Information#

The following example demonstrates how to use the ai-request-rewrite Plugin to redact sensitive information before the request reaches the Upstream service.

Send a POST request to the Route with some personally identifiable information:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "content": "John said his debit card number is 4111 1111 1111 1111 and SIN is 123-45-6789."
  }'

You should receive a response similar to the following:

{
  "args": {},
  "data": "{\"content\": \"John said his debit card number is **** **** **** 1111 and SIN is ***-**-***.\"}",
  ...,
  "json": {
    "content": "John said his debit card number is **** **** **** 1111 and SIN is ***-**-***."
  },
  "method": "POST",
  "origin": "192.168.97.1, 103.97.2.170",
  "url": "http://127.0.0.1/anything"
}

Reformat Data#

The following example demonstrates how to use the ai-request-rewrite Plugin to reformat data before the request reaches the Upstream service.

Send a POST request to the Route:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Book a flight from NYC to LA on April 10, 2022."
  }'

You should receive a response similar to the following:

{
  "args": {},
  "data": "{\n  \"intent\": \"BookFlight\",\n  \"parameters\": {\n    \"origin\": \"NYC\",\n    \"destination\": \"LA\",\n    \"date\": \"2022-04-10\"\n  }\n}",
  ...,
  "json": {
    "intent": "BookFlight",
    "parameters": {
      "date": "2022-04-10",
      "destination": "LA",
      "origin": "NYC"
    }
  },
  "method": "POST",
  "origin": "192.168.97.1, 103.97.2.167",
  "url": "http://127.0.0.1/anything"
}

Summarize Information#

The following example demonstrates how to use the ai-request-rewrite Plugin to summarize information before the request reaches the Upstream service.

Send a POST request to the Route with lengthy content:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Hey! So, I'\''m planning a trip to Japan next spring for about three weeks, and I want to visit Tokyo, Kyoto, and Osaka, but I'\''m not sure how to split my time between them. I really love history and cultural sites, so temples and shrines are a must. I'\''m also a big foodie, especially into ramen and sushi, so I'\''d love recommendations on the best spots. I prefer quieter areas for accommodation, but I don'\''t mind traveling into busy areas for sightseeing. Oh, and I'\''d also like to do a day trip somewhere outside these cities—maybe Hakone or Nara? I heard the cherry blossoms might still be in bloom in early April, so I'\''d love to catch them if possible. Also, what'\''s the best way to get around—should I get a JR Pass, or would individual tickets be better? Thanks!"
  }'

You should receive a response similar to the following:

{
  "args": {},
  "data": "The individual is planning a three-week trip to Japan in the spring, looking to visit Tokyo, Kyoto, and Osaka. They are interested in history, culture, temples, and shrines. They love ramen and sushi, so are seeking food recommendations. Accommodation should be in quieter areas, but they are open to busy sites for sightseeing. Along with these cities, they plan to make a day trip to either Hakone or Nara, hoping to see the cherry blossoms in early April. The best transport method between buying the JR Pass or individual tickets is also a query.",
  ...,
  "method": "POST",
  "origin": "192.168.97.1, 103.97.2.171",
  "url": "http://127.0.0.1/anything"
}

Send Request to an OpenAI-Compatible LLM#

The following example demonstrates how to use the ai-request-rewrite Plugin with an OpenAI-compatible LLM provider by setting provider to openai-compatible and configuring the custom endpoint in override.endpoint.