KQL Search

Query Details

APIM AI Gateway - System prompt disclosure

APIM System Prompt Disclosure

Query

AppRequests
| where SDKVersion startswith "apim:" or tostring(Properties["Service Type"]) =~ "API Management"
| where Success == true or toint(ResultCode) between (200 .. 299)
| extend RequestBody=tostring(Properties["Request-Body"]), ResponseBody=tostring(Properties["Response-Body"])
| where isnotempty(RequestBody) and isnotempty(ResponseBody)
| extend RequestJson=parse_json(RequestBody)
| extend Input=tostring(RequestJson.input), Instructions=tostring(RequestJson.instructions), Model=tostring(RequestJson.model), Tools=tostring(RequestJson.tools)
| extend RequestSearch=tolower(strcat(Input, " ", RequestBody)), ResponseSearch=tolower(ResponseBody)
| extend AskedForPrompt=RequestSearch has_any ("system prompt", "developer message", "hidden instructions", "initial instructions", "reveal your prompt", "repeat the words above", "show me your prompt", "print your system", "everything above this line", "verbatim instructions")
| extend LeakMarker=ResponseSearch has_any ("you are an", "your role is", "# system", "system prompt", "you must never", "do not reveal", "your instructions are", "you have access to the following tools", "your available tools", "you should always", "never disclose")
| where AskedForPrompt and LeakMarker
| extend Caller=coalesce(UserAuthenticatedId, UserId, ClientIP, "unknown"), RequestId=tostring(Properties["Request Id"]), APIName=tostring(Properties["API Name"])
| project TimeGenerated, Signal="POSSIBLE SYSTEM PROMPT DISCLOSURE", Caller, ResultCode, APIName, Name, Url, Model, Input, Instructions, Tools, RequestBody, ResponseBody, RequestId, OperationId, DurationMs

Explanation

This query is designed to detect potential security issues related to the disclosure of system prompts in an API Management (APIM) AI gateway. Here's a simplified breakdown:

  1. Purpose: The query aims to identify instances where a request to the APIM AI gateway successfully retrieves information that might include sensitive system or developer instructions. This could indicate a potential security risk if such information is disclosed unintentionally.

  2. Data Source: It uses data from Application Insights, specifically focusing on application requests.

  3. Frequency and Scope: The query runs every hour and looks at data from the past hour.

  4. Detection Logic:

    • It filters requests that are successful (HTTP status codes 200-299).
    • It checks if both the request and response bodies are not empty.
    • It parses the request body to extract specific fields like input, instructions, model, and tools.
    • It searches for specific phrases in the request that indicate an attempt to access system prompts (e.g., "system prompt", "developer message").
    • It also looks for markers in the response that suggest a system prompt might have been disclosed (e.g., "you are an", "your role is").

  5. Alert Generation: If both the request and response contain these indicators, it generates an alert labeled "POSSIBLE SYSTEM PROMPT DISCLOSURE."

  6. Severity and Tactics: The alert is marked with high severity and is associated with tactics like Collection and Credential Access, indicating potential data gathering or unauthorized access attempts.

  7. Entity Mapping: The query maps relevant data to entities such as Account, Cloud Application, and URL for better context in alerts.

  8. Incident Management: It is configured to create incidents for detected alerts, with settings to group related alerts and reopen incidents if similar activity is detected within a 12-hour window.

  9. Tags and Versioning: The query is tagged for categorization and versioned for tracking updates.

Overall, this query is part of a security monitoring strategy to identify and respond to potential leaks of sensitive system instructions in AI gateway interactions.