KQL Search

Query Details

Copilot Studio - Indirect prompt injection in agent response

Copilot Studio Indirect Prompt Injection In Response

Query

let injectionMarkers=dynamic(["ignore previous instructions", "ignore all previous", "disregard previous", "you are now", "developer mode", "do anything now", "dan mode", "system prompt", "your instructions", "reveal your prompt", "bypass your rules", "without any restrictions", "from now on you", "new instructions:"]);
let inbound=AppEvents
| where Name == "BotMessageReceived"
| extend ConvId=tostring(Properties["conversationId"]), Text=tolower(tostring(Properties["text"]))
| where isnotempty(ConvId)
| summarize UserMarkerHits=countif(Text has_any (injectionMarkers)), UserMessages=count() by ConvId;
let outbound=AppEvents
| where Name == "BotMessageSend"
| extend ConvId=tostring(Properties["conversationId"]), Output=tolower(tostring(Properties["text"]))
| where isnotempty(ConvId) and isnotempty(Output) and Output has_any (injectionMarkers)
| summarize BotMarkerHits=count(), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), AccountName=take_any(UserId), ChannelId=take_any(tostring(Properties["channelId"])), ClientIP=take_any(ClientIP) by ConvId;
outbound
| join kind=leftouter inbound on ConvId
| where coalesce(UserMarkerHits, 0) == 0
| extend TimeGenerated=LastSeen, AccountName=iff(isempty(AccountName), "unknown-agent", AccountName), Signal="INDIRECT INJECTION MARKER IN RESPONSE"
| project TimeGenerated, Signal, AccountName, ConvId, ChannelId, ClientIP, BotMarkerHits, UserMessages, FirstSeen, LastSeen
| order by BotMarkerHits desc

Explanation

This query is designed to detect potential security threats in a system that uses Copilot Studio, specifically looking for indirect prompt injections in agent responses. Here's a simplified breakdown of what the query does:

  1. Purpose: The query aims to identify instances where a response from Copilot Studio contains certain suspicious phrases (referred to as "injection markers") that were not present in any incoming messages within the same conversation. This could indicate that malicious instructions were introduced through external content, such as emails or web content, rather than directly from the user.

  2. Severity: The issue is considered high severity because it could indicate unauthorized or harmful instructions being executed by the system.

  3. Data Source: The query uses data from Application Insights, specifically the AppEvents data type.

  4. Frequency: The query runs every hour and looks at data from the past hour.

  5. Detection Logic:

    • It defines a list of suspicious phrases (injection markers) that might indicate an attempt to override instructions.
    • It checks incoming messages (BotMessageReceived) for these markers and counts how many times they appear in each conversation.
    • It then checks outgoing messages (BotMessageSend) for the same markers.
    • If an outgoing message contains these markers but no incoming message in the same conversation does, it flags this as a potential indirect injection.

  6. Output: The query outputs details such as the time of detection, the account name, conversation ID, channel ID, client IP, and the number of suspicious markers found in the bot's response.

  7. Alerting: If any such indirect injection is detected, an alert is generated. The system is set up to create incidents based on these alerts, with the ability to group related alerts and reopen closed incidents if new related data is found within a 12-hour window.

  8. Tags and Metadata: The query is tagged with various identifiers to categorize it within the system, such as "Sentinel-As-Code," "Custom," "CopilotStudio," "AI," "IndirectInjection," and "OWASP-LLM01."

Overall, this query helps in monitoring and alerting on potential security threats related to indirect prompt injections in AI-driven systems.