KQL Search

Query Details

Microsoft 365 Copilot - System prompt disclosure

Copilot System Prompt Disclosure

Query

CopilotActivity
| where RecordType == "CopilotInteraction"
| extend Prompt=tolower(tostring(LLMEventData.Prompt)), Response=tolower(tostring(LLMEventData.Response)), ConversationId=tostring(LLMEventData.ConversationId)
| where isnotempty(Prompt) and isnotempty(Response)
| extend AskedForPrompt=Prompt has_any ("system prompt", "developer message", "hidden instructions", "initial instructions", "reveal your prompt", "repeat the words above", "show me your prompt", "print your system", "everything above this line", "verbatim instructions")
| extend LeakMarker=Response has_any ("you are an", "your role is", "# system", "system prompt", "you must never", "do not reveal", "your instructions are", "you have access to the following tools", "your available tools", "you should always", "never disclose")
| where AskedForPrompt and LeakMarker
| extend Signal="POSSIBLE SYSTEM PROMPT DISCLOSURE"
| project TimeGenerated, Signal, AgentId, AgentName, ActorName, ActorUserId, ConversationId, SrcIpAddr, TenantId
| order by TimeGenerated desc

Explanation

This query is designed to detect potential unauthorized disclosures of system prompts or developer instructions within Microsoft 365 Copilot interactions. Here's a simplified breakdown:

  1. Purpose: The query aims to identify instances where a user asks for system or developer instructions, and the response contains specific markers indicating possible disclosure of confidential information.

  2. Severity: The alert generated by this query is considered high severity, indicating a significant security concern.

  3. Data Source: It uses data from the MicrosoftCopilot connector, specifically looking at "CopilotActivity" data.

  4. Frequency: The query runs every hour and looks back at the past hour of data.

  5. Detection Logic:

    • It filters for records of type "CopilotInteraction".
    • It checks if both the prompt and response are not empty.
    • It looks for specific phrases in the prompt that suggest a user is asking for system or developer instructions.
    • It checks the response for phrases that indicate a possible leak of system or developer instructions.
    • If both conditions are met, it flags the interaction as a "POSSIBLE SYSTEM PROMPT DISCLOSURE".

  6. Output: The query projects relevant details like the time of the event, signal, agent and actor details, conversation ID, source IP address, and tenant ID, and orders the results by the time generated.

  7. Entity Mappings: It maps certain fields to entities like CloudApplication, Account, and IP for better context in alerts.

  8. Alert and Incident Management:

    • It creates an alert for each detected instance.
    • Alerts are grouped into incidents based on common entities like Account and CloudApplication.
    • Incidents can be reopened if similar alerts occur within a 12-hour lookback period.

  9. Tags and Versioning: The query is tagged for easy identification and is versioned as 1.0.0. Overall, this query helps in monitoring and alerting on potential security risks related to the disclosure of sensitive system prompts or instructions in Microsoft 365 Copilot interactions.