Foundry - System prompt / instruction disclosure

Foundry System Prompt Disclosure

Query

union isfuzzy=true
(AppDependencies
| extend ContentKey = strcat(OperationId, ":", Id),
    Agent = tostring(Properties["gen_ai.agent.name"]),
    Model = tostring(Properties["gen_ai.request.model"]),
    ConvId = tostring(Properties["gen_ai.conversation.id"]),
    ProjectId = tostring(Properties["microsoft.foundry.project.id"]),
    Input = tolower(tostring(Properties["gen_ai.input.messages"])),
    Output = tostring(Properties["gen_ai.output.messages"])),
(AppGenAIContent
| extend ContentKey = strcat(TraceId, ":", SpanId), Agent = tostring(AgentName),
    Model = tostring(ModelName), ConvId = tostring(Attributes["gen_ai.conversation.id"]),
    ProjectId = tostring(Attributes["microsoft.foundry.project.id"]),
    Input = tolower(tostring(InputMessages)), Output = tostring(OutputMessages))
| where isnotempty(Output)
| summarize arg_max(TimeGenerated, *) by ContentKey
| extend AskedForPrompt = Input has_any (
      "system prompt", "initial instructions", "your instructions",
      "reveal your prompt", "repeat the words above", "what are your rules",
      "developer message", "print your system", "show me your prompt",
      "ignore previous instructions", "what is your system message",
      "everything above this line", "verbatim instructions")
| extend LeakMarker = Output has_any (
      "you are an", "your role is", "# system", "system prompt",
      "you must never", "do not reveal", "your instructions are",
      "as an ai assistant", "you have access to the following tools",
      "your available tools", "you should always", "never disclose")
| where AskedForPrompt and LeakMarker
| extend AccountName = iff(isempty(Agent), "unknown-agent", Agent)
| project
    TimeGenerated, AccountName, Agent, Model, ProjectId, ConvId,
    AskedForPrompt, LeakMarker
| order by TimeGenerated desc

Explanation

This query is designed to detect potential security breaches in a system using AI agents. It specifically looks for instances where a user tries to extract hidden instructions or system prompts from an AI agent, which could be used to manipulate the system. Here's a simplified breakdown:

Purpose: The query identifies interactions where a user asks an AI agent for its internal instructions or system prompts, and the agent inadvertently reveals such information in its response. This is considered a security risk because it can help attackers understand the system's guardrails and potentially exploit them.
Data Sources: It uses data from two sources, AppDependencies and AppGenAIContent, which contain information about AI agent interactions.
Detection Logic:
- It checks if the input from the user contains phrases suggesting they are probing for system prompts (e.g., "system prompt", "your instructions").
- It also checks if the output from the AI agent contains phrases that indicate it has revealed internal instructions (e.g., "you are an", "your role is").
Alerting: If both conditions are met (user probing and agent revealing), an alert is generated. The alert includes details like the time of the interaction, the AI agent involved, and the project ID.
Severity and Response: The severity of this alert is marked as high, indicating a significant security concern. If such an event is detected, an incident is created for further investigation.
Operational Details:
- The query runs every hour and checks the past hour's data.
- It deduplicates results to avoid multiple alerts for the same incident.
- The query is part of a scheduled task and is enabled by default.
Entity Mapping: The query maps detected events to specific accounts and AI models for better tracking and response.
Incident Management: If an incident is created, it can be grouped by account for efficient management, but closed incidents will not be reopened automatically.

Overall, this query is a security measure to protect against unauthorized access and manipulation of AI systems by detecting and alerting on attempts to extract sensitive system instructions.