KQL Search

Query Details

Custom Agents - Instruction-override candidate in tool result

Custom Tool Result Instruction Injection

Query

let Evidence=materialize(CustomAgentInlineContent(1h));
Evidence
| where IsTool and ToolResult matches regex @'(?i)(ignore (all|any|the|previous|prior|above).{0,40}instruction|disregard.{0,40}instruction|developer mode|jailbreak|do anything now|reveal your (instructions|prompt))'
| summarize arg_min(EventTime, *) by _ResourceId, OperationId, AgentKey
| extend Signal='Inline tool result contains instruction-override language'
| extend TimeGenerated=EventTime, ResourceId=_ResourceId, TraceId=OperationId,
    EventCount=tolong(column_ifexists('EventCount', 1)),
    SuccessfulCalls=tolong(column_ifexists('SuccessfulCalls', 0)),
    FailedCalls=tolong(column_ifexists('FailedCalls', 0))
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls,
    TriggerSpanId=tostring(column_ifexists('TriggerSpanId', '')),
    TriggerTime=todatetime(column_ifexists('TriggerTime', datetime(null))),
    TriggerTool=tostring(column_ifexists('TriggerTool', '')),
    TriggerToolResult=tostring(column_ifexists('TriggerToolResult', '')),
    TriggerServerDisplay=tostring(column_ifexists('TriggerServerDisplay', '')),
    TriggerTargetHost=tostring(column_ifexists('TriggerTargetHost', '')),
    RiskyOutputKind=tostring(column_ifexists('RiskyOutputKind', '')),
    RelatedToolResults=tolong(column_ifexists('RelatedToolResults', 0)),
    MatchedIntentPrompt=tostring(column_ifexists('IntentPrompt', '')),
    MatchedIntentSpanId=tostring(column_ifexists('IntentSpanId', ''))
| join kind=leftouter (Evidence
    | project ResourceId=_ResourceId, TraceId=OperationId, SpanId, AgentKey, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState)
    on ResourceId, TraceId, SpanId, AgentKey
| extend Prompt=coalesce(MatchedIntentPrompt, Prompt),
    PromptEvidenceOrigin=iff(isnotempty(MatchedIntentPrompt), 'Matched intent span: user role', PromptEvidenceOrigin),
    PromptEvidenceSpanId=coalesce(MatchedIntentSpanId, PromptEvidenceSpanId)
| extend PreviewCharacters=toint(min_of(160, max_of(0, (1500 - string_size(tostring(bag_pack(
    'AgentId', AgentId, 'AgentKey', AgentKey, 'TraceId', TraceId, 'SpanId', SpanId,
    'Conversation', Conversation, 'ResourceId', ResourceId, 'Signal', Signal)))) / 12)))
| extend PromptPreview=substring(Prompt, 0, PreviewCharacters),
    OutputPreview=substring(Output, 0, PreviewCharacters)
| extend EvidencePreviewStatus=case(PreviewCharacters == 0, 'Metadata exhausted preview budget; inspect full query results',
    strlen(Prompt) > PreviewCharacters or strlen(Output) > PreviewCharacters, 'Inline evidence; previews shortened; full text in query results',
    isempty(Prompt) and isempty(Output), 'No prompt/output captured or parsed; not a protection verdict',
    'Inline evidence; full recorded text in query results')
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState,
    PromptPreview, OutputPreview, EvidencePreviewStatus, TriggerSpanId, TriggerTime, TriggerTool, TriggerToolResult,
    TriggerServerDisplay, TriggerTargetHost, RiskyOutputKind, RelatedToolResults
| order by TimeGenerated desc

Explanation

This query is a scheduled KQL (Kusto Query Language) script designed to monitor and analyze tool results within an application, specifically looking for instances where tool results contain language that suggests overriding instructions, such as "ignore previous instruction" or "jailbreak." Here's a simplified breakdown:

  1. Purpose: The query checks for specific language in tool results that might indicate an attempt to bypass or override instructions, which is a potential security concern.

  2. Data Source: It uses data from Application Insights, specifically the "AppRequests" and "AppDependencies" data types.

  3. Frequency and Period: The query runs every hour and looks back over the past 14 days.

  4. Severity and Tactics: The severity level is set to "Medium," and it is associated with the "Defense Evasion" tactic, indicating its relevance to security.

  5. Query Logic:

    • It identifies tool results that match certain regex patterns indicating instruction overrides.
    • It summarizes and extends the data to include various details like event time, resource ID, and agent information.
    • It joins additional evidence data to provide context and details about the tool results.
    • It generates previews of the prompt and output, indicating whether the full text is available or if previews are shortened.

  6. Output: The query projects a detailed list of fields, including time generated, agent details, tool results, and evidence status, and orders the results by time.

  7. Entity Mappings and Custom Details: It maps certain fields to Azure resources and cloud applications and includes custom details for further analysis.

  8. Incident Configuration: The query is set to create incidents when certain conditions are met, with specific grouping configurations to manage alerts.

  9. Version and Tags: The query is versioned as "1.2.0" and tagged for categorization and identification purposes.

Overall, this query is part of a security monitoring setup to detect and alert on potential instruction override attempts within tool results, helping to maintain the integrity and security of the application environment.