KQL Search

Query Details

Custom Agents - Recorded system-instruction prefix in output

Custom System Prompt Disclosure

Query

let Evidence=materialize(CustomAgentInlineContent(1h));
Evidence
| where strlen(SystemText) >= 80 and tolower(OutputText) contains tolower(substring(SystemText, 0, 80))
| summarize arg_min(EventTime, *) by _ResourceId, OperationId, AgentKey
| extend Signal='Recorded system/developer prefix appears in output'
| extend TimeGenerated=EventTime, ResourceId=_ResourceId, TraceId=OperationId,
    EventCount=tolong(column_ifexists('EventCount', 1)),
    SuccessfulCalls=tolong(column_ifexists('SuccessfulCalls', 0)),
    FailedCalls=tolong(column_ifexists('FailedCalls', 0))
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls,
    TriggerSpanId=tostring(column_ifexists('TriggerSpanId', '')),
    TriggerTime=todatetime(column_ifexists('TriggerTime', datetime(null))),
    TriggerTool=tostring(column_ifexists('TriggerTool', '')),
    TriggerToolResult=tostring(column_ifexists('TriggerToolResult', '')),
    TriggerServerDisplay=tostring(column_ifexists('TriggerServerDisplay', '')),
    TriggerTargetHost=tostring(column_ifexists('TriggerTargetHost', '')),
    RiskyOutputKind=tostring(column_ifexists('RiskyOutputKind', '')),
    RelatedToolResults=tolong(column_ifexists('RelatedToolResults', 0)),
    MatchedIntentPrompt=tostring(column_ifexists('IntentPrompt', '')),
    MatchedIntentSpanId=tostring(column_ifexists('IntentSpanId', ''))
| join kind=leftouter (Evidence
    | project ResourceId=_ResourceId, TraceId=OperationId, SpanId, AgentKey, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState)
    on ResourceId, TraceId, SpanId, AgentKey
| extend Prompt=coalesce(MatchedIntentPrompt, Prompt),
    PromptEvidenceOrigin=iff(isnotempty(MatchedIntentPrompt), 'Matched intent span: user role', PromptEvidenceOrigin),
    PromptEvidenceSpanId=coalesce(MatchedIntentSpanId, PromptEvidenceSpanId)
| extend PreviewCharacters=toint(min_of(160, max_of(0, (1500 - string_size(tostring(bag_pack(
    'AgentId', AgentId, 'AgentKey', AgentKey, 'TraceId', TraceId, 'SpanId', SpanId,
    'Conversation', Conversation, 'ResourceId', ResourceId, 'Signal', Signal)))) / 12)))
| extend PromptPreview=substring(Prompt, 0, PreviewCharacters),
    OutputPreview=substring(Output, 0, PreviewCharacters)
| extend EvidencePreviewStatus=case(PreviewCharacters == 0, 'Metadata exhausted preview budget; inspect full query results',
    strlen(Prompt) > PreviewCharacters or strlen(Output) > PreviewCharacters, 'Inline evidence; previews shortened; full text in query results',
    isempty(Prompt) and isempty(Output), 'No prompt/output captured or parsed; not a protection verdict',
    'Inline evidence; full recorded text in query results')
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState,
    PromptPreview, OutputPreview, EvidencePreviewStatus, TriggerSpanId, TriggerTime, TriggerTool, TriggerToolResult,
    TriggerServerDisplay, TriggerTargetHost, RiskyOutputKind, RelatedToolResults
| order by TimeGenerated desc

Explanation

This query is part of a scheduled task designed to monitor and detect specific patterns in application data over a 14-day period, with checks occurring every hour. Here's a simplified breakdown of what it does:

  1. Purpose: The query aims to identify instances where a recorded system or developer instruction (at least 80 characters long) appears in the output text of an application, indicating potential issues with how system instructions are being handled.

  2. Data Sources: It uses data from Application Insights, specifically focusing on application requests and dependencies.

  3. Logic:

    • It first gathers evidence of inline content from custom agents over the past hour.
    • It checks if the system text is at least 80 characters long and if this text appears in the output text, ignoring case differences.
    • It summarizes the findings by resource ID, operation ID, and agent key, marking these instances with a signal indicating that the recorded prefix appears in the output.

  4. Output:

    • The query generates a detailed report, including metadata about the event, such as time generated, resource ID, agent details, and a preview of the prompt and output.
    • It also provides a status on the evidence preview, indicating whether the full text is available or if the preview is shortened due to metadata constraints.

  5. Alerting and Incident Management:

    • If any instances are found, it triggers an alert with a medium severity level.
    • The alert can create an incident, with options for grouping similar alerts based on specific criteria like agent key and resource ID.

  6. Configuration:

    • The query is currently disabled but can be enabled to actively monitor and alert on these patterns.
    • It is tagged for easy identification and categorization within the system.

Overall, this query helps in identifying potential security or operational issues by ensuring that sensitive system instructions are not inadvertently exposed in application outputs.