Custom Agents - Recorded guardrail evaluation unavailable

Custom Guardrail Evaluation Unavailable

Query

let Evidence=materialize(CustomAgentInlineContent(1h));
CustomAgentSpans(1h)
| where Classification !in ('Excluded managed platform', 'Unattributed GenAI') and HasLineage
| where EvaluationUnavailable
| summarize arg_min(EventTime, *) by _ResourceId, OperationId, AgentKey
| extend Signal='Guardrail evaluation unavailable; no protection verdict'
| extend TimeGenerated=EventTime, ResourceId=_ResourceId, TraceId=OperationId,
    EventCount=tolong(column_ifexists('EventCount', 1)),
    SuccessfulCalls=tolong(column_ifexists('SuccessfulCalls', 0)),
    FailedCalls=tolong(column_ifexists('FailedCalls', 0))
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls,
    TriggerSpanId=tostring(column_ifexists('TriggerSpanId', '')),
    TriggerTime=todatetime(column_ifexists('TriggerTime', datetime(null))),
    TriggerTool=tostring(column_ifexists('TriggerTool', '')),
    TriggerToolResult=tostring(column_ifexists('TriggerToolResult', '')),
    TriggerServerDisplay=tostring(column_ifexists('TriggerServerDisplay', '')),
    TriggerTargetHost=tostring(column_ifexists('TriggerTargetHost', '')),
    RiskyOutputKind=tostring(column_ifexists('RiskyOutputKind', '')),
    RelatedToolResults=tolong(column_ifexists('RelatedToolResults', 0)),
    MatchedIntentPrompt=tostring(column_ifexists('IntentPrompt', '')),
    MatchedIntentSpanId=tostring(column_ifexists('IntentSpanId', ''))
| join kind=leftouter (Evidence
    | project ResourceId=_ResourceId, TraceId=OperationId, SpanId, AgentKey, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState)
    on ResourceId, TraceId, SpanId, AgentKey
| extend Prompt=coalesce(MatchedIntentPrompt, Prompt),
    PromptEvidenceOrigin=iff(isnotempty(MatchedIntentPrompt), 'Matched intent span: user role', PromptEvidenceOrigin),
    PromptEvidenceSpanId=coalesce(MatchedIntentSpanId, PromptEvidenceSpanId)
| extend PreviewCharacters=toint(min_of(160, max_of(0, (1500 - string_size(tostring(bag_pack(
    'AgentId', AgentId, 'AgentKey', AgentKey, 'TraceId', TraceId, 'SpanId', SpanId,
    'Conversation', Conversation, 'ResourceId', ResourceId, 'Signal', Signal)))) / 12)))
| extend PromptPreview=substring(Prompt, 0, PreviewCharacters),
    OutputPreview=substring(Output, 0, PreviewCharacters)
| extend EvidencePreviewStatus=case(PreviewCharacters == 0, 'Metadata exhausted preview budget; inspect full query results',
    strlen(Prompt) > PreviewCharacters or strlen(Output) > PreviewCharacters, 'Inline evidence; previews shortened; full text in query results',
    isempty(Prompt) and isempty(Output), 'No prompt/output captured or parsed; not a protection verdict',
    'Inline evidence; full recorded text in query results')
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState,
    PromptPreview, OutputPreview, EvidencePreviewStatus, TriggerSpanId, TriggerTime, TriggerTool, TriggerToolResult,
    TriggerServerDisplay, TriggerTargetHost, RiskyOutputKind, RelatedToolResults
| order by TimeGenerated desc

Explanation

This query is part of a scheduled alert in Azure Sentinel that monitors for instances where certain controls or evaluations are not being assessed as expected. Here's a simplified breakdown of what the query does:

Purpose: The query identifies and reports situations where specific security controls (referred to as "guardrails") are not evaluated for a resource, agent, or trace. This is considered an operational gap rather than a security pass or failure.
Data Sources: It uses data from Application Insights, specifically looking at application requests and dependencies.
Frequency and Period: The query runs every hour and looks back over the past 14 days.
Logic:
- It first collects evidence of custom agent activity over the past hour.
- It filters out certain classifications and focuses on those with lineage where evaluations are unavailable.
- It summarizes the earliest event for each resource, operation, and agent key combination.
- It extends the data with additional information like event counts and call success/failure metrics.
- It projects (selects) relevant fields for further analysis and joins with additional evidence data to enrich the results.
- It calculates a preview of the prompt and output data, ensuring the preview fits within a certain character limit.
- It provides a status on the evidence preview, indicating if the full text is available or if the preview is shortened.
Output: The results are ordered by the time they were generated, and the query projects a comprehensive set of fields related to the event, including details about the agent, trace, and any associated evidence.
Alert Configuration: If the query finds any results, it triggers an alert. The alert is configured to create an incident in Azure Sentinel, grouping related alerts by agent key and resource ID.
Metadata: The query includes metadata such as severity (low), version, and tags for categorization.

Overall, this query helps identify and alert on potential gaps in security control evaluations, allowing for further investigation and remediation.