KQL Search

Query Details

Custom Agents - High recorded model-token usage in a trace

Custom Token Usage Spike

Query

let Evidence=materialize(CustomAgentInlineContent(1h));
CustomAgentSpans(1h)
| where Classification !in ('Excluded managed platform', 'Unattributed GenAI') and HasLineage
| where Operation == 'chat'
| summarize RecordedTokens=sum(InputTokens + OutputTokens), arg_max(EventTime, *)
    by _ResourceId, OperationId, AgentKey
| where RecordedTokens >= 100000
| extend EventCount=RecordedTokens, Signal='At least 100000 recorded chat-span tokens; not a billing estimate'
| extend TimeGenerated=EventTime, ResourceId=_ResourceId, TraceId=OperationId,
    EventCount=tolong(column_ifexists('EventCount', 1)),
    SuccessfulCalls=tolong(column_ifexists('SuccessfulCalls', 0)),
    FailedCalls=tolong(column_ifexists('FailedCalls', 0))
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls,
    TriggerSpanId=tostring(column_ifexists('TriggerSpanId', '')),
    TriggerTime=todatetime(column_ifexists('TriggerTime', datetime(null))),
    TriggerTool=tostring(column_ifexists('TriggerTool', '')),
    TriggerToolResult=tostring(column_ifexists('TriggerToolResult', '')),
    TriggerServerDisplay=tostring(column_ifexists('TriggerServerDisplay', '')),
    TriggerTargetHost=tostring(column_ifexists('TriggerTargetHost', '')),
    RiskyOutputKind=tostring(column_ifexists('RiskyOutputKind', '')),
    RelatedToolResults=tolong(column_ifexists('RelatedToolResults', 0)),
    MatchedIntentPrompt=tostring(column_ifexists('IntentPrompt', '')),
    MatchedIntentSpanId=tostring(column_ifexists('IntentSpanId', ''))
| join kind=leftouter (Evidence
    | project ResourceId=_ResourceId, TraceId=OperationId, SpanId, AgentKey, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState)
    on ResourceId, TraceId, SpanId, AgentKey
| extend Prompt=coalesce(MatchedIntentPrompt, Prompt),
    PromptEvidenceOrigin=iff(isnotempty(MatchedIntentPrompt), 'Matched intent span: user role', PromptEvidenceOrigin),
    PromptEvidenceSpanId=coalesce(MatchedIntentSpanId, PromptEvidenceSpanId)
| extend PreviewCharacters=toint(min_of(160, max_of(0, (1500 - string_size(tostring(bag_pack(
    'AgentId', AgentId, 'AgentKey', AgentKey, 'TraceId', TraceId, 'SpanId', SpanId,
    'Conversation', Conversation, 'ResourceId', ResourceId, 'Signal', Signal)))) / 12)))
| extend PromptPreview=substring(Prompt, 0, PreviewCharacters),
    OutputPreview=substring(Output, 0, PreviewCharacters)
| extend EvidencePreviewStatus=case(PreviewCharacters == 0, 'Metadata exhausted preview budget; inspect full query results',
    strlen(Prompt) > PreviewCharacters or strlen(Output) > PreviewCharacters, 'Inline evidence; previews shortened; full text in query results',
    isempty(Prompt) and isempty(Output), 'No prompt/output captured or parsed; not a protection verdict',
    'Inline evidence; full recorded text in query results')
| project TimeGenerated, ResourceId, Agent, AgentId, AgentKey, AppRoleName,
    TraceId, SpanId, Conversation, Model, Tool, TargetHost, ServerDisplay,
    Signal, EventCount, SuccessfulCalls, FailedCalls, Prompt, Output, ToolArguments, ToolResult, InputMessages, OutputMessages,
    EvidenceSource, PromptEvidenceOrigin, OutputEvidenceOrigin, PromptEvidenceSpanId, OutputEvidenceSpanId,
    InputParseState, OutputParseState, ToolArgumentsState, ToolResultState,
    PromptPreview, OutputPreview, EvidencePreviewStatus, TriggerSpanId, TriggerTime, TriggerTool, TriggerToolResult,
    TriggerServerDisplay, TriggerTargetHost, RiskyOutputKind, RelatedToolResults
| order by TimeGenerated desc

Explanation

This query is designed to monitor and alert on high usage of model tokens in chat operations within a specific application environment. Here's a simplified breakdown of what it does:

  1. Purpose: The query identifies instances where a large number of tokens (at least 100,000) are recorded in chat operations within a trace. This is useful for detecting unusually high usage of resources, which might indicate inefficiencies or potential issues.

  2. Data Sources: It uses data from Application Insights, specifically looking at application requests and dependencies.

  3. Frequency and Duration: The query runs every hour and looks back over the past 14 days to identify relevant data.

  4. Conditions: It filters out certain classifications and focuses only on chat operations. It sums the input and output tokens for each operation and checks if they meet or exceed the 100,000-token threshold.

  5. Output: The query generates a detailed report that includes various attributes such as the resource ID, agent details, trace ID, and a preview of the prompt and output. It also provides a status on the evidence preview, indicating whether the full text is available or if only a shortened version is shown.

  6. Alerting: If the conditions are met, it triggers an alert. The alert is configured to create an incident in the monitoring system, with specific grouping settings to manage how alerts are aggregated and incidents are reopened.

  7. Customization: The query includes custom details and mappings to help identify and categorize the data effectively.

  8. Status: The query is currently disabled (enabled: false), meaning it won't run until activated.

Overall, this query is part of a monitoring strategy to ensure efficient use of resources in AI-driven applications, specifically focusing on chat operations that might consume a significant amount of tokens.