Foundry - Tool error and retry storm

Foundry Tool Error Storm

Query

let failThreshold=8;
AppDependencies
| where isnotempty(Properties["gen_ai.tool.name"])
| extend Agent=tostring(Properties["gen_ai.agent.name"]), Model=tostring(Properties["gen_ai.request.model"]), ProjectId=tostring(Properties["microsoft.foundry.project.id"]), ConvId=tostring(Properties["gen_ai.conversation.id"]), ToolName=tolower(tostring(Properties["gen_ai.tool.name"])), Success_=tostring(column_ifexists('Success', '')), ToolResultCode=tostring(column_ifexists('ResultCode', ''))
| extend Failed=Success_ =~ "false" or ToolResultCode startswith "4" or ToolResultCode startswith "5"
| summarize TotalCalls=count(), FailedCalls=countif(Failed), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), ResultCodes=make_set(ToolResultCode, 10), Agent=take_any(Agent), Model=take_any(Model), ProjectId=take_any(ProjectId) by ConvId, ToolName
| extend FailRatio=round(todouble(FailedCalls) / todouble(TotalCalls), 2), DurationMin=datetime_diff('minute', LastSeen, FirstSeen)
| where FailedCalls >= failThreshold and FailRatio >= 0.5
| extend TimeGenerated=LastSeen, AccountName=iff(isempty(Agent), "unknown-agent", Agent), Signal="TOOL ERROR OR RETRY STORM"
| project TimeGenerated, Signal, AccountName, Agent, Model, ProjectId, ConvId, ToolName, FailedCalls, TotalCalls, FailRatio, DurationMin, ResultCodes, FirstSeen, LastSeen
| order by FailedCalls desc

Explanation

This query is designed to detect potential issues with a tool in the Foundry system by monitoring for a pattern of errors. Here's a simple breakdown of what it does:

Purpose: The query identifies situations where there are at least eight failed attempts to use the same Foundry tool within a single conversation over the past hour. Additionally, these failures must account for at least 50% of the total attempts to use the tool in that conversation.
Potential Issues Detected: Such patterns could indicate problems like malformed-argument fuzzing, capability probing, or a runaway retry loop. These are signs of availability issues or potential abuse, but not necessarily malicious intent.
Data Source: It uses data from Application Insights, specifically focusing on application dependencies.
Severity: The issue is flagged with a medium severity level.
Frequency and Period: The query runs every hour and looks at data from the past hour.
Alert Trigger: An alert is triggered if there is at least one instance meeting the criteria.
Output: The query outputs details such as the time of the last error, the tool name, the number of failed and total calls, the failure ratio, and other related information.
Entity Mapping: It maps entities like accounts and cloud applications to provide context in alerts.
Incident Management: If an issue is detected, an incident is created, and similar incidents can be grouped together for better management.
Tags and Version: The query is tagged for categorization and is versioned as 1.0.0. Overall, this query helps in monitoring and managing the reliability and potential misuse of tools within the Foundry system.