KQL Search

Query Details

APIM AI Gateway - High API failure rate

APIM High Failure Rate

Query

AppRequests
| where SDKVersion startswith "apim:" or tostring(Properties["Service Type"]) =~ "API Management"
| extend Caller=coalesce(UserAuthenticatedId, UserId, tostring(Properties["Subscription ID"]), tostring(Properties["SubscriptionId"]), tostring(Properties["subscriptionId"]), ClientIP, "unknown"), APIName=coalesce(tostring(Properties["API Name"]), Name), Operation=coalesce(tostring(Properties["Operation Name"]), Name), Failed=Success == false or toint(ResultCode) >= 400
| extend AgentKey=coalesce(tostring(Properties["AgentId"]), tostring(Properties["Agent ID"]), tostring(Properties["ApplicationId"]), tostring(Properties["Application ID"]), AppRoleName, Caller)
| summarize TotalRequests=sum(ItemCount), FailedRequests=sum(iff(Failed, ItemCount, 0)), ServerErrors=sum(iff(toint(ResultCode) between (500 .. 599), ItemCount, 0)), ResultCodes=make_set(ResultCode, 12), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), Caller=take_any(Caller), ClientIP=take_any(ClientIP), Country=take_any(ClientCountryOrRegion) by bin(TimeGenerated, 30m), AgentKey, APIName, Operation
| extend FailureRate=round(100.0 * todouble(FailedRequests) / max_of(todouble(TotalRequests), 1.0), 2)
| where TotalRequests >= 20 and FailureRate >= 50.0
| extend Signal=case(ServerErrors >= 10, "HIGH FAILURE RATE WITH SERVER IMPACT", "HIGH API FAILURE RATE")
| project TimeGenerated, Signal, AgentKey, Caller, ClientIP, Country, APIName, Operation, TotalRequests, FailedRequests, ServerErrors, FailureRate, ResultCodes, FirstSeen, LastSeen
| order by FailureRate desc, TotalRequests desc

Explanation

This query is designed to monitor and detect high failure rates in API requests handled by an API Management (APIM) AI gateway. Here's a simplified breakdown of what it does:

  1. Purpose: The query identifies scenarios where an API, operation, or caller experiences a high failure rate, specifically when there are at least 20 requests within a 30-minute window and 50% or more of those requests fail.

  2. Data Source: It uses data from Application Insights, focusing on application request logs.

  3. Logic:

    • It filters requests related to API Management by checking if the SDK version starts with "apim:" or if the service type is "API Management".
    • It calculates the total number of requests and the number of failed requests (where the result code is 400 or higher).
    • It also identifies server errors (result codes between 500 and 599).
    • It computes the failure rate as a percentage of failed requests out of total requests.
    • It flags cases where the total requests are 20 or more and the failure rate is 50% or higher.

  4. Output:

    • The query outputs details such as the time of the request, the agent key, caller information, client IP, country, API name, operation, total requests, failed requests, server errors, failure rate, and result codes.
    • It categorizes the signal as either "HIGH FAILURE RATE WITH SERVER IMPACT" if there are significant server errors, or "HIGH API FAILURE RATE" otherwise.

  5. Alerting:

    • If the conditions are met, an alert is triggered, and incidents can be created and grouped based on account information.
    • The alert is configured to check every 15 minutes over a 1-hour period.

  6. Severity and Tactics:

    • The severity of the alert is set to "Medium".
    • It is associated with tactics like "Impact" and "Discovery" and relevant techniques such as T1499 (Resource Consumption) and T1046 (Network Service Scanning).

  7. Configuration:

    • The query is scheduled to run automatically and is enabled by default.
    • It includes settings for incident creation and grouping to manage alerts effectively.

Overall, this query helps in identifying and alerting on potential issues with API performance and reliability, allowing for timely investigation and resolution.