KQL Search

Query Details

APIM AI Gateway - Tail-latency anomaly against baseline

APIM Latency Baseline Anomaly

Query

let recentWindow=1h;
let baselineWindow=7d;
let Historical=AppRequests
| where TimeGenerated between (ago(baselineWindow) .. ago(recentWindow))
| where SDKVersion startswith "apim:" or tostring(Properties["Service Type"]) =~ "API Management"
| extend APIName=coalesce(tostring(Properties["API Name"]), Name), Operation=coalesce(tostring(Properties["Operation Name"]), Name)
| summarize HourP95=percentile(DurationMs, 95), HourRequests=sum(ItemCount) by APIName, Operation, bin(TimeGenerated, 1h)
| summarize BaselineP95=percentile(HourP95, 50), HistoricalTailP95=percentile(HourP95, 95), BaselineHours=count() by APIName, Operation;
AppRequests
| where TimeGenerated > ago(recentWindow)
| where SDKVersion startswith "apim:" or tostring(Properties["Service Type"]) =~ "API Management"
| extend APIName=coalesce(tostring(Properties["API Name"]), Name), Operation=coalesce(tostring(Properties["Operation Name"]), Name)
| summarize CurrentRequests=sum(ItemCount), CurrentP95=percentile(DurationMs, 95), CurrentP99=percentile(DurationMs, 99), Failures=sum(iff(Success == false, ItemCount, 0)), LastSeen=max(TimeGenerated) by APIName, Operation
| join kind=inner Historical on APIName, Operation
| extend P95Ratio=round(CurrentP95 / max_of(BaselineP95, 1.0), 2), FailureRate=round(100.0 * todouble(Failures) / max_of(todouble(CurrentRequests), 1.0), 2)
| where BaselineHours >= 6 and CurrentRequests >= 10 and CurrentP95 >= 1000 and P95Ratio >= 1.5
| project LastSeen, APIName, Operation, CurrentRequests, CurrentP95, CurrentP99, BaselineP95, HistoricalTailP95, P95Ratio, Failures, FailureRate, BaselineHours
| order by P95Ratio desc

Explanation

This query is designed to identify potential performance anomalies in API operations by comparing recent request durations to historical data. Here's a simplified breakdown:

  1. Data Collection: The query examines API request data from two time periods:

    • Recent Period: The last hour.
    • Baseline Period: The previous seven days, excluding the last hour.

  2. Historical Analysis:

    • It calculates the 95th percentile (p95) of request durations for each API operation on an hourly basis over the baseline period.
    • It then summarizes these hourly p95 values to determine a baseline p95 and a historical tail p95 (95th percentile of the hourly p95s) for each API operation.

  3. Current Analysis:

    • For the recent period, it calculates the current p95 and p99 request durations, total requests, and failures for each API operation.

  4. Comparison and Filtering:

    • It joins the current data with the historical data based on API name and operation.
    • It calculates the ratio of the current p95 to the baseline p95. - It filters the results to highlight operations where:
      • There are at least 6 hours of baseline data.
      • At least 10 requests were made in the recent period.
      • The current p95 is 1000 milliseconds or more.
      • The p95 ratio is 1.5 or higher, indicating a significant increase in latency.

  5. Output:

    • The query outputs relevant details such as the last seen time, API name, operation, request counts, p95 and p99 values, failure counts, and failure rates.
    • Results are ordered by the p95 ratio in descending order to prioritize the most significant anomalies.

This query is useful for detecting potential performance issues in API operations, which can then be further investigated in conjunction with other factors like failures, dependencies, and backend health.