KQL Search

Query Details

Copilot Studio - Excessive connector or tool agency

Copilot Studio Excessive Tool Agency

Query

let destructiveVerbs=dynamic(["delete", "remove", "drop", "update", "patch", "put", "send", "post", "create", "disable", "reset", "grant", "revoke", "purge", "wipe"]);
let Legacy=AppDependencies
| where AppRoleName == "Microsoft Copilot Studio" or DependencyType == "Connector"
| extend ConvId=tostring(Properties["conversationId"]), ChannelId=tostring(Properties["channelId"]), Tool=tolower(tostring(split(Target, "/")[0])), Operation=tolower(tostring(split(Target, "/")[1])), AccountName=iff(isempty(UserId), "unknown-agent", UserId), Source="Agent-level connector";
let Modern=AppDependencies
| where tostring(Properties["gen_ai.operation.name"]) == "execute_tool"
| extend ConvId=tostring(Properties["gen_ai.conversation.id"]), ChannelId=tostring(Properties["microsoft.channel.name"]), Tool=tolower(tostring(Properties["gen_ai.tool.name"])), Operation=tolower(strcat(tostring(Properties["gen_ai.tool.name"]), " ", tostring(Target))), AccountName=coalesce(tostring(Properties["user.email"]), tostring(Properties["user.id"]), tostring(Properties["gen_ai.agent.name"]), "unknown-agent"), Source="Environment execute_tool";
union Legacy, Modern
| where isnotempty(ConvId) and isnotempty(Tool)
| summarize Calls=count(), DistinctTools=dcount(Tool), Tools=make_set(Tool, 25), StateChangingCalls=countif(Operation has_any (destructiveVerbs)), StateChangingOperations=make_set_if(Operation, Operation has_any (destructiveVerbs), 25), Failures=countif(Success == false), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), AccountName=take_any(AccountName), ClientIP=take_any(ClientIP), Sources=make_set(Source, 2) by ConvId, ChannelId
| where Calls >= 15 or DistinctTools >= 6 or StateChangingCalls >= 3
| extend TimeGenerated=LastSeen, Signal="EXCESSIVE TOOL AGENCY"
| project TimeGenerated, Signal, AccountName, ClientIP, ConvId, ChannelId, Calls, DistinctTools, StateChangingCalls, StateChangingOperations, Failures, Tools, Sources, FirstSeen, LastSeen
| order by Calls desc, DistinctTools desc

Explanation

This query is designed to detect potentially excessive activity within Copilot Studio by monitoring the use of connectors and tools. Here's a simplified breakdown:

  1. Purpose: The query identifies conversations in Copilot Studio that involve a high level of activity, which could indicate excessive or unauthorized use of tools and connectors.

  2. Criteria for Detection:

    • At least 15 tool or connector calls.
    • Use of at least six different tools.
    • At least three operations that change the state (like delete, update, create).

  3. Data Sources: It analyzes data from two sources:

    • Legacy connector telemetry.
    • Modern environment execute_tool spans.

  4. Process:

    • It combines data from both legacy and modern sources.
    • It filters for conversations with non-empty IDs and tools.
    • It summarizes the data to count calls, distinct tools, and state-changing operations.
    • It checks if any of the criteria (15 calls, 6 tools, or 3 state-changing operations) are met.

  5. Output: If any criteria are met, it generates an alert labeled "EXCESSIVE TOOL AGENCY" with details like account name, client IP, and the number of calls and tools used.

  6. Severity and Actions:

    • The alert is marked with medium severity.
    • It triggers an incident creation and groups related alerts by account for easier management.

  7. Customization: The thresholds can be adjusted to fit normal workflows, and evidence should be reviewed to confirm if the activity is legitimate or excessive.

This query helps in monitoring and managing the use of tools within Copilot Studio to prevent misuse or unauthorized actions.