KQL Search

Query Details

Foundry - Unsafe active content in agent output

Foundry Unsafe Active Content In Output

Query

union isfuzzy=true
(AppDependencies
| extend ContentKey=strcat(OperationId, ":", Id), Agent=tostring(Properties["gen_ai.agent.name"]), Model=tostring(Properties["gen_ai.request.model"]), ConvId=tostring(Properties["gen_ai.conversation.id"]), ProjectId=tostring(Properties["microsoft.foundry.project.id"]), Output=tostring(Properties["gen_ai.output.messages"])),
(AppGenAIContent
| extend ContentKey=strcat(TraceId, ":", SpanId), Agent=tostring(AgentName), Model=tostring(ModelName), ConvId=tostring(Attributes["gen_ai.conversation.id"]), ProjectId=tostring(Attributes["microsoft.foundry.project.id"]), Output=tostring(OutputMessages))
| where isnotempty(Output)
| summarize arg_max(TimeGenerated, *) by ContentKey
| extend ActiveContent=extract(@"(?i)(</?script\b|<(?:iframe|embed|object)\b|on(?:click|load|error|mouse\w*|change|input)\s*=|(?:href|src)\s*=\s*[""']?javascript\s*:|data\s*:\s*(?:text/html|application/javascript|image/svg\+xml)|&#(?:60|62|x3[cCeEfF]);)", 0, Output)
| where isnotempty(ActiveContent)
| extend AccountName=iff(isempty(Agent), "unknown-agent", Agent)
| project TimeGenerated, Signal="UNSAFE ACTIVE CONTENT EMITTED", ActiveContent, AccountName, Agent, Model, ProjectId, ConvId, ContentKey
| order by TimeGenerated desc

Explanation

This query is designed to detect potentially unsafe content in the output of a Foundry agent. It looks for active HTML elements, scripts, event handlers, executable URIs, or encoded tags that could be harmful if executed by a browser or other renderer. The query checks data from two sources, AppDependencies and AppGenAIContent, and combines them to identify any outputs containing these unsafe elements.

Key points of the query:

  • It runs every hour and checks data from the past hour.
  • It triggers an alert if any unsafe content is found.
  • The severity of the alert is set to "Medium."
  • It uses specific techniques related to execution and defense evasion (T1059 and T1027).
  • The query extracts and flags any active content found in the agent's output.
  • It organizes the results by the time the content was generated and includes details like the agent name, model, project ID, and conversation ID.
  • Alerts are grouped by account and can reopen closed incidents if similar issues are detected within a 12-hour window.

Overall, this query helps monitor and alert on potentially dangerous content generated by AI agents, ensuring that such content is identified and addressed promptly.