KQL Search

Query Details

Detecting Non Latin Characters With KQL

Query

let foreignCharPattern = @"[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}\p{Arabic}\p{Hebrew}\p{Devanagari}\p{Bengali}\p{Tamil}\p{Thai}]";

About this query

Detecting Non-Latin Characters with KQL

Query Information

I recently discovered how easy it is to detect characters from non-Latin writing systems using Unicode script categories:

A simple regular expression can identify scripts such as Chinese, Japanese, Korean, Arabic, Hebrew, and several Indic scripts. A small but useful insight for text analysis and data quality checks.

Explanation

This KQL query is designed to identify email subjects that mention file-sharing platforms like Teams or SharePoint and contain characters from non-Latin scripts. Here's a simple breakdown:

  1. Define Non-Latin Character Pattern: The query starts by defining a pattern (foreignCharPattern) that includes characters from various non-Latin scripts such as Chinese, Japanese, Korean, Arabic, Hebrew, and several Indic scripts.

  2. Specify Platforms of Interest: It sets up a list (SharingPlatform) containing the names of file-sharing platforms like "Teams" and "SharePoint".

  3. Filter Email Events: The query searches through the EmailEvents table for emails received in the last 30 days.

  4. Check for Platform Mentions: It filters these emails to find those whose subjects mention any of the specified platforms.

  5. Detect Non-Latin Characters: It further filters these subjects to find those containing at least one character from the non-Latin scripts defined earlier.

  6. Extract Foreign Characters: For subjects that meet these criteria, it extracts all the non-Latin characters found and lists them in a new column called ForeignChars.

This query helps identify potentially suspicious emails that mix Latin and non-Latin scripts, which could be indicative of phishing attempts or other security concerns. It can be customized by changing the platforms, time range, or scripts of interest.