A Shift in Detection Methodology
AI has brought about a slew of benefits and problems for the information security space. Whether it’s the rate of vulnerabilities being discovered across all code bases, automated actions by agents, or AI-enabled administrators, we have quite a variety of items to handle.
However, I’ve noticed in particular an interesting uptick in detections stemming from system administrators utilizing AI to perform whatever necessary task. This isn’t necessarily problematic, as script generation and comprehension of output can be a great thing. But, LLMs are increasingly generating scripts and commands that set off EDR systems.
Why?
It turns out, from the efforts of LLMs and administrators in wanting to solve problems, they become particularly interested in verbose outputs that are able to identify a problem or find an answer more specifically. This has resulted in methods that we’ve traditionally flagged as suspicious because previously, only attackers would’ve been using such methods for nefarious reasons.
An example of this is testing a connection to a Microsoft Exchange server. Normally, you could simply use the PowerShell Cmdlet: Connect-ExchangeOnline
But in this case, the LLM generated script output something that was reading a raw stream of data, enabling TLS communication with a certificate, accepting user credentials, and establishing a connection with a remote server.
The inner workings of the PowerShell Cmdlet… more or less do the same thing. But it was the fact that the LLM and the administrator were working outside of the context of commonly established tooling that resulted in the heuristic detection.
As such, with these types of activities becoming more common among administrators, it would behoove us to consider a different approach to the standard detection methodology. One that looks at the actual content of the execution rather than the historic negative view on more verbose engineering methods.
Regardless, an analyst’s perspective will always be necessary at the end of the day to make the judgement call on whether or not an activity is malicious. AI-powered or not.
~barkeep