Microsoft Recall, the new AI feature where a little AI will peek at your Windows monitor all the time, record what you are doing, and analyze it, is creepy, totally violates privacy and confidentiality, and is probably the worst feature Microsoft has dreamt up so far.
It also opens up some really interesting possibilities. If you can trick the AI that is looking at the screen into doing certain things based on, for example, an avatar or an image on a web page, that could mean even static web pages and completely anonymous instant messengers could potentially become attack vectors against Windows machines.
We have seen this kind of thinking from Microsoft before. In 2001, Internet Explorer had a vulnerability in its MIME handling where simply viewing malicious content could execute arbitrary code. You didn't have to deliberately run an attachment. A malicious web page, HTML email or newsgroup posting could be enough. Because other Microsoft applications used Internet Explorer to render HTML, the vulnerability extended beyond the browser itself.
Something that is supposedly safe, like an image or text file, can be used to exploit a system if whatever consumes it is buggy or contains vulnerabilities. A famous Microsoft example was the 2004 GDI+ JPEG vulnerability, CVE-2004-0200. A specially crafted JPEG could cause a buffer overrun while being processed by Microsoft's JPEG handling code, potentially allowing arbitrary code execution. The nasty part was its reach: applications using the vulnerable GDI+ component could become attack vectors simply by processing an image. JPEG decompressors are pretty complex and difficult to implement correctly and securely. An AI system like Recall has a vastly larger and more complex processing stack. If something on a screen, text or images, can trigger a zero-day exploit, either in a library used (for example, to convert the image before the AI interprets it) or can present itself as a prompt to the AI, then really bad things could happen. If Recall is used on millions of PCs, expect clever hackers and intelligence agencies to find ways to exploit it.
I'm really looking forward to seeing a proof of concept using the Recall feature by delivering something that would be completely harmless on any sane operating system.
The Banner image was generated by ChatGPT.