Singularity University

Large Language Models do not store facts. They predict them. These systems are probabilistic, not deterministic. They function as giant statistical maps, generating the next most likely word based on patterns across billions of data points. As Tricia Wang puts it, the intelligence in these models is extremely narrow. At their core, LLMs are highly advanced auto-suggest tools.

If an AI has been trained on sensitive data, or simply has access to it in its context window, it can unpredictably surface that information as a probabilistic output. This happens by design, a direct result of how these systems fundamentally work.

AI cannot be trusted to naturally safeguard sensitive data it has been exposed to. Those who don't understand this are making decisions with an incomplete mental model of the technology guiding their organizations.

Too many are already implementing AI without asking the foundational questions first. Before going further, Neil Desai urges executives to answer: what growth do you honestly need to achieve?