https://www.heinz.cmu.edu/media/2023/July/generative-ai-is-a-math-problem-left-unchecked-it-could-be-a-real-problem

What is Generative AI?

Disturbingly, many business leaders, and even executives of major AI vendors touting this technology, don’t appear to know the answer to this basic question.

As a result, although Generative AI does a somewhat decent job at what it is designed to do, the uses for which it is widely being promoted have no relation to what it was designed to accomplish.

Mathematically, though, it’s very simple. A Generative AI model is just a probability distribution over human expression.  This model can generate samples from this distribution.  If you understand that, you are already ahead of the vast majority of AI “experts”.

Now it is clear that “hallucination” of plausible sounding but false samples is a feature, not a bug, of Generative AI.  The many, many misguided attempts to “fix” the hallucination problem inevitably fail because the fundamental nature of the model is that hallucinations cannot be eliminated without also destroying the probability information that is the entire purpose of the model.

This also demystifies the dark arts of “prompt engineering”, revealing them as simply a matter of conditional probabilities within human expression - conditional, that is, on the sample starting with the given “prompt” string.  So for example, the improved responses when the prompt includes “please” are not evidence of any desire to help, only replication of the conditional probabilities within the human generated training data.

So what does one use such a probability model for?

By definition this type of model only produces “probable” output, which among other things inherently contains hallucinations. But in most business applications, people would much prefer “factual” output, which is a completely unrelated objective.  Yet this flawed use case is one of the primary ones being pushed by AI “experts” in an attempt to improve business productivity.

Even people who want randomized output will be disappointed. They want “creative” output. But by definition, a Generative AI model will only produce random samples from the very same distribution on which it was trained. The exact sample might never have produced before, but there is nothing novel or creative in it that goes beyond the distribution represented by the training data.

To move forward, there are three options:
1. Respect the math, don’t fight it. Then you may be able to find use cases where the technology makes sense.
2. Go much, much deeper into the math than the basic introduction presented here.  This can open a broader set of use cases for Generative AI.
3. Realize that Generative AI by its nature represents zero progress towards actual intelligence.  It’s just a probabilistic model derived from things that actually intelligent humans have said in the past. Instead, pursue alternative approaches to AI that address the real problem.