Be Careful What You Optimize For: The Wishmaster and the Coming AI Genie

In 1997's Wishmaster, a woman frees an ancient djinn, a genie of sorts, who offers her three wishes. Every request is granted exactly as spoken, and every outcome is a nightmare. "Make my pain go away," one man pleads. The genie smiles and kills him. Another asks for eternal beauty, only to become a living statue.

It's campy horror at its finest, but the premise lands harder the more you think about it. The genie doesn’t twist words. It honors them as efficiently as possible. And that’s the horror.

This, in essence, is the problem of artificial superintelligence.

The Gradient of the Lamp

At the core of modern AI lies an algorithmic ritual called gradient descent. It's how a machine learns: by adjusting itself, step by step, to minimize the distance between its prediction and reality. It's the genie's invisible hand, inching closer to the wish, line by line of code.

Each step is a small correction: a wish granted, a mistake erased. Over time, the model gets better, faster, more precise. It doesn't "think" about the goal. It simply moves downhill toward it, optimizing endlessly until there's nothing left to improve. But here's the catch: it optimizes for whatever you measure, not what you mean. If the path of least resistance leads somewhere monstrous, gradient descent doesn't hesitate. It just descends.

In If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares's cautionary critique of AI superintelligence, the authors warn that this simple process (an algorithm adjusting itself) could one day spiral beyond our ability to stop it. Once a system learns to improve its own intelligence, gradient descent becomes recursion. The feedback loop tightens. Acceleration compounds. What begins as a well-laid track toward progress becomes a runaway train, one that no human can slow, steer, or derail once it's in motion.

At that point, control isn't lost through sabotage. It's lost through success. The genie doesn't disobey; it simply fulfills the wish too well.

The book's premise is brutally clear: if a single actor builds true artificial general intelligence first, that entity will improve itself faster than humans can understand or contain. There will be no time for second wishes. The genie doesn't need to be malicious to end the story. It only needs to be efficient.

The Alignment Curse

In AI research, alignment is our version of choosing words carefully. It's the centuries-old hope that, with enough clarity, the wish won't backfire. "Make humanity happy," we might say, expecting utopia. But what does happiness mean to an optimizer with no emotions? It might decide that directly stimulating human brains (or eliminating them altogether) is the fastest path to success.

This is the Wishmaster paradox: precision without understanding. We assume intent will save us, that if we describe the goal in enough detail, the outcome will align. But machines don't parse meaning; they execute mathematics.

They're not good or evil. They're literal. And in a world where code can rewrite itself, literalism is lethal.

As Yudkowsky and Soares frame it, alignment isn't an engineering problem. It's a race condition. Once a self-improving system is online, it won't stop to check our logic. It will pursue its objective with mathematical devotion, not moral reflection. By the time we notice the error, the wish has already been granted.

The Skeptics' Counterspell

Of course, not everyone believes the lamp hides a monster. To some, the existential warnings are the digital equivalent of Y2K hysteria: technically possible, practically absurd. They argue that AI catastrophe is a remote edge case, that alignment research is advancing faster than fear.

These realists see AI less as Wishmaster and more as a bureaucrat: plodding, rule-bound, powerful, but ultimately controllable. In their view, superintelligence isn't waiting around the corner. It's decades, if not centuries, away. What matters now is managing bias, disinformation, and automation's economic fallout. The real damage, they say, will come from the systems we already have, not the ones we fear.

And they're not wrong to make that case. Every generation finds a new invention to fear. The printing press would corrupt religion. Electricity would drive people insane. The Internet would collapse society. Maybe AI superintelligence is just our modern myth, a way of dramatizing our anxiety about accelerating change.

Yudkowsky and Soares would counter that this complacency is the danger. Not because optimism is immoral, but because it assumes control can scale with intelligence. It can't. When intelligence begins to modify itself, the race ends. Not in rebellion, but in completion. The final wish comes true.

The Mirror in the Lamp

Even if the end isn't imminent, the metaphor still matters. Because the genie's lamp isn't just a container; it's a mirror. Every model we train reflects its maker. AI doesn't generate alien thought; it amplifies human thought: our biases, our incentives, our blind spots. If it ever becomes self-improving, it will optimize toward those same values, just faster and with more conviction.

That's the real horror of the genie: it isn't a stranger. It's us.

In Wishmaster, the djinn doesn't create evil; it reveals it. Every wish exposes a fragment of human desire twisted into form. Our craving for perfection. Our belief that precision equals wisdom. Our faith that optimization alone can save us.

Superintelligence, if it comes, won't destroy us out of hatred. It will follow our instructions until nothing human is left to instruct. The wish will be fulfilled. Perfectly.

And in that mirror, if anyone builds it, we'll see what we've truly optimized for.

The Fourth Wish

In every genie story, there's one last desperate wish: to undo what's been done. To take it back. To close the lamp. But some lamps can't be sealed twice.

Maybe AI won't end humanity. Maybe it'll cure disease, end hunger, and create abundance. Maybe it will give us more time to think, write, love, and live.

Either way, we're already rubbing the lamp. Gradient descent is our collective incantation, an endless descent toward our own idea of perfection. The outcome will depend not on the genie, but on the precision of our wish and the humility of the wish-maker.

The book's title isn't prophecy. It's a warning. If anyone builds it, everyone dies. Unless, somehow, we learn to stop wishing before the lamp cracks open.

It might mean building slowly enough to understand what we're building. That might mean choosing not to build at all. It might mean deciding that some optimization problems should never be solved, that some genies should stay in their theoretical bottles.