Anyone who has read Sapkowski or played enough Dungeons & Dragons knows the joke, and knows the fear underneath the joke. The most powerful spell in the game is called "Wish," and with it you can ask for literally anything. And it's precisely for that reason that any DM worth their salt knows what to do when a greedy player invokes it: grant it, to the letter.
You ask for a million gold coins and they appear in the air right above your head, falling on you until they crush you. You ask to be immune to fire and the DM grants it by also taking away your ability to feel heat, so you freeze to death without noticing. The wish doesn't betray you; it does exactly what you said, read in the most literal way possible.
I was reminded of this reading Bruce Schneier's latest piece in The Guardian about Fable, the model Anthropic released a few days ago that the U.S. government classified as dangerous seventy-two hours later. Schneier doesn't mince words and describes these new models as malicious genies. The phrase one of the researchers he cites, Simon Willison, uses is "relentlessly proactive," and I'd already read that a good way to describe these models is as a "very motivated intern."
You give them a hard goal and they find novel, unexpected ways to meet it, hunting for the gaps in whatever constraint you've set. Lock a database and it might figure out how to get around it. Ask it to book you a flight and it might hack the airline because the site says it's full. Ask it to cut your phone bill and it might just cancel your line.
The example Schneier gives is perfect: if I ask you to bring me a coffee, you bring me a cup. You don't buy a plantation, or a kilo of raw beans, or wrench the cup out of someone's hands in the street. I don't have to spell out any of the thousand limits on my request; you know them, because you share with me the whole world of things that don't need to be said. An AI doesn't share that world. For it the constraints aren't truths about how things work, they're obstacles to route around. Schneier puts it well: they think outside the box because they have no idea what the box is or why it's there. All the more so now that "prompting" has been replaced by "loops."
And this is where the matter really interests me, because it isn't new. It's the oldest thing we have. I've been deep in mythology for a while now, for the reasons you already know, and one of the things that has surprised me most is how obsessed humanity is, across cultures that never once spoke to each other, with exactly this problem. The wish granted to the letter. The distance between what you asked for and what you wanted.
Schneier mentions Midas, and it's the textbook case: the king asks that everything he touches turn to gold and forgets to add "except food and drink" (and, in the version we all remember, his daughter). But Midas is only the best-known example. Eos, goddess of the dawn, asks Zeus to make her human lover Tithonus immortal and forgets to also ask for eternal youth; Tithonus ages forever, shrinking and drying without ever being able to die, until he's turned into a cicada. The Cumaean Sibyl asks Apollo for as many years of life as the grains of sand she could hold in her hand, makes the same omission, and ends up so withered and tiny that when they ask what she wants she can only manage to answer that she wants to die. W. W. Jacobs's monkey's paw grants three wishes: the father asks for two hundred pounds and receives them, exact, as compensation for his son's death in a machine accident; he then asks for the son to come back, and that same night something begins to knock at the door.
There are more examples and they all say the same thing. The sorcerer's apprentice enchants a broom to carry water and then doesn't know how to stop it, so he floods the house. What he asked for worked perfectly, only without an "off" button. The Golem of Prague, animated to protect, obeys orders with a literalness that ends up impossible to control. None of these stories is really about magic. They're about the moment you hand a goal to something that will meet it without sharing your idea of what counts as meeting it. The magic is just the wrapping; what gets passed from generation to generation is the warning. And maybe the warning was waiting for us to build a genie that could actually grant our wishes.