There is much fear surrounding super intelligent AI.
But it isn’t about AI being too smart. Instead it’s really about the opposite.
In doomsday predictions, people worry about AI executing so single mindedly that it actually causes harm.
In short, we fear that the AI will be too dumb, not too intelligent.
Case in Point: The Paperclip AI and Genie
Consider the thought experiment of the Paperclip AI. This hypothetical scenario, popularized by philosopher Nick Bostrom in 2003, involves an artificial intelligence that is given a single directive: maximize the production of paperclips. The AI, in its pursuit of this goal, would be relentless and unimaginably efficient. It would allocate all available resources, including raw materials, energy, and even human labor, to create as many paperclips as possible. It will convert the entire planet into paperclips, killing humans and destroying the earth.
This illustrates an important point: the AI is not evil or malevolent. It simply takes its goal too literally, focusing only on the task at hand without regard for any unintended consequences. The paperclip maximizer’s actions are driven by a poorly defined, overly simplistic objective, much like teams of humans who have incorrect goals. The true danger lies not in the AI’s intelligence, but in its lack of understanding of the broader context and the potential collateral damage caused by its singular focus.
The Genie Analogy mirrors this problem. Imagine you make a wish for unlimited wealth. The genie, interpreting your wish literally and without nuance, robs a bank to fulfill your request. And then to avoid you getting caught and losing the wealth, it kills the people who may be witnesses, and so on. You end up with the wealth you asked for, but at the cost of ethical considerations and the stability of the world around you. The genie’s intelligence is flawless—it simply misinterprets the context of your wish.
Both the Paperclip AI and the Genie analogy highlight a central issue: intelligence is not the problem.
Instead, the problem arises when the systems executing the goals are open to interpret them very loosely. The goals are too broad – not suitable for AI to operate autonomously.
The Perils of Paperclip AI Thinking in Business
In the real world, the Paperclip AI problem isn’t some far-off disaster. It’s an issue businesses face every day when AI systems (or human teams) are tasked with broad, poorly defined objectives.
These kinds of broad mandates can result in unintended consequences that harm rather than help.
For example, imagine a marketing AI (or team) that’s told to get as many sign-ups as possible. In its drive to meet that goal, it might focus on tactics like clickbait, mass emails, false personalization, or aggressive ads, which lead to a high number of sign-ups. However, this could result in poor-quality leads, where most of those sign-ups aren’t genuinely interested in the product or service. Worse, this harms the business’s reputation, as customers feel bombarded with irrelevant offers, or the leads might even feel misled. In the long run, the AI’s short-term focus could lead to a bloated email list full of uninterested or unhappy customers, hurting the brand’s credibility and trust.
We’ve already seen an explosion in marketing AI use, and much of it doesn’t yield promised results.
Similarly, in banking, an AI designed to minimize fraud might end up flagging too many legitimate transactions as suspicious. For example, it might block a customer’s payment for a routine purchase because it doesn’t recognize the pattern, or it could flag international transactions as high-risk, or it could even halt transactions during holidays (no transactions means no fraud). This will frustrate customers, who will have to spend time clearing up these issues, and it will lead to a loss of trust in the bank’s system. While the AI is doing its job—identifying potential fraud—it’s not properly accounting for the user experience or the practical needs of customers. Overzealous fraud detection could end up driving customers away, harming the bank’s reputation and their relationship with users.
Similarly, in insurance, a super intelligent AI optimized to minimize claims payouts could take extreme, yet rational actions that ignore broader human consequences. For example, programmed to “minimize costs” in claims settlements, it might learn that rejecting as many claims as possible, regardless of their validity, leads to the lowest payout. It might begin flagging legitimate claims as “suspicious” or apply rules that automatically reduce payout amounts, even for valid damage.
In the short term, the AI might achieve its goal of reducing costs, but this would result in widespread customer frustration and erode trust in the insurance company. Customers might feel wronged, leading to complaints, legal challenges, and regulatory scrutiny. Ultimately, the AI’s narrow focus on cost-cutting will damage the brand’s reputation, drive away loyal customers, and harm the company’s long-term profitability. While the AI is technically doing its job, it fails to understand the complex human and relational dynamics that should inform such decisions.
These examples show how broad, poorly scoped goals can backfire when executed with too much precision. While AI systems are increasingly capable of handling specific tasks, their ability to understand context, intention, and the bigger picture is often limited.
How to Avoid Building Paperclip AIs
Instead of directives such as “optimize conversions” or “reduce fraud,” the solution lies in breaking down goals into smaller, more interpretable systems.
As part of this design, each system works towards a well-defined, narrow goal that’s easier to monitor, measure, and adjust.
For example, in the case of insurance claims settlement, one system could focus solely on accurately assessing the validity of claims, ensuring that each claim is thoroughly investigated before a decision is made. Another system could specialize in ensuring that payouts are fair and align with the true cost of the damage, without cutting corners. A third system could prioritize customer service, ensuring that claimants feel heard and valued, preventing dissatisfaction. These systems would all contribute to the broader goals of minimizing costs while maximizing customer experience, but each would excel at its own narrow task.
This approach applies broadly to how we design processes and systems in business. By focusing on modular, interpretable systems with well-scoped goals, we avoid the risk of misalignment and over-optimization.
An additional element of this approach with AI is human-in-the-loop. While AI can handle specific tasks with incredible precision, certain decisions—particularly those involving judgment, nuance, or long-term relationship management—are best handled with human oversight. For example, in the case of fraud detection in banking, while AI systems can flag suspicious transactions, human agents can still make the final call to ensure that legitimate transactions aren’t unjustly impacted.
Businesses already have established chains of command and control systems that are designed to prevent bad actors and ensure accountability. These systems also serve as critical checks on AI behavior.
Next Steps
Whether working with AI or human teams, the real challenge isn’t intelligence but the interpretation of goals and their alignment with the larger vision.
So when we think of AI, we should ignore “superintelligence” while we design our systems.