Illustration by Natalya on Unsplash
[This article was originally published in Harvard Business Review on June 13, 2025.
It’s virtually impossible to have a conversation about the future of business without talking about AI. What’s more, the technology is evolving at a furious pace. What started as AI chatbots and image generators are becoming AI “agents”—AI systems that can execute a series of tasks without being given specific instructions. No one knows exactly how all this will play out, but details aside, leaders are considering the very real possibility of wide-scale organizational disruption. It’s an exciting time.
It's also a nerve-wracking one. Ever-evolving AI brings a new suite of risks, which makes it extraordinarily difficult to meet what I call “The Ethical Nightmare Challenge.” In the face of AI hallucinations, deepfakes, the threat of job loss, IP violations, discriminatory outputs, privacy violations, its black box nature, and more, the challenge asks leaders to:
Identify the ethical nightmares for their organizations that may result from wide-scale AI use.
Create the internal resources that are necessary for nightmare avoidance.
Upskill employees so they can use those resources, along with their updated professional judgment, to avoid those nightmares.
The challenge is significant, but the reward—wide-scale, safe deployment of transformative technologies—is worth it.[HB: This line is repeated in the conclusion. Cut/rework one? I also included it in the exec summary.
Agentic AI makes rising to this challenge more urgent: It introduces compounding risks that, if not managed, can create business and brand-defining disasters. As someone who helps companies navigate the ethical risks posed by new technologies, I’ve seen the mistakes companies often make as they try to meet this problem, and what should be done instead. Often, that requires significant changes throughout a company, not merely a risk assessment to be tacked onto existing assessments.
How Ethical Nightmares Multiply with AI Advances
At most organizations, AI risk management was developed to tackle the potential harms of narrow AI—and this has remained the foundation for how AI risk is handled. To protect their organizations as they adopt new AI tools, leaders need to understand how the risk landscape changes as we move to generative AI, and how narrow and generative AI serve as building blocks for a dizzying array of possibilities that lead to a minefield.
Narrow AI
Narrow AI, also called “traditional” and “predictive” AI, is in the business of making predictions in a narrow domain. Well-known examples include facial recognition software and scoring models (e.g., predicting the risk of defaulting on a mortgage, likelihood of being a good candidate for the job, and so on, With narrow AI, as I’ve previously written about, prominent ethical risks include biased or discriminatory outputs, an inability to explain how AI arrives at its outputs, and privacy violations.
At most companies, risk programs started by focusing on mitigating the possible harms of narrow AI. There are four important things to understand about how those were constructed so we can see where they fail when it comes to generative AI and beyond:
The context for how a narrow AI will be used tends to be understood in advance. For instance, if you’re developing a resume-reading AI, chances are it will be used in hiring. (In other cases it can be less clear—e.g., developing facial recognition software that can be used in a variety of contexts.)
Data science expertise is needed to perform risk assessments, monitor performance, mitigate risk, and explain how the AI works. Downstream users, on the other hand, play a role in submitting data to the tool, where in all likelihood they did not create the data themselves (e.g., the HR professional didn’t write the resumes, the insurance professional didn’t fill out the application, and so on).
There’s often an expert “human in the loop” checking AI outputs before they’re acted upon. For instance, while an AI may predict the likelihood of someone developing diabetes in the next two years, a doctor interprets that output and offers the relevant advice/prescriptions to the patient. The outputs are generated at a pace the human can handle, and they have the capacities to responsibly vet the output.
Monitoring and intervention can be relatively straightforward. Tools for assessing performance abound, and if the AI is performing poorly on some relevant metric, you can stop using it. The disruption caused by ceasing to use the tool is fairly contained.
Generative AI
With generative AI, however, things change rather drastically:
The contexts of deployment explode. People use LLMs for all sorts of things. Think of the many ways they may be used in every company by every department by every role for every task. Then add a few thousand more for good measure. This makes testing for “how will the model perform in the intended context of use” phenomenally difficult to determine.
Monitoring the AI in the wild becomes immensely important. Because there are so many contexts of deployments, developers of AI cannot possibly foresee all of them, let alone introduce appropriate risk mitigations for them all. Monitoring the AI as it behaves in these unpredictable contexts—before things go off the rails with no one noticing—becomes crucial.
A human in the loop is still relatively straightforward—but increased training is needed. For instance, LLMs routinely make false statements (a.k.a. “hallucinations”). If users are not appropriately trained to fact check LLM outputs, then organizations should expect employees to regularly create both internal- and external-facing reports and other documents with false information.
Gen AI requires extra training to use responsibly. That’s because while narrow AI’s outputs are primarily a function of how data scientists built the AI, generative AI’s outputs are largely a function of the prompts that end users enter; change the prompt and you’ll change the outputs. This means that responsible prompting is needed (e.g., not putting company sensitive data into an LLM that sends the data to a third party).
Risk assessment and mitigation happens in more places. For the most part, organizations are procuring models from companies like Microsoft, Google, Anthropic, or OpenAI rather than building their own. Developers at those companies engage in some degree of risk mitigation, but those mitigations are necessarily generic. When enterprises have their own data scientists modify the pre-trained model—either at the enterprise or department level—they also create the need for new risk assessments. This can get complex quickly. There are multiple points at which such assessments can be performed, and it’s not clear whether they're needed before and after every modification. Who should perform what risk assessment at what point in this very complex lifecycle of the AI is difficult to determine not only in itself, but also because that decision must be made while balancing other important considerations—e.g., operational efficiency.
For those organizations that have built AI ethical risk/Responsible AI programs, the risks discussed so far are covered (though how well they're covered/managed depends on how well the program was built, deployed, and maintained). But as AI evolves, those programs begin to break under the strain of massively increased complexity, as we’ll see.
Multi-Model AI and Agentic AI
Now things get phenomenally complicated, and that’s because narrow and generative AI are building blocks to creating complex systems. Let’s take this in stages. There are different ways of carving up these stages, but the point here is to give you a sense of how the complexity scales quickly and easily:
Stage 1: You take an LLM and connect it to another generative AI—say, one that generates video clips. In this case, you might tell your LLM you want three different videos of cows jumping over the moon, and it may connect to a video-generating AI tool to do so. Or you can connect your LLM to a narrow AI and a database; you might tell your LLM to connect to the resume database to collect the resumes for a particular position, run them through the resume-scoring narrow AI, and report on the top five results. Now you’ve got multi-model AI.
Stage 2: Connect your LLM to 30 databases, 50 narrow AIs, 5 generative AIs, and the entire internet. No special name for this; just remember that the internet contains all sorts of crazy, biased, false information that your AI may pull from.
Stage 3: Add to your multi-model AI the ability to take digital actions (e.g., perform financial transactions). Now you’ve got multi-model agentic AI.
Stage 4: Give your multi-model agentic AI the ability to talk to other multi-model AI agents in your organization. Now you have internal multi-model multi-agentic AI.
Stage 5: Give your internal multi-model multi-agentic AI agent the ability to talk to AI agents outside of your organization. Now you have a head-spinning quagmire of incalculable risk. (Note: Not a technical term.)
This progression shows executives where their organization sits on the complexity curve—and more importantly, what capabilities they need to build before moving to the next stage. In my work helping Fortune 500 companies design and implement AI ethical risk programs, I have yet to encounter an organization that has the internal resources or trained personnel to handle Stage 2, let alone the later stages. Here’s why The Ethical Nightmare Challenge just got so much harder:
First, who should perform what risk assessment at what time gets incredibly difficult to determine. There are so many nodes of interaction (and let’s not forget the risk assessments needed within a single model) that performing a risk assessment at each and every node is pragmatically impossible. Careful cost/benefit analyses need to be performed to determine where it's necessary and where it’s a nice to have in relation to the organization’s risk appetite.
Second, the ability for an end user (i.e., the human in the loop) to wisely stand between system outputs and impacts decreases drastically even at Stage 1, let alone Stage 5. There's just too much data for any human to possibly process in real time (or even after time, for that matter).
Third, related to human in the loop breaking down and the speed at which models interact, enormous weight must be put on go/no-go decisions for deployment. Senior executives and technical leads must determine whether AI systems are ready for use, but this presupposes they know how to rigorously test and evaluate these systems before giving final approval. Currently, most organizations lack these critical pre-deployment evaluation frameworks, leaving them to deeply under-informed high-stakes decisions.
Fourth, since no pre–green light decision can possibly consider all the ways in which things may go sideways (especially at Stage 5), monitoring in real time is of tremendous importance. Without real-time monitoring, the pace at which things can unravel is diabolical.
Fifth, methods must be designed to intervene in the system when that light starts blinking red. However, since these are phenomenally complex systems interacting with other complex systems—all of which require tremendous resources to operate and are presumably only deployed because of their tremendous benefits—methods of intervention that minimally disrupt the system while decreasing the risks to appropriate levels are needed. For instance, it’s better to identify that narrow AI model #23 is the source of the problem and to shut off access to that model than to shut the whole system down. (That said, some risks arise from the system itself instead of any particular node in the system, in which case it will likely be necessary to shut down the system.)
Sixth, it cannot be overstated how important upskilling employees is in all this. General education and training that goes well beyond annual 30-minute compliance video watching is required for all. Specialized training is needed at least at the department level and in many cases at the role level. This training must result in, among other things, employees who can regularly procure, develop, use, and monitor AI and AI systems responsibly and who can smell smoke when something isn’t right. This is not a one-and-done training. This is something that must be continued as long as AI use continues to scale and evolve. Indeed, the most successful companies with whom I’ve worked have one thing in common: They invested and continue to invest heavily in employee training before deploying the technology, not after problems emerge.
Rising to the Ethical Nightmare Challenge
AI will continue evolving from narrow tools to sophisticated multi-agentic systems that operate at speeds and scales beyond human oversight. Organizations face a stark choice. They can either proactively rise to The Ethical Nightmare Challenge now, while the complexity is still manageable, or they can wait until a catastrophic failure forces their hand—likely at far greater cost and with significant damage to their relationships with customers, clients, and investors.
The companies that will thrive in the agentic AI era are those that recognize this inflection point for what it is: not just a technological upgrade, but a fundamental transformation in how organizations manage risk, train employees, and make decisions. They understand that moving from Stage 1 to Stage 5 without the proper infrastructure isn't innovation—it's recklessness.
The good news is that organizations don't need to solve everything at once. They need to honestly assess where they are on the complexity curve, build the capabilities required for their current stage, and create the infrastructure to evolve safely to the next. This means investing in comprehensive employee training, developing robust monitoring systems, and creating intervention protocols before they're desperately needed. It’s a demanding challenge, but the alternatives – charging ahead recklessly or with the organizational foot forever hovering over the gas pedal – is far worse.