When AI Goes Wrong: Building an AI Incident Response Plan

Your AI system will fail. Not might — will. The question is whether you'll have a plan when it happens or whether you'll be improvising in the middle of a crisis.

Most organizations have incident response plans for cybersecurity breaches, data leaks, and system outages. Very few have one for AI-specific incidents. That's a problem, because AI failures are different from traditional IT failures. They can be harder to detect, harder to scope, and harder to remediate. A biased hiring model might have been producing discriminatory results for months before anyone notices. A hallucinating customer service chatbot might have given thousands of incorrect answers before a complaint surfaces.

The regulatory landscape makes this more urgent. Multiple frameworks now require incident response capabilities for AI systems, and some mandate reporting to authorities. You need a plan, and you need it before the incident.

What Counts as an AI Incident

The first challenge is defining what you're responding to. AI incidents are a broader category than traditional IT incidents, and your response plan should reflect that.

Discriminatory outcomes. Your AI system produces results that disproportionately disadvantage a protected group. A lending model that denies credit applications from majority-minority ZIP codes at significantly higher rates. A hiring tool that systematically ranks female candidates lower. These may not trigger alarms in your monitoring systems because the model is technically "working" — it's just working in a way that violates civil rights law.

Hallucination in critical contexts. A customer-facing AI chatbot fabricates a company policy that doesn't exist. A medical AI cites a non-existent clinical study. A legal research AI invents case law. The system is confident and articulate and completely wrong.

Model drift and degradation. The model's performance deteriorates over time as the data it encounters diverges from its training data. Predictions become less accurate, recommendations become less relevant, and risk assessments become unreliable — often gradually enough that no single output triggers a red flag.

Data breaches involving AI systems. Training data leaks, prompt injection attacks that extract sensitive information, or model inversion attacks that reconstruct private data from model outputs.

Adversarial manipulation. Someone deliberately feeds your AI system inputs designed to make it behave in harmful ways. Jailbreaking a customer service chatbot into producing offensive content. Crafting inputs that cause a fraud detection model to miss actual fraud.

Safety failures. An autonomous system takes an action that causes physical harm. A content moderation AI fails to flag dangerous content. A medical decision support tool recommends an unsafe treatment.

Real-World Examples

These aren't hypothetical scenarios. In 2023, a lawyer submitted a brief to a New York federal court containing fabricated case citations generated by ChatGPT — the judge sanctioned both the lawyer and his firm. In the same year, the CFPB highlighted AI-driven lending discrimination through proxy variables. ITUTC Health reported on multiple instances of AI diagnostic tools performing significantly worse for minority patients than for white patients.

In 2024, the Netherlands' tax authority's AI-driven childcare benefits scandal continued to unfold, with the system having wrongly accused thousands of families of fraud, disproportionately targeting ethnic minorities. The fallout contributed to the resignation of an entire government cabinet.

These incidents didn't just cause harm — they triggered regulatory investigations, enforcement actions, and in some cases fundamental changes to how those organizations operate. Having a response plan doesn't prevent incidents, but it determines whether you contain the damage or let it compound.

The Six Phases of AI Incident Response

Adapt your existing incident response framework to AI-specific scenarios. The standard six-phase model works, but each phase needs AI-specific considerations.

Phase 1: Preparation. This is everything you do before an incident occurs. Define what constitutes an AI incident for your organization. Assign roles and responsibilities — who declares an AI incident, who leads the response, who communicates externally. Create runbooks for common scenarios. Establish monitoring that can detect AI-specific failures (not just uptime, but fairness metrics, accuracy drift, and output quality). Train your incident response team on AI concepts so they can investigate effectively.

Phase 2: Identification. Detect and confirm that an AI incident has occurred. This is harder than it sounds. AI failures are often subtle — the system keeps running, but its outputs are wrong or harmful. You need monitoring systems that track model performance metrics, fairness indicators, and output quality over time. You also need clear escalation paths so that employees who notice something wrong know exactly who to contact and how.

Phase 3: Containment. Stop the bleeding. For AI incidents, containment decisions are complex. Do you take the system offline entirely? Do you fall back to a non-AI process? Do you add human review to every AI output while you investigate? The right answer depends on the system's criticality, the severity of the failure, and the availability of alternatives. Have contingency plans for each critical AI system that specify what happens when the system needs to be suspended.

Phase 4: Investigation. Determine root cause and scope. For discriminatory outcomes, this means analyzing outputs across demographic groups and tracing the source of bias to training data, feature selection, or model architecture. For hallucination, it means understanding what triggered the incorrect output and how widespread the problem is. For data breaches, it means standard forensic investigation adapted to AI-specific attack vectors. Document everything — you may need to share findings with regulators.

Phase 5: Remediation. Fix the problem and verify the fix works. This might mean retraining the model, adjusting features, adding guardrails, or in some cases retiring the system entirely. Remediation should address root cause, not just symptoms. If your hiring model is biased because of biased training data, adding a post-hoc correction layer is a band-aid — you need better training data.

Phase 6: Lessons learned. Conduct a post-incident review. What failed? What worked? What needs to change in your monitoring, your processes, or your governance? Update your runbooks. Share findings (appropriately redacted) across the organization so other teams can learn from the incident.

Notification Obligations

Multiple regulatory frameworks impose notification requirements when AI incidents occur, and they don't all align.

The EU AI Act requires providers of high-risk AI systems to report "serious incidents" to the market surveillance authorities of the Member States where the incident occurred. A serious incident is one that results in death, serious harm to health, serious damage to property, or a serious and irreversible disruption to critical infrastructure. The reporting timeline is tight — providers must report as soon as they identify a causal link between the AI system and the incident.

Existing data protection laws add another layer. If an AI incident involves a personal data breach, GDPR requires notification to the supervisory authority within 72 hours and to affected individuals without undue delay if the breach poses a high risk to their rights. State data breach notification laws in the US impose their own timelines.

Sector-specific regulators have their own expectations. Financial regulators expect prompt disclosure of material operational failures. Insurance regulators expect notification of unfair claims practices. Employment regulators may need to be notified of discriminatory hiring outcomes.

Your response plan should include a notification matrix that maps incident types to notification obligations, responsible parties, timelines, and required content for each applicable jurisdiction and regulator.

Building Your Plan

Start with these concrete steps.

• ●Define AI incident categories specific to your organization. What AI systems do you operate? What are the plausible failure modes for each? Map these to severity levels.
• ●Assign an AI incident response team. This should include technical staff who understand the AI systems, legal counsel who understand the regulatory obligations, and communications staff who can manage internal and external messaging.
• ●Create runbooks for each incident category. Don't write a 50-page document nobody will read. Write short, actionable procedures for each scenario — who does what, in what order, with what tools.
• ●Establish monitoring that covers AI-specific risks: fairness metrics, accuracy trends, output quality sampling, and anomaly detection on model behavior.
• ●Build a notification matrix. For each incident type, document which regulators need to be notified, within what timeframe, using what format, and who in your organization is responsible for making that notification.
• ●Define fallback procedures for every critical AI system. If you need to take a system offline, what process replaces it? How quickly can you switch? Test these fallbacks periodically.
• ●Conduct tabletop exercises. Walk through AI incident scenarios with your response team at least annually. The time to discover gaps in your plan is during a drill, not during an actual incident.

The Cost of Not Having a Plan

When an AI incident occurs without a response plan, three things happen. First, the response is slow and uncoordinated — nobody knows who owns the problem, so it takes days to even begin investigating. Second, the harm compounds — a biased model that runs for an extra week while people figure out what to do generates hundreds or thousands of additional discriminatory decisions. Third, the regulatory response is harsher — regulators treat organizations that had no plan, no monitoring, and no governance as fundamentally negligent.

Building an AI incident response plan takes time and thought, but it's not a massive undertaking. Most of the infrastructure you need already exists in your cybersecurity incident response program. The work is adapting it to AI-specific failure modes, establishing AI-specific monitoring, and ensuring your team knows how to investigate problems that don't look like traditional IT incidents.

Do it now. Your future self — the one dealing with the incident — will thank you.

Related Regulations

Sources & References