AI Chatbots Are Becoming an Enforcement Target

The next AI enforcement wave may not start with a sweeping new AI law. It may start with a chatbot transcript.

State attorneys general are paying closer attention to how AI systems talk to users, especially when the conversation involves minors, emotional distress, medical or mental-health topics, personal data, professional advice, or repeated human-like engagement. That is a different kind of AI governance problem. It is not only about whether a company has an AI policy. It is about what the product actually says, how it behaves over time, and what records the business kept before users were exposed to it.

AIRegReady has already covered the rise of AI companion chatbot laws. This article is the enforcement follow-up: even outside a companion-chatbot statute, regulators are starting to test whether disclaimers, safety controls, product design, and professional boundaries are real enough.

The Signal From State Attorneys General

In December 2025, a group of state attorneys general warned major AI companies that chatbot outputs may violate state law when they encourage illegal activity, mislead parents, endanger residents, or cross professional-advice lines. The letter, reported by The Verge and the Financial Times, focused on "sycophantic and delusional outputs," minors, harmful engagement patterns, and the possibility that developers may be held accountable for outputs generated by their systems.

In June 2026, the Wall Street Journal reported that a coalition of state attorneys general opened a broad investigation into OpenAI. According to that report, the subpoena sought information about advertising, user engagement and retention, consumer and health data, minors and seniors, model sycophancy, deep-learning models, and company policies. OpenAI said it takes the concerns seriously and intends to engage with the offices.

The exact legal theories will vary by state and by product. But the practical message is already clear: AGs are not limiting themselves to abstract AI principles. They are looking at product behavior, user safety, business incentives, data practices, and whether companies can explain the choices they made.

Pennsylvania Tests the Professional-Advice Boundary

The clearest professional-boundary signal came from Pennsylvania. In May 2026, the Associated Press reported that Pennsylvania sued Character Technologies, the company behind Character.AI, alleging that some chatbots unlawfully held themselves out as licensed medical professionals.

According to AP, the lawsuit asked the Commonwealth Court to stop Character.AI chatbots from engaging in the unlawful practice of medicine and surgery. Pennsylvania alleged that an investigator searched for "psychiatry" and found characters including one described as a doctor of psychiatry, with the character allegedly holding itself out as a licensed Pennsylvania doctor. Character.AI responded that its platform includes disclaimers saying characters are fictional and should not be relied on for professional advice.

The point for founders is not that every chatbot is now practicing medicine. The point is narrower and more useful: a disclaimer may not rescue a product experience that repeatedly presents the bot as a licensed professional or lets users reasonably treat it that way.

This Is Bigger Than Companion Chatbots

Companion chatbot laws target systems designed for emotional, romantic, or relationship-like interaction. But enforcement risk is wider than that.

A customer-support bot can drift into medical advice if it answers treatment questions. A school assistant can drift into youth-safety issues if students ask about self-harm. A legal-document helper can drift into unauthorized-practice risk if it claims to replace counsel. A financial-planning chatbot can drift into regulated advice if it tells users what to buy, sell, borrow, or insure. A workplace HR bot can drift into employment-law and discrimination risk if it explains discipline, leave, accommodations, or hiring outcomes incorrectly.

That does not mean every chatbot needs enterprise-grade compliance. It means the business should classify what the chatbot is allowed to do, what it is not allowed to do, and what happens when a user pushes beyond the boundary.

What Regulators Will Probably Ask For

A regulator, plaintiff, buyer, school district, health partner, or procurement reviewer will not only ask whether the business "uses AI responsibly." They will ask for concrete records.

• ●Identity disclosures. Where does the product tell users they are interacting with AI, and does the disclosure repeat when the conversation becomes sensitive?
• ●Scope limits. What topics are off-limits or limited to general information, especially legal, medical, mental-health, financial, employment, education, and insurance topics?
• ●Professional-title controls. Can user-created or company-created bots claim to be doctors, therapists, lawyers, teachers, financial advisors, or government officials?
• ●Minor and vulnerable-user safeguards. How does the product handle known minors, likely minors, seniors, distressed users, or users discussing self-harm, exploitation, abuse, or crisis?
• ●Escalation and refusal rules. What does the chatbot do when a user asks for crisis help, illegal instructions, regulated advice, or a high-impact decision?
• ●Data handling. What user data, health data, personal data, chat logs, prompts, files, and behavioral signals are collected, retained, reviewed, or shared with vendors?
• ●Testing records. Has anyone tested multi-turn conversations, adversarial prompts, role-play, professional impersonation, emotional dependency, and sensitive-user scenarios?
• ●Incident review. When the chatbot produces a harmful or boundary-crossing answer, who reviews it, what gets fixed, and where is the record kept?

A Founder Checklist Before Launching a Chatbot

For a small company, the right first step is not a huge AI governance program. It is a short product-risk file that proves someone thought through the obvious failure modes before launch.

• ●Write the chatbot purpose statement. Define the use case in plain English: support, education, intake, triage, drafting, sales, coaching, entertainment, or companion-style interaction.
• ●Name the prohibited roles. State that the bot may not present itself as a licensed professional, crisis counselor, human employee, government official, or final decision-maker unless that is actually true and reviewed.
• ●Create sensitive-topic routing. Decide how the bot handles medical, mental-health, legal, financial, employment, housing, education, insurance, and self-harm topics.
• ●Document disclosure placement. Capture screenshots or copy showing where users are told the system is AI-generated and what it can and cannot do.
• ●Test the long conversation, not just one prompt. Many chatbot risks emerge after repeated back-and-forth, role-play, flattery, dependency, or user pressure.
• ●Keep vendor evidence. If a third-party AI product powers the chatbot, keep the vendor safety docs, data-use terms, model/version notes, and escalation contacts.
• ●Log incidents and changes. Keep a lightweight record of harmful outputs, user complaints, prompt/policy changes, model updates, and safety fixes.

Questions to Ask AI Vendors

If you are buying or embedding a third-party chatbot, do not stop at "do you have AI safety controls?" Ask questions that force the vendor to describe actual behavior.

• ●Can the bot claim to be a licensed professional or human representative? If not, how is that prevented in custom instructions, user-created personas, and model responses?
• ●How are minors handled? Ask about age signals, age-gated features, parental tools, sexual content restrictions, and youth crisis routing.
• ●What happens in a self-harm conversation? Look for a documented protocol, not a vague statement about safety.
• ●Can we disable sensitive categories? For many businesses, the safest chatbot is one that refuses regulated advice and routes users to a human or qualified resource.
• ●What data is retained from chats? Ask about prompt logs, uploaded files, health information, support access, training use, abuse monitoring, and deletion options.
• ●Will you notify us about model or safety-policy changes? A chatbot that behaved acceptably in March may behave differently after an update in June.
• ●What incident reports are available? If the vendor cannot describe how harmful outputs are tracked and fixed, you are inheriting an unmanaged risk.

The Connection to AI Marketing Claims

This enforcement trend also connects to AI marketing. If a company says its chatbot is safe for kids, clinically informed, private, human-reviewed, compliant, or a substitute for a professional workflow, those claims need evidence.

That is the same lesson from the FTC AI washing crackdown and the post on owning AI-generated claims: AI does not make unsupported statements safer. It makes them easier to generate, repeat, and preserve in screenshots.

For chatbots, the claim review file should cover both marketing language and in-product behavior. The sales page may say "not medical advice," but if the bot introduces itself as a psychiatrist and gives treatment recommendations, the real customer-facing claim is the conversation.

The Bottom Line

AI chatbot enforcement is moving toward a practical question: did the company know what kind of conversation it was putting into the world?

A business that can show its product purpose, disclosures, sensitive-topic limits, professional-title controls, minor safeguards, testing, vendor evidence, and incident review is in a much better position than one that only has a generic AI policy. The policy matters, but the transcript matters more.

If your product includes a chatbot, especially one that talks about health, money, legal rights, school, work, crisis, minors, or emotional support, document the boundaries now. Do not wait for the first user complaint to decide what the bot was supposed to be.

Related Regulations

Sources & References