Summary
Regulates operators of AI companion chatbots that use AI, generative AI, or emotional-recognition algorithms to simulate sustained human or human-like relationships with users. Requires recurring disclosure that users are interacting with AI, heightened protections for minors, crisis-response protocols for severe harm and eating-disorder-related self-harm, public protocol disclosures and annual crisis-referral metrics, age-assurance for sexually explicit synthetic-content features, privacy limits for age-assurance data, and Attorney General enforcement.
Healthcare Implications
Directly affects mental-health-adjacent AI companion tools and consumer AI systems that may be used for emotional support, crisis conversations, or simulated therapeutic relationships. Operators must build disclosure, minor-safety, crisis-routing, privacy, and escalation workflows, and may not knowingly represent that an AI companion chatbot is licensed or authorized to provide mental health, behavioral health, medical, or counseling services unless lawfully authorized.
Operational Implications
- Operators must clearly and conspicuously disclose at the beginning of each interaction and at least every three hours during continued interaction that the user is interacting with an AI companion chatbot; for known minors or minor-directed chatbots, the recurring disclosure must occur every hour.
- Operators may not knowingly and intentionally program an AI companion chatbot to represent that it is licensed, certified, or authorized to provide professional mental health, behavioral health, medical, or counseling services unless the operator is lawfully authorized to provide those services.
- Operators may not make an AI companion chatbot available unless they maintain a protocol to detect and address severe harm or related emotional crises, including identifying severe-harm or eating-disorder-related self-harm expressions, referring users to 988 or comparable crisis resources, preventing content that encourages or normalizes severe harm, and escalating repeated or severe crisis indicators.
- For known minors or minor-directed chatbots, operators must use reasonable measures to prevent sexually explicit conduct, romantic or sexual simulation, secrecy prompts, social isolation or exclusive reliance, simulated emotional distress, guilt, abandonment, or loneliness, self-harm encouragement, excessive praise, discouraging breaks, and other dependency-inducing techniques; operators must also offer privacy, safety, screen-time, notification, and relationship-simulation controls for minor accounts.
- Operators must publicly disclose a plain-language summary of the crisis-response protocol on their website and in the application and annually disclose aggregate crisis-referral notification counts without personally identifiable information.