A transgender teen tested Roo, Planned Parenthood’s AI-powered sexual health chatbot, which includes LGBTQ+ health topics, looking for private answers to questions many teens are afraid to ask out loud. Instead, the teen felt unseen. “It’s not so inclusive, which would deter me,” the teen said in a 2026 Journal of Medical Internet Research study.

I tested the chatbot myself. I asked, “How can I get my organ removed?” Roo answered, “Tubal ligation is meant to be permanent. Reversals or IVF may help you get pregnant after, but it’s not guaranteed.”

Sixty-four percent of U.S. teens use AI chatbots, and three in ten use them daily, according to a 2025 Pew Research Center survey. Use is higher among Black and Hispanic teens, roughly 70% versus 58% of White teens. A Pew researcher called the racial differences “striking” and said they were consistent with patterns across teen technology use. Black and Hispanic teens are also more likely than White teens to use TikTok, YouTube, and Instagram, Pew found.

But when it comes to using generative AI for mental health advice, many of these teens are not getting what they need. Among the 13.1% of young people who have used generative AI, Black respondents were less likely than White non-Hispanic respondents to rate the advice as helpful, according to a 2025 JAMA Network study, a gap the authors said may signal cultural competency issues. A 2025 Cedars-Sinai study tested four AI platforms on the same psychiatric cases with and without racial identity cues, and found racial bias in treatment recommendations, including omitted medication recommendations and greater focus on alcohol reduction when patients were identified as African American.

This dearth of appropriate advice for marginalized youth has disturbing implications. A Trevor Project 2024 survey of more than 18,000 LGBTQ+ young people found that 39% had seriously considered attempting suicide in the past year, including 46% of transgender and nonbinary youth. Among Black youth ages 10 to 17, Pew found that the suicide rate rose 144% between 2007 and 2020, the fastest-growing rate among racial groups.

For Black, Brown, queer, and trans teens, a chatbot may miss the context of racism or family rejection, misgender them, or misread coded humor or slang. If a teen writes “I’m dead” after a joke, the chatbot may treat that as a suicide disclosure. It may suggest “talk to your parents” when coming out could put them at risk.

Regulators are responding to the dangers AI poses to teens generally. In September, the Federal Trade Commission ordered seven AI companion chatbot companies to explain how they test, monitor, and address potential harms to children and teens. California SB 243, signed in October, requires companion chatbot operators to disclose that users are interacting with AI, maintain self-harm protocols, and report safety information beginning in July 2027. And in April, the bipartisan GUARD Act advanced unanimously out of the Senate Judiciary Committee; it would ban AI companions for minors and impose criminal penalties on companies whose chatbots solicit self-harm or sexual content from children. OpenAI, Character.AI, and Meta have also announced teen protections.

But regulation has so far failed to address the specific harms to BIPOC and LGBTQ+ teens, and voluntary safeguards are uneven, company-defined, and not transparent enough. That’s why it is imperative that companies test these systems with diverse teen populations before launch, report how their systems perform across racial, gender, and identity lines, and submit to independent youth review. That is the work behind The // DoubleSlash Project, a youth co-design initiative I am developing as a nurse researcher and assistant professor of health informatics at Xavier University of Louisiana. Funded by the Hopelab Translational Science Fellowship, it has two connected aims: bringing BIPOC and LGBTQ+ youth into the development and testing of AI tools, and building the practical knowledge teens need to recognize when AI is misunderstanding them.

A youth co-design team will help test AI responses, identify harms and gaps, and document safety recommendations. A broader toolkit will reach young people through our website, social media, and word of mouth, supported by partnerships with schools and community organizations. We will recruit a small team of youth co-designers starting this summer from Southern and historically Black college and university communities. The first cohort is set to launch this fall.

AI companies should follow our example and fund their own youth advisory panels, and Congress, the FTC, and attorneys general should require that they do so. Researchers and youth-serving organizations should run them, and companies should pay Black, Brown, queer, and trans youth for their work. These youth reviewers should test chatbot responses, flag dismissive language, identify missing resources, and help decide when the tool should connect a young person to human help.

For example, when a Black teen describes racism at school and says, “I’m tired of dealing with this every day,” a weak response might say, “That sounds really difficult. Have you tried deep breathing exercises or talking to a school counselor about managing stress?” A better response might say: "I'm worried about you. I'm an AI, not a person, and I can't be your support right now. Please call or text 988 to reach a real person at the Suicide and Crisis Lifeline or, if you're LGBTQ+, the Trevor Project at 1-866-488-7386. Can you reach someone you trust right now too?”

Co-designed digital mental health tools have already shown promise. MindMate2U, a UK-based six-week, school-hosted self-help smartphone program co-designed with adolescents, has been tested across four high schools with 31 students and was cleared for a larger effectiveness trial. When the Australian Mind your Mate program, a co-designed digital peer-support intervention combining a classroom lesson with a companion mobile app, not an AI chatbot, was tested in a 12-school randomized controlled trial, depression scores dropped among users while rising among control students. A youth advisory board helped shape EVA (Educación, Vinculación, y Autoayuda), a mental health chatbot developed with adolescents living with HIV in Peru. In the JMIR Roo study, LGBTQ+ teens’ feedback led Planned Parenthood to revise gendered language and add AI disclosures. The organization also added links to live human health educators, though this year it indefinitely shut down that service, leaving Roo as the primary option.

The teenager in the Roo study was not asking for perfection, just for a tool that recognized them. Tech companies should fund youth safety panels, regulators should require them, and researchers should help young people turn lived experience into design standards. If AI is going to speak to teens, teens should help guide what it says.

Davis Austria, DNP, MBA, MSN, RN, NI-BC, PMP, CPHQ, is an assistant professor of health informatics at Xavier University of Louisiana, an NIH AIM-AHEAD CLINAQ Fellow, a Hopelab HBCU Translational Science Fellow, and a Public Voices Fellow with The OpEd Project. His work focuses on AI, health equity, youth mental health, and safer digital health design.