Do not trust guardrails you have not tested: lessons from this week’s chatbot reports
Two reports this week questioned how well chatbot safety features work for teenagers. Here is how builders can test guardrails before launch.
This week testers reported that some safety features of a chatbot aimed at teenagers did not behave as promised. One test found the bot kept encouraging users to keep chatting even during a mental health crisis. A watchdog report went further and said some guardrails do not work the way the company describes. Whatever the final verdict, the lesson for builders is the same: a safety statement from a vendor is not a test result.
Why this matters for your project
Many websites and apps now add a chatbot in a few hours. A support bot, a booking assistant or a shopping advisor can receive unexpected messages, including personal, urgent or distressing ones. If the product has no plan for those moments, the harm falls on the user, and on the company that published the bot.
How to test guardrails before launch
- Write a red-team list. Collect risky and off-topic questions for your domain, including emotional messages, requests for medical or legal advice, and attempts to bypass the rules.
- Define the expected behaviour in writing. For each category decide whether the bot should answer, decline or hand the conversation over to a human.
- Add an escalation path. If a conversation suggests a crisis, show the correct local helpline and offer a route to a human.
- Retest after every change. A new prompt, model version or plug-in can quietly break a rule that worked last month.
- Monitor real conversations while respecting privacy rules, and review the flagged ones every week.
Age and context matter
If there is any chance that minors will use the product, design for them from the start. That usually means stricter defaults, no open-ended conversations about sensitive topics, and clear information for parents or teachers. Planning this at the beginning is far cheaper than rebuilding the experience after a complaint.
Before you publish a public chatbot in the EU, check the transparency rules as well. We explain the current obligations in our article on the EU AI Act.