NYC AI Chatbot Advised Small Businesses to Break Labor and Housing Laws

In March 2024, New York City launched an AI-powered chatbot designed to help small business owners navigate city regulations and requirements. The chatbot was intended to provide guidance on permits, licensing, and compliance issues to support the city's small business community. However, testing by The Markup revealed that the system was providing dangerously incorrect legal advice that could expose business owners to significant legal liability. The Markup's investigation documented multiple instances where the chatbot advised users to engage in illegal practices. When asked about housing discrimination, the bot told users they could discriminate against tenants based on source of income, despite this being explicitly prohibited under NYC law. The system also incorrectly advised restaurant owners that they could keep workers' tips for themselves and that they didn't need to pay minimum wage to tipped employees, both violations of labor law. Additional problematic responses included advice that businesses could charge different prices based on protected characteristics and guidance that contradicted established employment regulations. The investigation revealed that the chatbot lacked proper safeguards to prevent the dissemination of illegal advice. The system appeared to have been trained on general business guidance materials without adequate grounding in New York City's specific legal framework. There was no apparent human review process for responses that contained legal advice, nor were there guardrails to flag potentially problematic guidance for verification. The chatbot also failed to include appropriate disclaimers about the limitations of its legal knowledge. Following The Markup's report, New York City officials acknowledged the problems and took the chatbot offline for review and corrections. The incident highlighted the risks of deploying AI systems in government contexts without adequate oversight, particularly when providing guidance that could have legal implications for users. The city's response included plans to implement better training procedures and review processes before relaunching the service.