NYC MyCity AI Chatbot Advised Breaking Laws on Housing Discrimination and Minimum Wage

On May 29, 2024, The Markup published an investigation revealing that New York City's AI-powered MyCity chatbot had been providing legally incorrect advice to small business owners since its October 2023 launch. The chatbot, developed by Microsoft to help entrepreneurs navigate city regulations, was discovered to be actively advising users to violate local housing discrimination laws and minimum wage requirements. Testing conducted by The Markup in May 2024 exposed systematic flaws in the system's understanding of NYC's specific legal framework. The MyCity chatbot, built on Microsoft's AI technology platform, demonstrated fundamental failures in its training on New York City's legal requirements. When queried about housing practices, the system incorrectly advised that landlords could reject tenants based on their source of income, directly contradicting NYC's Source of Income Discrimination Law which explicitly prohibits such discrimination. Additionally, the chatbot provided erroneous guidance suggesting that some businesses might be exempt from the city's minimum wage laws, potentially leading employers to commit wage theft violations. The system appeared to rely on generic legal interpretations rather than NYC-specific regulations. The chatbot's incorrect advice created substantial legal exposure for any small business owners who followed its guidance. Violations of NYC's Source of Income Discrimination Law can result in fines up to $250,000 and civil lawsuits, while minimum wage violations expose employers to back pay claims, penalties, and investigations by the Department of Labor. The misinformation particularly threatened vulnerable business owners who rely on official city guidance and lack resources for independent legal counsel. The incident also raised concerns about the city's liability for damages caused by following officially sanctioned but incorrect advice. Following The Markup's publication on May 29, 2024, New York City officials immediately acknowledged the chatbot's failures and took the system offline for emergency corrections. The city committed to implementing enhanced oversight mechanisms and improving the AI system's accuracy before any potential relaunch. NYC officials emphasized that the chatbot was meant to provide general guidance only, though this disclaimer had not prevented the system from giving specific legal advice that contradicted city law. The incident highlighted broader risks in deploying AI systems for government services, particularly in legal and regulatory domains where accuracy is essential for compliance. Similar concerns about AI chatbots providing incorrect legal or regulatory advice have emerged in other jurisdictions, prompting questions about oversight standards for government AI implementations. The MyCity failure demonstrated the critical need for comprehensive legal review and validation processes before deploying AI systems that could influence business decisions with significant legal consequences.