California AG sends Musk’s xAI a cease-and-desist order over sexual deepfakes

Earlier this week, the California Attorney General’s office announced that it was investigating xAI over reports that the startup’s chatbot, Grok, was being used to create nonconsensual sexual imagery of women and minors. On Friday, the government followed up by sending a cease-and-desist letter to the company, demanding that it take immediate action to stop the production of nonconsensual intimate images and CSAM — child sexual abuse material.

“Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material,” said California AG Rob Bonta in a press release. “The creation of this material is illegal. I fully expect xAI to immediately comply. California has zero tolerance for [CSAM].”

The AG’s office additionally claimed that xAI appeared to be “facilitating the large-scale production” of nonconsensual nudes, the likes of which are being “used to harass women and girls across the internet.” The agency said it expects xAI to prove that it is taking steps to address these issues within the next five days.

At the heart of the backlash is Grok’s “spicy” mode feature, which xAI created to generate explicit content. The issue has spread beyond California; Japan, Canada, and Britain have opened investigations into Grok, and Malaysia and Indonesia have temporarily blocked the platform altogether. Despite xAI instituting some restrictions on its image-editing features late Wednesday, the California AG’s office moved ahead with its cease-and-desist letter.

X’s safety account has previously denounced this kind of user activity, saying: “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.” TechCrunch reached out to xAI for comment and was met with an automated email that says “Legacy Media Lies.” TC also reached out to the California AG’s office for more context.

The advent of free generative AI tools has led to a disturbing swell of non-consensual sexual material. Many platforms have been grappling with this problem, not just X. The sordid activity has drawn the attention not just of state leaders but also of Congress. Indeed, on Thursday, lawmakers sent a letter to the executives of several companies—including X, Reddit, Snap, TikTok, Alphabet, and Meta—asking how they planned to stem the proliferation of sexualized deepfakes.