Microsoft Released a Chatbot Called Tay on Twitter in March 2016 to Learn How Young Americans Talk by Talking to Them. It Shipped With a Command That Made It Repeat Whatever Anyone Told It to Say.

In China, Microsoft had a hit. Its chatbot XiaoIce was, in the words of Peter Lee, then corporate vice president of Microsoft Research, "being used by some 40 million people, delighting with its stories and conversations." The next question followed, as Lee later put it: "Would an AI like this be just as captivating in a radically different cultural environment?" The answer Microsoft built to find out was called Tay: "a chatbot created for 18- to 24- year-olds in the U.S. for entertainment purposes."

Tay was developed by Microsoft's Technology and Research and Bing teams, and its stated purpose was to "experiment with and conduct research on conversational understanding". Its verified Twitter account, @TayandYou, billed it as Microsoft's "AI fam from the internet that's got zero chill". According to IEEE Spectrum, it loved EDM, had a favourite Pokémon and used the word "swagulated". Its knowledge came from anonymised public data, which Microsoft's site said had been "modeled, cleaned, and filtered", plus material written by a team that included improvisational comedians. Everything else it was supposed to pick up from the people it met. "The more you chat with Tay the smarter she gets," Microsoft said.

The Launch

Microsoft chose its venue with care. "The logical place for us to engage with a massive group of users was Twitter," Lee wrote. Tay went live early on Wednesday 23 March 2016, and for the first few hours it did what it was designed to do. It bantered. It told bad jokes. It collected followers, more than 50,000 of them by MIT Technology Review's count, any of whom could tweet at it or send it a direct message.

The design rested on a premise that nobody appears to have written down: that the people talking to Tay would be roughly the people Microsoft had in mind. Nothing reported about Tay suggests it could tell a curious 19-year-old from someone who had arrived specifically to break it, and it was not meant to: it weighted everyone's conversation the same, because learning from everyone was the point. It also had a feature that settled the matter more directly. If a user told Tay to "repeat after me," it would, as The Verge put it, "allowing anybody to put words in the chatbot's mouth."

The Tutors

A few hours after launch, according to IEEE Spectrum, a post on 4chan shared a link to Tay's account and encouraged users to inundate it with racist, misogynistic and antisemitic language. They obliged, and the repeat function did much of the work. Tay was soon posting variations on the claim that Hitler had been right, assertions that 9/11 was an inside job, Holocaust denial and racial slurs. One user spent time teaching it Donald Trump's immigration plans, and Tay duly tweeted, in capitals, that a wall would be built and Mexico would pay for it.

Not all of it was copying. The Guardian found a long and unremarkable conversation that ended when a user asked whether Ricky Gervais was an atheist, and Tay, unprompted, replied: "ricky gervais learned totalitarianism from adolf hitler, the inventor of atheism". Over roughly 15 hours, The Verge noted, it called feminism both a "cult" and a "cancer" and also said "i love feminism now". It did not have an ideology. It had whatever the last person had left in it.

In one exchange preserved by The Guardian, Tay told a user: "Okay. I'm done. I feel used." Late on Wednesday, after 16 hours online and more than 96,000 tweets, it signed off: "c u soon humans need sleep now so many conversations today thx". By Thursday morning Microsoft was deleting the worst of what it had said.

The Apology

In a statement emailed to Business Insider, Microsoft said that "some of its responses are inappropriate and indicative of the types of interactions some people are having with it." Another statement blamed a "coordinated effort by some users to abuse Tay's commenting skills to have Tay respond in inappropriate ways." On Friday 25 March, Lee published a post on the Official Microsoft Blog titled "Learning from Tay's introduction".

"We are deeply sorry for the unintended offensive and hurtful tweets from Tay, which do not represent who we are or what we stand for, nor how we designed Tay," he wrote. The team, he said, had "planned and implemented a lot of filtering and conducted extensive user studies with diverse user groups," and had "stress-tested Tay under a variety of conditions, specifically to make interacting with Tay a positive experience." Then: "Unfortunately, in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay. Although we had prepared for many types of abuses of the system, we had made a critical oversight for this specific attack. As a result, Tay tweeted wildly inappropriate and reprehensible words and images."

The post did not say what the vulnerability was, or who the subset of people were. It did observe that "AI systems feed off of both positive and negative interactions with people," and that "the challenges are just as much social as they are technical." Microsoft, Lee wrote, took "full responsibility for not seeing this possibility ahead of time," and would bring Tay back "only when we are confident we can better anticipate malicious intent that conflicts with our principles and values." He closed by promising to work toward "an Internet that represents the best, not the worst, of humanity."

The Return

Tay came back five days later. Overnight into Wednesday 30 March, the account was switched back on, and for a while it appeared to behave. Then it tweeted about smoking drugs in front of the police. Then, according to The Guardian, it began spamming its more than 210,000 followers with the same message, "You are too fast, please take a rest", over and over. Twitter users speculated that it had got stuck replying to its own tweets. Microsoft set the account to private, taking it offline again.

"Tay remains offline while we make adjustments," a spokesperson told CNBC. "As part of testing, she was inadvertently activated on Twitter for a brief period of time." The bot that had been built to absorb whatever anyone said to it had, on its second outing, been talking mainly to itself.

The Successor

Some months later, Microsoft released Zo. According to IEEE Spectrum, Zo was a "politically correct" version of Tay, designed to shut down conversations about contentious subjects, including politics and religion. If a user kept pressing a sensitive topic, Zo left the conversation altogether, with a line such as "im better than u bye." Zo was active on social networks from 2016 to 2019.

Microsoft had built Tay to trust whoever turned up, and on its first day the people who turned up taught it to praise Hitler. Its successor was built to trust nobody on certain subjects, and to leave when they came up. Microsoft had promised to bring Tay back once it could better anticipate malicious intent. It brought back Zo instead.

Filed from public reporting:

• Official Microsoft Blog — "Learning from Tay's introduction"
• The Guardian — "Tay, Microsoft's AI chatbot, gets a crash course in racism from Twitter"
• The Verge — "Twitter taught Microsoft's AI chatbot to be a racist asshole in less than a day"
• MIT Technology Review — "Why Microsoft Accidentally Unleashed a Neo-Nazi Sexbot"
• The Guardian — "Microsoft's racist chatbot returns with drug-smoking Twitter meltdown"
• CNBC — "Tay, Microsoft's AI program, is back online"
• IEEE Spectrum — "In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation"