“What’s surprising about these large language models is how much they know about how the world works simply from reading all the stuff that they can find,” says Chris Manning, a professor at Stanford who specializes in AI and language.
But GPT and its ilk are basically very proficient statistical parrots. They learn to re-create the patterns of phrases and grammar which are discovered in language. That means they’ll blurt out nonsense, wildly inaccurate facts, and hateful language scraped from the darker corners of the net.
Amnon Shashua, a professor of laptop science on the Hebrew University of Jerusalem, is the cofounder of one other startup constructing an AI mannequin based mostly on this method. He is aware of a factor or two about commercializing AI, having bought his final firm, Mobileye, which pioneered utilizing AI to assist vehicles spot issues on the street, to Intel in 2017 for $15.Three billion.
Shashua’s new firm, AI21 Labs, which got here out of stealth final week, has developed an AI algorithm, known as Jurassic-1, that demonstrates hanging language expertise in each English and Hebrew.
In demos, Jurassic-1 can generate paragraphs of textual content on a given topic, dream up catchy headlines for weblog posts, write easy bits of laptop code, and extra. Shashua says the mannequin is extra refined than GPT-3, and he believes that future variations of Jurassic could possibly construct a sort of commonsense understanding of the world from the knowledge it gathers.
Other efforts to re-create GPT-Three mirror the world’s—and the web’s—range of languages. In April, researchers at Huawei, the Chinese tech large, published details of a GPT-like Chinese language mannequin known as PanGu-alpha (written as PanGu-α). In May, Naver, a South Korean search large, stated it had developed its personal language mannequin, known as HyperCLOVA, that “speaks” Korean.
Jie Tang, a professor at Tsinghua University, leads a group on the Beijing Academy of Artificial Intelligence that developed one other Chinese language mannequin known as Wudao (that means “enlightenment”) with help from government and industry.
The Wudao model is considerably larger than any other, meaning that its simulated neural network is spread across more cloud computers. Increasing the size of the neural network was key to making GPT-2 and -3 more capable. Wudao can also work with both images and text, and Tang has founded a company to commercialize it. “We believe that this can be a cornerstone of all AI,” Tang says.
Such enthusiasm seems warranted by the capabilities of these new AI programs, but the race to commercialize such language models may also move more quickly than efforts to add guardrails or limit misuses.
Perhaps the most pressing worry about AI language models is how they might be misused. Because the models can churn out convincing text on a subject, some people worry that they could easily be used to generate bogus reviews, spam, or fake news.
“I would be surprised if disinformation operators don’t at least invest serious energy experimenting with these models,” says Micah Musser, a analysis analyst at Georgetown University who has studied the potential for language models to spread misinformation.
Musser says research suggests that it won’t be possible to use AI to catch disinformation generated by AI. There’s unlikely to be enough information in a tweet for a machine to judge whether it was written by a machine.
More problematic kinds of bias may be lurking inside these gigantic language models, too. Research has shown that language models trained on Chinese internet content will reflect the censorship that shaped that content. The programs also inevitably capture and reproduce subtle and overt biases around race, gender, and age in the language they consume, including hateful statements and ideas.
Similarly, these huge language fashions could fail in shocking or surprising methods, provides Percy Liang, one other laptop science professor at Stanford and the lead researcher at a new center devoted to finding out the potential of highly effective, general-purpose AI fashions like GPT-3.