> Humans achieve their performance levels while only ever observing a tiny fraction of that training data
This is really the key detail and the hole in Gwern's argument that AGI is around the corner. You can't just compare the result. You also have to look at what it took to train the model and at what the model is actually doing.
If you look at GPT-3's output, it only superficially makes sense. Is there evidence of true understanding or is it just a really really good text generator?
Regardless of AGI though, I do think that models like this will eventually mean the end of social media and perhaps all wide open discourse on the Internet. When this stuff gets easy and cheap enough that spammers and propagandists can use it, it's over. How much money in compute/storage would it take to train GPT-3 to advocate for Donald Trump or Joe Biden all day long, or to shill products, or to just generate superficially comprehensible text to execute a kind of denial of service attack on a community?
I don't buy the "we have GPT-3, therefore we may soon have artificial general intelligence" (AGI) notion.
AGI might happen tomorrow; it might happen in decades, in centuries, or never. GPT-3 is basically a straightforward scaling of GPT-2, but I see no evidence that simply scaling GPT-2 or GPT-3 will lead to AGI. The problem is we don't know what else is needed.
Maybe the end of anonymous ungated social media, yes. We can still talk to humans and sources we know and trust. Probably that's what we need to do already.
given the idiosyncrasies of GPT style text generation, the cost of detecting generated text is likely to be orders of magnitude lower than the cost of generating the text. Such detection may also capture low-quality content on the internet and prove a boon for high-quality commentary.
This is really the key detail and the hole in Gwern's argument that AGI is around the corner. You can't just compare the result. You also have to look at what it took to train the model and at what the model is actually doing.
If you look at GPT-3's output, it only superficially makes sense. Is there evidence of true understanding or is it just a really really good text generator?
Regardless of AGI though, I do think that models like this will eventually mean the end of social media and perhaps all wide open discourse on the Internet. When this stuff gets easy and cheap enough that spammers and propagandists can use it, it's over. How much money in compute/storage would it take to train GPT-3 to advocate for Donald Trump or Joe Biden all day long, or to shill products, or to just generate superficially comprehensible text to execute a kind of denial of service attack on a community?