GPT-3, a machine learning model from OpenAI, has taken the world by storm over the last couple of weeks. Natural language generation, a branch of computer science focused on creating texts from a batch of data, entered a golden age with last year’s release of GPT-2. The release of GPT-3 last month only confirmed this. In this article, I want to take a look at why GPT-3 is such a huge deal for the machine learning community, for entrepreneurs, and for anyone working with technology.
What Is GPT-3?
GPT-3 is a 175 billion parameters Transformer deep learning model. That might sound complicated, but it boils down to an algorithm that was taught to predict the next word based on the sentence you input. After you provide a sentence and the algorithm fills in the gaps. For example, you could put in “How to successfully use content marketing?” and you would get a text on the subject of content marketing.
GPT stands for generative pre-training. The generative part of that term should be clear. You want the model to generate a text for you based on some input. Pre-training refers to the fact that the model was trained on a massive corpus of text and its knowledge of language comes from the examples it has seen before. It doesn’t copy fragments of texts verbatim, however. The process involves randomness due to the fact that the model tries to predict the next word based on what came before, and this prediction has a statistical component to it. All this also means that GPT-3 doesn’t truly “understand” the language it’s processing; it can’t make logical inferences like a human can, for instance.
GPT-3 doesn’t feature a real breakthrough on the algorithmic side. It’s more of the same as GPT-2, although it was trained with substantially more data and more computing power. OpenAI used a C4 (common crawl) data set from Google, which Google used in training their T5 model.
So why is GPT-3 amazing at all? Its transformative nature all boils down to its applications, which is where we can really measure its robustness.
GPT-3 Applications
Imagine you want to build a model for translation from English to French. You’d take a pre-trained language model (say BERT) and then feed an English word or sentence into it as date along with a paired translation. GPT-3 can perform this task and many others without any additional learning, whereas you’d need to fine-tune earlier machine learning models like BERT on each task. You simply provide a prompt (asking sentence or phrase):
“Translate English to French: cheese =>” to get “fromage”
Providing a command without additional training is what we call zero-shot learning. You gave no prior examples of what you wanted the algorithm to achieve, yet it understood that you wanted to make a translation. You could, for example, give “Summarize” as an input and provide a text that you wanted a synopsis of. GPT-3 will understand that you want a summary of the text without any additional fine-tuning or more data.
In general, GPT-3 is a few-shot learner, which means that you simply need to describe to it a couple of examples of what you want, and then it can figure out the rest. The most surprising applications of this include various human-to-machine interfaces, where you write in simple English and get a code in HTML, SQL, Python, or app design in Figma.
For example, this GPT-3 powered app lets you write “How many users have signed up since the start of 2020?” The app would then give you an SQL code: “SELECT count(id) FROM users WHERE created_at > ‘2020-01-01’” that does just that. In other words, GPT-3 allows you to make queries about spreadsheets using natural language — English in this case.
Another great GPT-3 powered app lets you describe a design you want in simple English (“Make a yellow registration button”) and get Figma files with the button ready to be implemented in your app or website.
There are plenty of other examples that feature GPT-3 translating from English to a coding language, making the interaction between humans and machines much easier and faster. And that’s why GPT-3 is truly groundbreaking. It points us toward new, different human-machine interfaces.
GPT-3 and Democratization of Technology
So what does GPT-3 offer entrepreneurs, developers, and all the rest of us? Simplicity and the increasing democratization of technology.
GPT-3 and similar generative models won’t replace developers or designers soon, but they will allow for wider access to technology, be that designing new apps, websites, or researching and writing about various topics. Non-technical people won’t have to rely on developers to start playing around with their ideas or even build an MVP. They can simply describe it in English as they would to a software house to get what they want. This could well drive down the costs of entrepreneurship as you would no longer need developers to start.
What does that mean to developers, though? Will they become obsolete? Not at all. Instead, they will move higher up the stack. Their primary job is to communicate with the machine to make it do the things that the developer wants. With GPT-3 and similar generative models, that process will happen much faster. New programming languages emerge all the time for a reason: to make programming certain tasks easier and smoother. Generative language models can help build a new generation of programming languages that will power up developers to do incredible things much faster.
All in all, the impact of GPT-3 over the next five years is likely to be increasingly democratized technology. These tools will become cheaper and more accessible to anyone, just like the widespread access to the Internet did 20 years ago.
Regardless of the exact form it takes, with GPT-3, the future of technology definitely looks exciting.
P.S. If you want to test models similar to GPT-3 right now for yourself, visit Contentyze, a content generation platform I’m building with my team.
