Tech paper

Language Models are Few-Shot Learners (GPT-3)

Tom B. Brown, Benjamin Mann, Nick Ryder, et al. · 2020

Read this as

New to this. Just the gist and why it matters.

How deep

The gist, about 2 minutes.

In one line

The short of it

Make a language model enormous (175 billion parameters) and it starts doing new tasks from just a few examples in the prompt, no retraining. This paper is why prompting became the way we use AI.

Why it exists

The problem it tackles

Before GPT-3, using a model for a new task usually meant collecting a labeled dataset and fine-tuning. That is slow and expensive. The authors asked whether sheer scale could let a single model handle many tasks with only a few examples shown in the prompt.

In plain words

The key idea

Train one very large model to predict the next word on a huge slice of the internet, then give it a task by simply describing it and showing a couple of examples in the prompt. The model figures out the pattern on the fly, which the authors call in-context learning.

Think of it like: Like a widely-read generalist who, shown two or three examples of an unfamiliar puzzle, infers the rules and solves the next one, without going back to school.

What they did

The method

The team trained a 175-billion-parameter transformer on hundreds of billions of words and tested it on dozens of tasks using zero, one, and few example prompts.

What they found

The results

With just a few prompt examples, GPT-3 matched or beat fine-tuned models on many tasks, and revealed a smooth scaling trend.

Does it hold up?

The honest part. What is solid, and where to be careful.

Possible overlap between test benchmarks and training data (contamination) complicates some claims.
Still fails at parts of multi-step reasoning and can produce fluent falsehoods.
Enormous training cost and documented bias in outputs.

⚠ "Few-shot learning" is a strong framing; much of the behavior is pattern completion, not learning in the usual sense.

So what, for you

If you are a curious newcomer

This is why you can just ask an AI to do something and show it an example, instead of training it. GPT-3 showed that a big enough model picks up tasks from the prompt alone.