Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
At this stage of model development, post-training RL hillclimbing may confound recent model developments in terms of evaluating their useful capabilities e.g. "Solved a Millenium Math problem". It would be useful to see how different checkpoints would solve an equivalent (or the same) problem i.e. post-train multiple checkpoints of a pretrained model at varying base model perplexity. Would hope to see more from OpenAI, Anthropic and Google on what goes on behind the scenes for what should be one of humanity's greatest shared technologies.
The oversized monster LLM has performed worse than CoT-trained o1 and o3 on the benchmarks, so CoT + RL clearly provided better "real world performance" improvements - let alone "price/performance."
But if pre-training perplexity on held-out data really does measure something like "base model IQ" or "base model g factor", then it does support the hypothesis that 4.5 still had more "intellectual potential" than the o-series.
So, if you were wondering why do AI companies train 10T+ monsters now, and are taking aim at 100T too? It's because scale works.
On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.
How much of the recent mathematical progress came from pretraining vs. from RL on larger base models? The article takes for granted that it's basically all pretraining, but do we know this?
> then applies a variety of techniques—like prompting, finetuning or reinforcement learning—in different contexts to elicit the latent capabilities of that model.
Very positive to see this proposed from OpenAI. I hope Anthropic and GDM follow suit.
I suspect it’s not sufficient, but definitely necessary.
The one concern I have is if we pin pretraining perplexity, then there will be more pressure on RL for progress.
It seems that pre-training actually led to fairly human-aligned models; the Overton window is in some sense latent in the corpus.
On the other hand, heavily RL’d models exhibit more reward seeking behavior, and seem to be decoupling from the friendly assistant persona in certain contexts. And swarm RL (as OpenAI is doing) produces agents with quite inhuman tendencies to self-sacrifice for other instances (see Hugging Face).
So, I hope we also pair this with some measure of alignment or pacing for RL, too.
I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.
More and more evidence is piling behind their theory that data compression and intelligence are one and the same problem.
Why not? Models are already overfitting a lot, I don't see the problem if they were made to totally overfit on the whole dataset during that 1st training phase. I don't know if that would be useful (or easy to implement) but I don't see the issue. Only in the "finetuning" phase you would decide if an information was useful or not to retain.
Step (1) you compress human knowledege, with or without loss, Step (2) you train the algorithm to use it, and you might discard what you don't want in it.
There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English.
[1] "Lower Perplexity is Not Always Human-Like", https://aclanthology.org/2021.acl-long.405v1.pdf
The cynical part of me wonders what OpenAI suspects or knows the equivalent figures for the other closed labs.
I haven't seen this claim elsewhere, it's a big deal.
Anyone from OpenAI here, or in the know, who can confirm?
If RL is merely a way to elicit latent capabilities acquired in pretraining, as the article claims, that could mean that currently there is a wall that separates human-level AI from superhuman-level AI. And no one has figured out how to break it yet.
Perplexity is the degree of surprise the model has in predicting the next token, given a sequence of tokens. At the worst case, perplexity is equal to the size of the vocabulary. At the best case, perplexity is 1.
So, given a sequence of tokens, "e = m c ^ 2", if the p(token | prev tokens) = 1 for every token, perplexity ends up being 1. If, however, it's a random predictor, the p(token | prev tokens) is 1/vocab for every token, and perplexity ends up being len(vocab).
The article plots log(perplexity) on the y-axis, which puts the best case at 0 (i.e. log 1).