OpenAI Roadmap -- The price of GPU Shortages
What to expect from OpenAI moving forward -- 6/2/2023
The Facts
In a conversation with the team at HumanLoop, Sam Altman talked at some length about OpenAI’s short-term roadmap. The headline: most of their plans to improve their products are blocked by GPU access.
A quick overview of the features that are blocked by GPU shortages
The fine-tuning API for their newer models
The rollout of the GPT-4 32k context length model
The release of the GPT-4 multimodal model
Dedicated capacity
The result: OpenAI’s primary focus right now is improving the performance of GPT-4
Why it matters
The companies training the largest models currently have to make a tradeoff: use GPUs to serve or train models. Too much training and your product will be heavily limited (GPT-4 still has a really low rate limit); too much inference and you won’t have the capacity needed to experiment with bigger or better models.
There’s a lot to be experimenting with right now too! New fine-tuning methods are coming out every week, more efficient attention mechanisms have been tested in academia — all of those would improve the OpenAI product significantly. Testing these (especially at the scale of training a GPT-4 quality model) would severely limit their ability to serve their existing product.
My thoughts
A few bits —
Don’t expect much better models anytime soon from OpenAI. They’re stuck making their own product work.
There could be a weird version of the innovator’s dilemma plaguing OpenAI now — other teams building LLMs might have more bandwidth to train better models right now without the burden of any existing products.