Great read! 👏 Really captures the craziness of the AI race right now. The stakes are huge, and the amount of money being poured in is wild. Exciting times ahead!
LLM layer is a key battle. That is why you have Musk suing OpenAI and Zuckerberg giving away his reflection/Llama database. The challenge becomes that the best way to attack this layer and who will dominate is not clear. Or even if anybody CAN dominate. For example, Matt Shumer and Sahil Chaudhary just took the Meta model, added in reflection-tuning (think of it as fine tuning), and suddenly he has the fastest LLM out there. (If it gets replicated.) However, just the thought that a couple of guys can get up a model based on Meta to break records feels like we need a volatility scale for AI. Right now we'd be at 600.
Sarah, I've just recently discovered your blog (through the AI Daily Brief podcast by NLW) and it's positively fantastic! Please continue sharing your insights and opinions with us.
Let's assume that LLM GPT (n+1) variants do materially increase in reasoning and language skills which enable them to take on tasks with increasing complexity.
Will the COGS of servicing those "more complicated" tasks create a market dynamic where older and less performant LLMs are good enough, especially if they are radically cheaper to operate?
Maybe that implies some sort of distribution (bell curve?) where most tasks don't need state of the art LLMs, shifting usage and therefore dollars away from the highest performant LLMs. We might even get to a stage where open source models with near zero marginal cost to operate perform the vast majority of future task work.
If that future comes to pass, maybe the business models of LLMs will reflect the very nature of current human labor. i.e. We pay interns a lot less than a PhD. Maybe an LLM that can solve a sophisticated analytical problem will need to have a very different pricing / compensation model than one that can only perform more rudimentary and repeatable work.
Totally agree. The big question is how much Zuckerberg is willing to invest in training Llama. Hundreds of millions sure. Billion. Ok. $10? $100? The people monetizing the models will have all the incentive in the world. I wonder what happens with Meta.
Hi Sarah, thanks again for sharing your thoughts on such an important topic. Recently, Yann LeCun from meta, one of the most talented AI researchers, mentioned the fact that AGI will not be based on GenAI LLMs. I was wondering if this assertion can be a threat to the initial 600b$ bet?
thank you for the question! You are right to ask it. It's a major risk to that initial bet the big stack players are making. It could end up being the case that there is a far more efficient way to build these AI models that we haven't yet discovered. But in the absence of that breakthrough, all in on transformers!
"Until we reach an efficient frontier of the marginal value of adding more compute, or people lose faith in the transformer architecture, this is a contest now of “not blinking first”." - it could be the case that the transformer architecture doesn't take us much further than we are today, in which case that would be the threat to the initial $600b bet.... but the industry is certainly leaning towards the more optimistic case 🙂
fantastic article. crazy that GPT3.5 pricing is 2.5% of what it was 2 years ago. talk about an accelerated Moore's Law
Great read! 👏 Really captures the craziness of the AI race right now. The stakes are huge, and the amount of money being poured in is wild. Exciting times ahead!
LLM layer is a key battle. That is why you have Musk suing OpenAI and Zuckerberg giving away his reflection/Llama database. The challenge becomes that the best way to attack this layer and who will dominate is not clear. Or even if anybody CAN dominate. For example, Matt Shumer and Sahil Chaudhary just took the Meta model, added in reflection-tuning (think of it as fine tuning), and suddenly he has the fastest LLM out there. (If it gets replicated.) However, just the thought that a couple of guys can get up a model based on Meta to break records feels like we need a volatility scale for AI. Right now we'd be at 600.
Sarah, I've just recently discovered your blog (through the AI Daily Brief podcast by NLW) and it's positively fantastic! Please continue sharing your insights and opinions with us.
Let's assume that LLM GPT (n+1) variants do materially increase in reasoning and language skills which enable them to take on tasks with increasing complexity.
Will the COGS of servicing those "more complicated" tasks create a market dynamic where older and less performant LLMs are good enough, especially if they are radically cheaper to operate?
Maybe that implies some sort of distribution (bell curve?) where most tasks don't need state of the art LLMs, shifting usage and therefore dollars away from the highest performant LLMs. We might even get to a stage where open source models with near zero marginal cost to operate perform the vast majority of future task work.
If that future comes to pass, maybe the business models of LLMs will reflect the very nature of current human labor. i.e. We pay interns a lot less than a PhD. Maybe an LLM that can solve a sophisticated analytical problem will need to have a very different pricing / compensation model than one that can only perform more rudimentary and repeatable work.
Totally agree. The big question is how much Zuckerberg is willing to invest in training Llama. Hundreds of millions sure. Billion. Ok. $10? $100? The people monetizing the models will have all the incentive in the world. I wonder what happens with Meta.
Agree that most tasks don't need SOTA LLMs, just like the distribution of human labor.
I wonder how an intern level LLM will be valued in the future with near zero marginal cost.
Has there been any actual reports from these AI labs saying any of their AI investments have returned a profit yet?
Running out of electricity will probably happen sooner than later. Also, so regulations can hopefully curb this unsettling AI arms race.
https://epochai.org/blog/can-ai-scaling-continue-through-2030
Hi Sarah, thanks again for sharing your thoughts on such an important topic. Recently, Yann LeCun from meta, one of the most talented AI researchers, mentioned the fact that AGI will not be based on GenAI LLMs. I was wondering if this assertion can be a threat to the initial 600b$ bet?
thank you for the question! You are right to ask it. It's a major risk to that initial bet the big stack players are making. It could end up being the case that there is a far more efficient way to build these AI models that we haven't yet discovered. But in the absence of that breakthrough, all in on transformers!
"Until we reach an efficient frontier of the marginal value of adding more compute, or people lose faith in the transformer architecture, this is a contest now of “not blinking first”." - it could be the case that the transformer architecture doesn't take us much further than we are today, in which case that would be the threat to the initial $600b bet.... but the industry is certainly leaning towards the more optimistic case 🙂
are you thinking of any companies or use cases in particular?
not sure I understand your question