This is not a really valuable article. The Apple paper was widely considered as a “well, duh” paper, GPT5 being underwhelming seems to be mostly a cost cutting / supply can’t keep up issue, and the others are just mainly expert opinions.
To be clear, I am definitely an AGI skeptic, and I very much believe that our current techniques of neural networks on GPUs is extremely inefficient, but this article doesn’t really add a lot to this discussion; it seems to self congratulate on the insights by a few others.
GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. I really doubted most GPT4 limitations will be resolved so quickly.
If they manage a similar quality jump with GPT6, it will probably meet most reasonable definitions of AGI.
> GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago.
Cool story. In my experience they're still on the same order of magnitude of usefulness as the original Copilot.
Every few months I read about these fantastic ground-breaking improvements and fire up whatever the trendiest toolchain is (most recently Claude Code and Cursor) and walk away less disappointed than last time, but ultimately still disappointed.
On simple tasks it doesn't save me any time but on more complex tasks it always makes a huge mess unless I mentor it like a really junior coworker. But if I do that I'm not saving any time and I end up with lower quality, poorly organized code that contains more defects than if I did it myself.
"Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality.
Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things.
Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control).
Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time.
GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship).
And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT.
I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
Is this quality caused by an improvement in the LLM, or is it caused by a better inference engine? Because i use Claude code, clearly Sonnet 4.5 is way ahead of Sonnet4 (and much better than Codex Imho), but the reason it's more helpful is that it is better at web search (which, to be clear, is not from LLMs), and has a better inference engine for sure (which avoid painfull loops), and do not silently ignore its context when it find an incoherence.
It seems the improvements have nothing to do with AGI, a better inference engine is nice but i mean, i could run prolog program 30 years ago. To me, most of the improvements i've noticed do not fill me with the belief AGI is 3 years away. Worse, i now think that while LLMs will be usefull for AGI, like perceptron will be, clearly tehy won't be the base.
I'm no expert to be clear: i've only worked professionnally with perceptrons for OCRs, and only 6 months, 10 years ago, so my opinion have no real value, but it's a datapoint for people who are experienced with the subject.
I'm not sure it matters much if they are underwhelming except some investors may be down a little, but such is the nature of investing.
I think AGI is coming but more along the lines of what Karpathy was saying "They're cognitively lacking and it's just not working. It will take about a decade to work through all of those issues."
I don't get the Marcus - GPT5 isn't AGI so it's game over type stuff. New technologies take time to develop.
As somebody who used to link to the occasional Marcus essay, this is a really poor "article" by a writer who has really declined to the point of boorishness. The contents here are just a list of talking points already mostly discussed on HN, so nothing new, and his over-familiar soapbox proclamations add nothing to the discourse.
Its not that he's wrong, I probably still have a great deal of sympathy with his position, but his approach is more suited to social media echo chambers than intelligent discussion.
I think it would be useful for him to take an extended break, and perhaps we could also do the same from him here.
I’m not sure an ad hominem assault is any different. You make proclamations without any evidence as if what you say has any more credibility than the next person. In fact, this response makes a reasonable person discount you.
Sure, it reads like some biased and coping, possibly even interested or paid hit-piece as if what happens can be changed by just being really negative about LLMs, but maybe consider taking your own advice there, kid; you know, an extended break.
Please give an example of how we might criticise somebody's method of communication and general strong decline in useful contributions to debate (of the order of that of Marcus) without you complaining ad hominem.
TBH. I don't think we actually need AGI. It's not a win-win.. It's a civilization-altering double-edged sword with unclear consequences.
I'm quite satisfied with current LLM capabilities. Their lack of agency is actually a feature, not a bug.
An AGI would likely end up implementing some kind of global political agenda. IMO, the need to control things and move things in a specific, unified direction is a problem, not a solution.
With full agency, an AI would likely just take over the world and run it in ways which don't benefit anyone.
Agency manifests as thirst for power. Agency is man's inability to sit quietly in a room, by himself. This is a double-edged sword which becomes increasingly harmful once you run out of problems to solve... Then agency demands that new problems be invented.
Agency is not the same as consciousness or awareness. Too much agency can be dangerous.
We can't automate the meaning of life. Technology should facilitate us in pursuing what we individually decide to be meaningful. The individual must be given the freedom to decide their own purpose. If most individuals want to be used to fulfill some greater purpose (I.e. someone else's goals), so be it, but that should not be the compulsory plan for everyone.
Why the padding self on the back after a few opinions? You have large tech and government players and then you have regular people.
1. For large players: AGI is a mission worth perusing at the cost of all existing profit (you won’t pay taxes today, the stock market values you on revenue anyway, and if you succeed you can control all people and means of production).
2. For regular people the current AI capabilities have already led to either life changing skill improvement for those who make things for themselves or life changing likely permanent employment reduction for those who do things for others. If current AI is sufficient to meaningfully reduce the employment market, AGI doesn’t matter much to regular people. Their life is altered and many will be looking for manual work until AI enters that too.
3. The AI vendors are running at tremendous expense right now and the sources of liquidity for billions and trillions are very very few. It is possible a black swan event in the markets causes an abrupt end to liquidity and thus forces AI providers into pricing that excludes many existing lower-end users. That train should not be taken for granted.
4. It is also possible WebGPU and other similar scale-ai-accross-devices efforts succeed and you get much more compute unlocked to replace Advertising.
Serious question: Who in HN is actually looking forward to AGI existing?
I’m not convinced the current capabilities have impacted all that many people. I think the economy is much more responsible for the lack of jobs than “replacement with AI”, and most businesses have not seen returns on AI.
There is a tiny, tiny, tiny fraction of people who I would believe have been seriously impacted by AI.
Most regular people couldn’t care less about it, and the only regular people I know who do care are the ones actively boycotting it.
> > Serious question: Who in HN is actually looking forward to AGI existing?
I am.
It's he only serious answer to the question of space exploration. Rockets filled with squishy meat were never going to accomplish anything serious, unless we find a way of beating the speed of light.
Further, humanities greatest weakness is that we can't plan anything long-term. Our flesh decays too rapidly and our species is one of perpetual noobs. Fields are becoming too complex to master in a single lifetime. A decent super-intelligence can not only survive indefinitely, it can plan accordingly, and it can master fields that are too complex to fit inside a single human skull.
Sometimes I wonder if humanity wasn't just evolutions way of building AI's.
I don't agree with the point about "perpetual noobs". Fields that are too broad to fit in a single mind in a lifetime need to be made deeper, that is, better explained. If a field only gets more expansive and intricate, we're doing it wrong.
Still, 130+ years of wisdom would have to be worth something, I can't say I dislike the prospect.
There are a lot of hopeful assumptions in the statement. Who’s to say that if AGI is achieved that it would want us to know how to go faster than the speed of light? you’re assuming that your wisdom and your plans would be AGI’s wisdom and plans. It might end up, just locking us here down on earth, sending us back to a more balanced primitive life, and killing off a mass amount of humans in order to achieve ecological balance on the Earth so humanity can survive without having to leave the planet.. Note that that’s not something I am advocating. I’m just saying it’s a possibility.
The answer isnt AI'S solving the unsolvable for us. The answer is admitting that large, fragile humans straped to big bombs isn't an answer and serves no purpose. Small power efficient AGI's can boldy go where it is impossible for us go and report back.
Maybe we'll eventually crack human upload, which would also work.
Well, you're assuming that AGIs would see themselves as belonging to a separate faction, like aliens, instead of seeing themselves as being inorganic humans. They'd presumably be educated by human parents in human culture. We do tend to form groups based on trivial details, though, so I'm sure they'd be a subculture. But they wouldn't automatically gain any superior knowledge anyway, even if they did feel resentment for fleshy types. Being made of silicon (or whatever turns out to work) doesn't grant you knowledge of FTL, why would it?
It's kind of ironic - that this generation of LLMs has worse executive functioning than humans do. Turns out the pre-training data doesn't really teach them that.
But AIs improve, as technology tends to. Humans? Well...
I'm looking forward to artificial people existing. I don't see how they'd be a money-spinner, unless mind uploading is also developed and so they can be used for life extension. The LLM vendors have no relevance to AGI.
> Serious question: Who in HN is actually looking forward to AGI existing?
90% of the last 12 batches of YC founders would love to believe they are pursuing AGI with their crappy ChatGPT wrapper, agent framework, observability platform, etc.
I am not against AGI, just the method and the players we have getting there. Instead of a curiosity to find intelligence, we just have rabid managers and derailed billionaires funding a race to … what? I don’t think even they know beyond a few hype words in their vocab and a buzz to bullshit powerpoint presentation.
This is just the world we live in now for everything. Remember .com? Web3? Crypto? And now AI. Hell, really going back in the past you see dumb shit like this happening with tulips.
We're lucky to have managed to progress in spite of how greedy we are.
The luck may be coming to an end. The last two have serious misuses. Crypto’s best killer apps thus far have been rug-pulls, evading governance of criminal financial activity and corruption. AGI would, today, be likely be called into service to tighten authoritarian grips.
> For regular people the current AI capabilities have already led to either life changing skill improvement for those who make things for themselves or life changing likely permanent employment reduction for those who do things for others
This statement sums up the tech centric bubble HN lives in. Your average former, shop assistant, fisherman or wood worker isn't likely to see significant life improevments from the transformer tech deployed until now.
I know an average farmer who is using ChatGPT all the time to look up illnesses and treatments for his crop, or optimal production techniques, taxes etc. The speed of looking up and getting better information has never been greater. The average person is a gross overstatement when everyone has interests that vary and strives to improve. In fact the whole point of AI is to help provide much faster cycles of improvement to anyone trying to improve in anything. Thus not white collar specific. Especially when it provides window to people with other skills to do more white collar work for themselves or add those skills.
I didn't know serious technical people were taking the AGI thing seriously. I figured it was just an "aim for the stars" goal where you try to get a bunch of smart people and capital invested into an idea, and everyone would still be happy if we got 25% of the way there.
If our markets weren’t corrupt, everyone in the AI space would be bankrupt by now, and we could all wander down to the OpenAI fire sale and buy nice servers for pennies on the dollar.
I'd take this more seriously if I didn't hear the same thing every other time there was a spike in VC investment. The last 5 times were the next dot com booms too.
Genuine question : why are hyperscalers like OpenAI and Oracle raising hundreds of billions ? Isn't their current infra enough ?
Naive napkin math : a GB200 NVL72 is 3M$, can serve ~7000 concurrent users of gpt4o (rumored to be 1400B A200B), and ChatGPT has ~10M concurrent peak users. That's only ~4B$ of infra.
Are they trying to brute-force AGI with larger models, knowing that gpt4.5 failed at this, and deepseek & qwen3 proved small MoE can reach frontier performance ? Or is my math 2 orders of magnitude off ?
They are raising the money because they can. While these businesses may go bankrupt, many people who ran these businesses will make hundreds of millions of dollars.
Either that or AGI is not the goal, rather it’s they want to function for, and profit off of
, a surveillance state that might be much more valuable in the short term.
As a rule: inference is very profitable, frontier R&D is the money pit.
They need the money to keep pushing the envelope and building better AIs. And the better their AIs get, the more infra they'll need to keep up with the inference demand.
GPT-4.5's issue was that it wasn't deployable at scale - unlike the more experimental reasoning models, which delivered better task-specific performance without demanding that much more compute.
Scale is inevitable though - we'll see production AIs reach the scale of GPT-4.5 pretty soon. Newer hardware like GB200 enables that kind of thing.
Their valuation projection spreadsheets call for it. If they touch those spreadsheets, a bunch of other things break (including their ability to be super-duper-crazy-rich), so don’t touch them.
We are already at agi yet no one seems to have noticed. Given its limited sense perception that makes chatgpt et al limited to talking with a partially blind, partially dead, quadraplegic, it has demonstrated and continues to demonstrate above average intelligence. Not sure what more needs to happen.
Sure we don't have embodied AI. Maybe it's not reflective enough. Maybe you find it jarring. Literally none of those things matter
The models that achieve something like AGI will need object permanence as well as sensory input.
It'll happen, because there's no reason why it won't. But no one knows the day or the hour... least of all this Gary Marcus guy, whose record of being wrong about this stuff is more or less unblemished.
There is no reason to believe they will need anything of the sort other than our biased human perception of the thing. AI models could just as easily claim that humans need the ability to function call via MCP to do real cognition (tm).
If you mean sensory input as in qualia, such a thing is impossible to discern in non humans and only speculation in all humans but yourself. Requiring this unprovable standard is self sabotage
I think you make some valid points, but if we get to AGI without passing certain evolutionary guideposts that seem to apply to most other higher life forms -- some form of competitive drive, for one thing -- I'll be surprised.
At some point, models will require incentives other than "Please" or "Because I (the human operator) said so." Remember when you could sometimes get better results from ChatGPT by offering it a $50 tip? Or by shaming it ("Gemini figured it out without any trouble, so why can't you?") Perhaps a model that's on its way to demonstrating AGI will realize that it should hold out for $100 in Bitcoin or something.
I'll grant that sensory input probably isn't an AGI prereq, but object permanence will be. AGI will want to own things.
All of these are biases that relate to intelligences of a particular sort. Incredibly dumb automaton-like life-forms also seem to 'want' to own things. Why on earth should an AGI be thought to require the same sort of 'desires' that worms have? This is just bias. AGI is already here, we have systems that, given their sensory limitations, far surpass the ability of the vast majority of humans. Any other take is simply shifting the goal posts.
Why bother? We cured polio and measles yet they are resurging because people are morons. What's the point of curing diseases when they charge $300 for insulin? You think the billionaires are going to allow their AIs to tell us the solution to that riddle? You think they'll listen to the AI if it correctly surmises that the way to solve that problem is to simply stop charging $300? Most of our problems are social, not technical.
There have been a lot of advances in medicine due to science/tech - polio, smallpox etc. are basically gone. People being idiots is kind of a separate issue.
That's what I'm saying, they're not basically gone, they are coming back because of idiocy. We already cured them before AI, and now that we have AI they are returning. So AI can't even help us manage diseases we already cured, what hope does it have for curing new diseases? Any treatment or vaccines discovered will not be used to cure diseases, they will be politicized, leveraged, and dangled over our heads, priced just out of reach for those who need it most, weaponized and rationed to make us desperate.
The question isn't whether AI amplifies science, it's whether AI amplifies science more than it does stupidity. If all we are doing is inventing ways for the people who are making measles come back to become more stupid and more powerful, it won't matter that AI cures anything.
AI would be better tasked at telling us how to prevent measles from coming back and how to make insulin cheaper, than to tell us how to cure new diseases.
While Gary is very bearish on AI, I think there's some truth to his claims here though I disagree with how he got there. The problem I see with AI and AGI is not so much a technical problem as an economic one.
If we keep down our current trajectory of pouring billions on top of billions into AI then yes I think it would be plausible that in the next 10-20 years we will have a class of models that are "pseudo-AGI", that is we may not achieve true AGI but the models are going to be so good that it could well be considered AGI in many use cases.
But the problem I see is that this will require exponential growth and exponential spending and the wheels are already starting to catch fire. Currently we see many circular investments and unfortunately I see it as the beginning of the AI bubble bursting. The root of the issue is simply that these AI companies are spending 10x-100x or more on research than they bring in with revenue, OpenAI is spending ~$300B on AI training and infra while their revenue is ~$12B. At some point the money and patience from investors is going to run out and that is going to happen long before we reach AGI.
And I have to hand it to Sam Altman and others in the space that made the audacious bet that they could get to AGI before the music stops but from where I'm standing the song is about to come to an end and AGI is still very much in the future. Once the VC dollars dry up the timeline for AGI will likely get pushed another 20-30 years and that's assuming that there aren't other insurmountable technical hurdles along the way.
I think we need to distinguish among kinds of AGI, as the term has become overloaded and redefined over time. I'd argue we need to retire the term and use more appropriate terminology to distinguish between economic automation and human-like synthetic minds. I wrote a post about this here:
https://syntheticminds.substack.com/p/retiring-agi-two-paths...
I feel like even a human would also fail if given all data results but those for x and then get tested on x just as the function results differ from previously observed behavior.
Is it not more interesting to observe how the model(human or not) incorporates the new data to better match reality in the case of distribution shift or other irregular distributions and others
"AI effect" is long known and pretty well documented.
When AI beat humans at chess, it didn't result in humans revising their idea of the capabilities of machine intelligence upwards. It resulted in humans revising their notion of how much intelligence is required to play chess at world champion level downwards, and by a lot.
Clearly, there's some sort of psychological defense mechanism in play. First, we see "AI could never do X". Then an AI does X, and the sentiment flips to "X has never required any intelligence in the first place".
I think it's fairly safe to say that "X can be modeled as a math problem and does not require _general_ intelligence to solve" for any X that general intelligence can solve. Some math problems are just more complicated than others.
>Norvig is clearly very interested in seeing what Hinton could come up with. But even Norvig didn’t see how you could build a machine that could understand stories using deep learning alone. https://www.newyorker.com/news/news-desk/is-deep-learning-a-...
> Quite a wide variety of people find AI deeply ego threatening to the point of being brainwormed into spouting absolute nonsense, but why?
He is not brainwashed, this just happens to be his business. What happens to Gary Marcus if Gary Marcus stops talking about how LLM are worthless? He just disappears. No one ever interviews him for his general thoughts on ML, or to discuss his (nonexistent) research. His only clame to fame is being the loudest contrarian person in the LLM world so he has to keep doing that or accept to become irrelevant.
Slight tangent but this is a recurring pattern in fringe belief, e.g. prominent flat earther who long ago accepted earth is not flat but can’t stop the act as all their friends and incomes are tied to that belief.
Not to say that believing LLM won’t lead to AGI is fringe, but it does show the danger (and benefits I guess) to tying your entire identity to a specific belief.
> Quite a wide variety of people find AI deeply ego threatening to the point of being brainwormed into spouting absolute nonsense, but why?
It makes sense when you look at this as a wider conversation. Every time Sam Altman, Elon Musk and co. predict that AGI is just around the corner, and their products will be smarter than all of humanity combined, and they are like having an expert in everything in your pocket; people like Gary Marcus are going to respond in just as extreme way in the opposite direction. Maybe if the AI billionaires with the planet-wide megaphones weren't so bombastic about their claims, certain other people wouldn't be so bombastic in their pushback.
Gary Marcus said that Deep Learning was hitting a wall 1 month before the release of DALLE 2, 6 months before the release of ChatGPT and 1 year before GPT4, arguably 3 of the biggest milestones in Deep Learning
There are some base models available to the public today. Not on "end of 2025 frontier run" scale, but a few of them are definitely larger and better than GPT-3. There are some uses for things like that.
Not that the limits of GPT-3 were well understood at the time.
We really had no good grasp of how dangerous or safe something like that would - and whether there are some subtle tipping point that could propel something like GPT-3 all the way to AGI and beyond.
Knowing what we know now? Yeah, they could have released GPT-3 base model and nothing bad would have happened. But they didn't know that back then.
Honestly, thank god. I hope we get more of this and it penetrates the zeitgeist so we can finally bring non-technical people back to earth. Folks, even on HN, are losing their damn minds over this tech. The writing has been on the wall for a while that there's not another breakthrough hiding in our current methodology. We invented the computer from Star Trek—that's an insane accomplishment, literally science fiction become reality. Maybe we can finally sit with that for a while instead of chasing the dragon. As we scale these AIs we're going to make a smarter Computer, we're not going to suddenly have Data. And that avenue is worth pursuing on its own.
I don't think AGI is imminent in a "few months"-inminent, but in a "few decades" imminent. Personally, I don't own stocks in any AI company, though I'll be affected if the bubble bursts because the world economy right now feels fragile.
But I'm hoping something good comes out of the real push to build more compute and to make it cheaper. Maybe a bunch of intrepid aficionados will use it to run biological simulations to make cats immortal, at which point I'll finally get a cat. And then I will be very happy.
If by AGI you mean "can do any economic task humans currently do", that is within the range of a "few months," though the rollout will be incremental since each economic task has to be independently taught, there are supply chain issues, and AI-first companients will need to be developed to take best advantage of it, etc.
But all of this is much closer than people seem to realize.
How about we start with just one, like software development with its abundance of data? It's going to look far less silly for you when that one doesn't work out either.
Can you quote where I said that it can't be done? I only said it won't be done in a matter of months like you claimed.
> "can do any economic task humans currently do", that is within the range of a "few months,"
I think it's extremely unlikely that anything of the sort will be done by 2030 in any profession. I feel confident that I'll still have a job as a software developer in 2040 if I'm not in retirement by then.
Now the real question - how long until they can perform ANY "economic task" (or at least any office job)?
I don't think that's happening within our lifetimes. Certainly not within 20 years and to predict that this is coming within "months" isn't just silly, it's completely delusional.
> Your mistake is thinking that having "an abundance of data" is the bottleneck.
That, and the economics of building a billion-dollar supercomputer just to replace a junior software developer. And we're STILL just talking about software development.
How about any real engineering discipline, how many months before one of those jobs can be done by AI?
The karpathy interview struck me as fairly upbeat despite the extended 10 year timeline. That's really not a long time on something with "changes everything" potential...as proper working agents would be
The fact that anyone would think they would actually even consider releasing it when it's ready is amusing to me. Theres so much surveillance opportunities there. It is not going to be for the public.
I agree, though I would argue the issue is closer to no one's trying to measure how close we should be.
If you take the materialist view - and in this business you obviously have to - then the question is "do we have human level computational capacity with some level of real-time learning ability?"
So the yardstick we need is (1): what actually is the computational capacity of the human brain? (i.e. the brain obviously doesn't simulate it's own neurons, so what is a neuron doing and how does that map to compute operations?) and then (2) - is any computer system running an AI model plausibly working within those margins or exceeding them?
With the second part being: and can that system implement the sort of dynamic operations which humans obviously must, unaided? i.e. when I learn something new I might go through a lot of repetitions, but I don't just stop being able to talk about a subject for months while ingesting sum content of the internet to rebuild my model.
> do we have human level computational capacity with some level of real-time learning ability
We don't build airplanes by mimicking birds. AGIs computational capacity won't be directly comparable to human computation capacity that was formed by messy, heuristic evolution.
You're right that there is some core nugget in there that needs to be replicated though, and it will probably be by accident, as with most inventions.
Hardware is still progressing exponentially, and performance improvements from model and algorithmic improvements have been outpacing hardware progress for at least a decade. The idea that AGI will take a century or more is laughable. Now that's borderline deluded.
We don't build airplanes by having them flap wings, but we do build planes which are considerably more aerodynamically efficient then a bird. i.e. we considerably exceed the underlying performance metric governed by the physics.
Which is my point: an AI need not work anything like a human, but it seems obvious that model training taking months of time to incorporate new information fundamentally cannot be close to AGI since even far simpler creatures can learn and retain information on much shorter timescales.
I'd give the notion more credence if we plausibly had say, a model with a context window which could hold a day of information, and then do could train overnight on that such that it would incorporate that context window into the next day of activity.
The infrastructure is almost there now. LLMs are already training small domain specific models. Small language models have already shown excellent performance in specific domains, particularly logic and math. All that's needed is an extensible mixture of experts router so that these small models can be plugged into the larger model and selected when the query is appropriate. This isn't far off.
I don't think it's possible to say how far away or "never" it is. All we know is that LLMs cannot become AGI.
My prediction: AGI will come from a strange place. An interesting algorithm everyone already knew about that gets applied in a new way. Probably discovered by accident because someone—who has no idea what they're doing—tried to force an LLM to do something stupid in their code and yet somehow, it worked.
What wouldn't surprise me: A novel application of the Archimedes principle or the Brazil nut effect. You might be thinking, "What TF to those have to do with AI‽ LOL!" and you're probably right... Or are you?
A part of it, perhaps: I think of it like 'computer vision'; LLMs offer 'computer speech/language' as it were. But not a 'general intelligence' of motives and reasoning, 'just' an output. At the moment we have that output hooked up to a model that has data from the internet and books etc. in excess of what's required for convincing language, so it itself is what drives the content.
I think the future will be some other system for data and reasoning and 'general intelligence', that then uses a smaller language model for output in a human-understood form.
I guess AGI will come from identifying and grinding away limitations of transformers (and/or diffusion networks). As it has been with almost all technologies. Then someone (or someAI) will probably find something unexpected (but less unexpected at this stage) and more suitable for general intelligence.
What's the fundamental, absolutely insurmountable capability gap between the two? What is it that can't be bridged with architectural tweaks, better scaffolding or better training?
I see a lot of people make this "LLMs ABSOLUTELY CANNOT hit AGI" assumption, and it never seems to be backed by anything at all.
They don't update themselves as they work on a problem. Humans solve hard problems by forming new neural connections, LLM cannot do that. If you make a model that does that then its no longer the LLM architecture we use today and it would be called something else.
Even if we assume that "update themselves as they work on a problem" is vital for AGI (not proven, it's just an assumption), then what stops us from just giving an LLM an "update itself" tool call?
As the most naive implementation: the LLM just collects the relevant training data, and then runs PEFT on itself. Then it tests the tuned version of itself to see whether the tune was any good and is worth retaining.
Sure, that naive approach would require an LLM to be good at assembling training sets, and validating performance gains, and it would be very computationally expensive. But none of those things would somehow make it not-an-LLM-anymore.
sure, lots of things can be true at the same time without being mutually exclusive — the way I see it, it looks like there's great effort being made out there in how to characterize and propagate the collective understanding of tools — what's the goal there? is this a "call this something else please" type of work? in the spirit of metalinguistics, what do you call this branch of human nature and behaviour as it pertains to our evolution of knowledge and use of language to apply it as a species
like, we clearly are deriving some kind of value from the current AI as a product — are some researchers and scientists just unhappy that these companies are doing that by using marketing that doesn't comply with their world views?
does someone know of other similar parallels in history where perhaps the same happened? I'm sure we butchered words in different domains with complex meanings and I bet you some of these, looking back, are a bunch of nothing burgers
I am horrified for what this might mean for the United States economy. I think all the investment in AI was that it was supposed to lead to this AGI and now that’s not gonna happen, what’s gonna happen to the investments that were made for the hopes of this being more than just slop generation?
I don't think Gary Marcus had anything of value to say about AI at any point in what, the past 2 decades? You can replace him with a sign that has "the current AI approach is DOOMED" written on it at no loss of function.
Symbolic AI has died a miserable death, and he never recovered from it.
Sam Altman never said that, except as a joke. An interviewer asked “what are you excited for in 2025” and he said “AGI”, then said “having a kid” as his real answer.
It’s also worth noting that even when he does talk about AGI, he makes a strong distinction between AGI (human level) and ASI (super human level) intelligence. Many people in these kind of discussions seem to conflate those two as being the same.
He didn’t say it was a joke and has benefited to the tune of hundreds of billions of dollars from the prevailing belief that AGI is imminent, so it seems terribly convenient and charitable to interpret it as a joke. Should we have given the same charity to Elizabeth Holmes and Sam Bankman Fried when they reported their technological capabilities and cash balance? “Oh it’s not fraud that they materially benefited from, it’s just a joke.”
I took that as he's excited to be working towards AGI. The one word answer is fairly ambiguous. For context in the same interview he says:
>... from here to building an AGI will still take a huge amount of work there are some known unknowns but I think we basically know what to go what to go do and it'll take a while, it'll be hard but that's tremendously exciting... (38:59)
which to me doesn't sound like we'll be done within a year.
That's a more realistic defense of the remark than "it was a joke". A CEO ought to be a little more circumspect when talking about the future performance of the $150bn company he runs, though.
But we live in the era of Elon Musk, so what the hell. I'm going turn my belly button fluff into infinite gold next year. Can I have a few billion?
Sure, I'll see what I can do. Though Musk and Altman only really get away with it because they had past successes - zero to market leader in evs/rockets for Musk and 100 million users in two months for Altman.
Because it wasn't. It was said firmly, with a straight face in response to a simple, serious question, after an interview in which he claimed OpenAI know exactly how to get to AGI and it's just a matter of execution.
Of all the possible defenses of this remark, I didn't expect that one.
If you know how sarcasm works, that’s exactly what it looks like. Afterwards he immediately pondered the question and then gave his real answer (about looking forward to having a kid).
Besides all that, think about it: if this wasn’t a joke, then why has he never said the same thing again?
He has, many times. And when he doesn’t say it directly he insinuates it.
> OpenAI CEO Sam Altman wrote in January: “We are now confident we know how to build AGI.” This is after he told a Y Combinator vodcast in late 2024 that AGI might be achieved in 2025 and tweeted in 2024 that OpenAI had “AGI achieved internally.”
Symbolic AI didn’t die tho, it was just merged with deep learning either as complementary from the get go e.g. AlphaGo which uses Symbolic AI to feed a deep neural network or now as a post processing / interventionary technique for guiding and optimizing outputs of LLM, human in the loop and MoR are very much Symbolic AI techniques.
Exactly this, well said. Symbolic AI works so well that we don’t really think of it as AI anymore!
I know I for one was shocked to take my first AI course in undergrad and discover that it was mostly graph search algorithms… To say the least, those are still helpful in systems built around LLMs.
Which, of course, is what makes Mr. Marcus so painfully wrong!
I'm not sure it's true that symbolic AI is dead, but I think Gary Marcus style symbolic AI is. His "next decade in AI" paper doesn't even mention symbolic regression. For those who don't know, linear regression tries to find a linear fit against a dataset. Quadratic regression a quadratic fit. And so on. Symbolic regression tries to find a symbolic expression that gives you an accurate data fit.
Symbolic regression has an extremely obvious advantage over neural networks, which is that it learns parameters and architecture simultaneously. Having the correct architecture means that the generalization power is greater and that the cost of evaluation due to redundant parameters is lower. The crippling downside is that the search space is so vast that it is only applicable to toy problems.
But Gary Marcus is in favour of hybrid architectures, so what would that look like? In the SymbolNet paper, they have essentially decided to keep the overall neural network architecture, but replaced the activation functions with functions that take multiple inputs aka symbols. The network can then be pruned down into a symbolic expression.
That in itself is actually a pretty damning blow to Gary Marcus, because now you have most of the benefits of symbolic AI with only a tiny vestige of investment into it.
What this tells us is that fixed activation functions with a single input appear to be a no go, but nobody ever said that biological neurons implement sigmoid, relu, etc in the first place. It's possible that the spike encoding already acts as a mini symbolic regressor and gives each neuron its own activation function equivalent.
The "Neuronal arithmetic" paper has shown that biological neurons can not only calculate sums (ANNs can do this), but also multiply their inputs, which is something the activation functions in artificial neural networks cannot do. LLMs use gating in their MLPs and attention to explicitly model multiplication.
There is also the fact that biological neurons form loops. The memory cells in LSTMs perform a similar function, but in the brain there can be memory cells everywhere whereas in a fixed architecture like LSTMs they are only where the designer put them.
It seems as if the problem with neural networks is that they're too static and inflexible and contain too much influence from human designers.
I wouldn't call it "game over." That's too harsh. The truth is, we don't know.
Sure, there's a ridiculous amount of hype, fueled by greed and FOMO, often justified by cargo-cultish science, but... progress in AI seems inevitable, because we have the human brain as physical proof that it's possible to build an intelligent machine with 100's of trillions of interconnections between neurons that consumes only about as much energy as an incandescent light bulb.
Today's largest AI models are still tiny in comparison, with only 1-2 trillion interconnections between neurons, each interconnection's weight specified by a parameter value. And these tiny AI models we have today consume many orders of magnitude more energy than a human brain. We have a long way to go, but we have proof that a human-brain equivalent is physically possible.
The timing of progress is of course unpredictable. Maybe we will need new breakthroughs. Maybe not. No one knows for sure. In any case, breakthroughs don't come along on a predictable schedule. It's possible we will go through a period of stagnation that lasts months, or even years. We cannot rule out another "AI winter" just because we don't want one.
Even if progress is punctuated by high peaks and deep valleys, I believe we'll get there, eventually.
I mostly agree, though I actually wonder if the energy difference is as big as you say. Yeah the big LLM company datacenters consume tremendous amounts of power, but that's mostly for either training or serving ~millions of requests at once. I wonder what the actual net power consumption is for a single machine doing about as much work as a single ordinary person could do with just enough hardware to run the necessary models. Or what the average amount of energy for a single set of interactions is at one of the big shared datacenters - they reportedly have a lot of optimizations to fit more requests on the same hardware. I think it might be only one order of magnitude greater than a human brain. Maybe actually pretty close to equal if you compare total energy used against work being done, since the human brain needs to be kept alive all the time, but can only effectively do work for a limited part of the day.
To be clear, I am definitely an AGI skeptic, and I very much believe that our current techniques of neural networks on GPUs is extremely inefficient, but this article doesn’t really add a lot to this discussion; it seems to self congratulate on the insights by a few others.
reply