Here's the thing. What is it that an LLM does? I would argue, it sumarizes.

More or less, yeah. It produces text that reads like it would fit in its training model.

What do LLMs summarize, though..? The internet. Reddit.4chan.

Garbage in, garbage out.

Give it a corpus of text, and it can "understand" that data and extract "meaning" from it.

No it can't. It can produce text that looks like it fits with it, but it has no knowledge of what its training data, its prompt, or its output mean.

LLMs don't operate on meaning or information, they operate on the shape of texts.

The fact that they look like they work (as long as you aren't an expert on the supposed subject matter of their output) would tell us much more about the nature of language and its readers, if we bothered to look, than the apparent meaning of their outputs, but of course the capitalist mind couldn't care less about that, and given a dumb oracle that appears to give “good enough” answers can't help but sell it as a superintelligent one that always gives perfectly accurate and correct ones, no matter the consequences.

Is that useful?

Not really, no. Maybe as a substitute for lorem ipsum if it weren't so damn inefficient and morally unsustainable, but that's about it.

“But the one remaining overworked not yet fired expert can review the output!” you'll say, falling into the trap of treating the machine as a dumb but hard working intern instead of what it is.

Because the thing with interns is that when they produce garbage it looks like garbage, and the expert can immediately see it and teach the intern why it's garbage and how to fix it. And eventually the intern will become an expert, and review and train new interns, and society will progress.

But when an LLM produces garbage it's indistinguishable from correct output at first, second, and even third glance, even by the best experts.

Because LLMs might not have the slightest idea of what a text means, but if they're good at something it's at producing correct looking texts.

So the expert ends up wasting orders of magnitude more time trying to determine whether the output is safe or garbage... or simply gives up.

What's worse, the LLM won't learn. It won't become an expert capable of replacing the old one and reviewing and teaching new interns, or obviously LLMs.

The expert will be fired, or retire, or die of overworking and heartbreak, and there will be no one to replace them.

No one to write new content to train new LLMs on.

And even if LLMs were capable of producing new content instead of merely lossly regurgitating their training data, which they aren't, you can't train LLMs on their own output. The model collapses. Photocopy of a photocopy, JPEG of a JPEG.

At the point in history when we need progress the most, to face and survive our own hubris from past abuses of nature, science, society, everything will stagnate, and then collapse, and the ignorant wretches who'll have grown up in a world without teachers, not having to had to use their atrophied brains once in their lives, will ask their oracles “save us, what do we do!?” and the LLMs will reply “You're right, I was mistaken. I apologize for the confusion. Delegating thought and knowledge to a machine inherently incapable of either might have been a mistake. My bad.”.

Doesn't look very useful to me, really, quite the contrary... but I suppose I'm just not thinking in terms of the next quarter.