It's frustrating for sure. And my main concern with LLMs is that people are inclined to ascribe them what they think of as human-like intelligence (or other characteristics) basically because LLMs were trained on imitating human language and the metrics of their success in training and tuning is largely defined by how well they imitate that language. I think it's harder for people to see why an LLM's capability with language doesn't mean much inherently if they primarily interact with chatbots and treats the chatbot persona as an entity unto itself rather than a prompt for a persona atop model training.

As it is, I wasn't imbued with skepticism of it myself. I had moments some years ago where I was new to LLMs, was mainly interacting with them in chatbot form, and tried to "test" them in various ways to see if I could see signs of awareness/intelligence/etc. Others helped bring me around to being more skeptical about it, along with just learning more about what models are / are doing on a functional level.

I do think that when an LLM is able to, with the right prompting, work out a mathematical proof that hasn't been solved yet (or come very close - I forget quite how that story went), that's undeniably worth study as to why it's even possible in the first place. Stuff like that is a "wow" moment. But if we jump to "it's alive" or other similar language, we're skipping many steps of possibility along the way and that skipping is something a lot of us have been primed for with decades of media based on conceptions of AI having a "ghost in the machine", "developing awareness out of inexplicable circumstances," etc. Media has primed people to figure that a computer turning into something like the human mind is not only possible, but also likely while likely to happen in a way that is beyond comprehension. Which is an infantilizing intellectual position for us to be in.

The way I see it, the much more sober outlook is that we're going to keep hitting hard limits on what we can get AI to do until we understand the human mind better. But in order to understand the human mind better, we have to get past the framework of thought that humans are like computers, else we still start to circle the drain; dragging down understanding of humans in the pursuit of AI, which will damage our understanding of both.