I think you need to read this: https://aeon.co/essays/your-brain-does-not-process-information-and-it-is-not-a-computer

People have been doing this reductionist "humans are like the latest technological advancement" stuff for a long time. Gen AI is just another form of it. Creating an extremely rough computerized imitation of a conception of some neurons is not even close to recreating what humans are. Training an AI on human conversation so that it can do a really convincing p-zombie fake of human conversation does not make it any closer fundamentally to what a human is. People were getting drawn in by ELIZA, a very rudimentary chatbot, decades ago (circa 1960s), long before what chatbots are now: https://www.history.com/articles/ai-first-chatbot-eliza-artificial-intelligence-precursor-llms

Here's some snippets from that:

Forced to flee Nazi Germany as a young teen, Weizenbaum saw firsthand how the Nazis’ genocidal agenda had been aided by the scores of everyday people who looked the other way. He also recognized the powerful impact of the Nazis’ antisemitic propaganda, which dehumanized Jewish families like his in service of the Nazis’ eugenic effort to engineer a so-called perfect race. This campaign of dehumanization sowed complacency in the general public.

Given his background, Weizenbaum was especially troubled by the dehumanizing language some of his peers used when comparing human reasoning with the possibilities of AI. He bristled at MIT researcher Marvin Minsky’s description of human beings as “meat machines,” and chafed at leading technologists’ implication that future robots could present “a correction and improvement on nature.” Weizenbaum emphasized that this line of thinking had fueled some of the greatest atrocities in history.

“In the Nazi era, Jews were portrayed as vermin, a metaphor that legitimized mass murder,” Weizenbaum told German journalist Bernhard Pörksen in 1998. “Today…the idea that man is merely an information-processing machine that can be replaced by a robot is gaining substance and power.”

Sound familiar to what's going on right now in today's thinking about AI?

And a part that is not that far off in spirit from what I've been trying to warn about:

Though Weizenbaum was never dismissive of the future capabilities of technology like AI, the potential power of machines was never his chief concern. Instead, he feared that people’s projections onto these machines would hamper their better judgment. From the 1970s through the early 2000s, he was an advocate for technology ethics, urging scientists to confront the social consequences of their work.

The risk he foresaw was not that technologies like AI would evolve into civilization-destroying monsters, but that society would look upon them as false idols. The real threat to humanity came from within, he believed; technology simply gave it an outlet.

Weizenbaum repeatedly insisted to his friends: “I’m not an AI critic. I’m a critic of society."

When you tell me stuff like this:

LLMs are like Henry Molaison who got brain damage that disallowed him from forming long-term memories, so he could remember everything up to when he got that brain damage, but could only form short-term memories for a brief conversation after, because no one has figured out how to make re-training the neural network, something human brains do, efficient enough to do in real-time.

I know there is reason to be concerned. You make this dangerous reduction of saying LLMs are like a human being who got brain damage preventing long-term memories from forming. LLMs have almost NOTHING in common with a human being. The only thing in common I can think of is the attempt to create something approximating the best understanding of neurons we so far have, with complicated statistical vector math. That's it. You could try to point at something like "reasoning models" and say that's a sameness, but it's not because with the exception of logicians/mathematicians doing proofs or something, humans are not generally reasoning by writing out a bunch of text in real-time at high speed. In fact, for possibly most of our history as a species, we didn't even have writing or had very rudimentary forms with limited literacy. Certainly virtualized high speed writing with a keyboard and a computer is a very new invention.

It turns out that neuroscience has revealed that intelligence is really fucking boring and is just a bunch of neurons minimizing a cost function, but there is no “magic” in that and so people feel the need to constantly vaguepost about it lacks some sort of essence that they never specify.

I think the problem lies in the liberalism we're constantly surrounded with. Liberal media constantly ties intelligence/creativity to humanity, as in, more intelligent/creative people are the main characters, less intelligent/creative people are the npc subhumans.

So people need to always keep up an appearance of high intelligence, or their "humanity" is in danger. Now that machines have intelligence too, those people's "value as human being" is endangered. But instead of rejecting liberalism for tying a human's value to their intelligence, they desperately try to prove their value as humans by insisting their intelligence is some sort of supernatural magic.

Unintentional puns are the only good ones.