Well, this kind of AI won't ever be useful as a programmer. It doesn't think. It doesn't reason. It cannot make decisions besides using a ton of computational power and enormous deep neural networks to shit out a series of words that seem like they should follow your prompt. An LLM is just a really, really good next-word guesser.

So when you ask it to solve the Tower of Hanoi problem, great it can do that. Because it saw someone else's answer. But if you ask it to solve it for a tower than is 20 disks high it will fail because no one ever talks about going that far and it flounders. It's not actually reasoning to solve the problem - it's regurgitating answers it has ingested from stolen internet conversations. It's not even attempting to solve the general case because it's not trying to solve the problem, it's responding to your prompt.

That said - an LLM is also great as an interface to allow natural language and code as prompts for other tools. This is where the actually productive advancements will be made. Those tools are garbage today but they'll certainly improve.