Even without external libraries, it routinely comes up with solutions that are at least 3-4x the complexity and LOC than how I would implement it.

I watched it remove a JSON library include from the code because it wasn't used yet, and then in the very next prompt where we needed our again, it attempted to write a basic parser from scratch...

I guess the AI isn't iterating over its "ideas" and code yet. I come up with overcomplicated and overcomplex code too. But my natural neural network keeps reevaluating the code while I think about it, write it down, test it, and often even come back to it later after I moved on to another part of the codebase.
It looks to me like AI is using the infamous waterfall method to write code. It does the "thinking" phase and does bugfix iterations when compilation or tests fail. But never does it question the design it came up with after it wrote it down. It doesn't reevaluate whether the last round of fixes made tests work that test irrelevant implementation details. It doesn't question whether there really need to be four methods sharing 90% of their body for doing the basically the same thing only subtly different.
I hold the suspicion that AI will be as slow as humans when it becomes as reliable because what makes us reliable is the redundancy in our thinking processes. The constant reevaluation in subtly different states of mind helps to catch design flaws and silly mistakes.

Reevaluation of finalized code is probably not something you can just put into memory. You would need to modify the harness to automatically inject a new prompt triggering reevaluation of the design and implementation after each step. This would likely increase token use and execution time by an order of magnitude. And if you do it by spawning fresh agents, there would be a high chance of having the AI go in circles, switching between a few different versions of the same shitty design or implementation. So you need to keep the context, which grows each iteration. So context rot may become a real problem. The AI might deviate from the initial goal just like some human devs tend to do...

And natural neural networks learn on the job. They adapt to the codebase and industry they are working with/in. claude.md and memory text files help a lot. But they are still sometimes ignored. And I don't know why. Also, human rules are almost always somewhat fuzzy. Long lines are to be avoided - but unit tests and that one massive dict of dicts really are a lot more readable when ignoring that rule. There might be a lack of good training material on what makes code readable for humans.

Possibly motivated to do more complex problems to spend more tokens?

That's how i always describe them. Over-eager junior devs that are trying to impress you with how fast they can get stuff done. Have a lot of book learning but don't know when to apply certain things. Sometimes, i feel like it should really only be a tool for senior devs because it feels a lot like mentoring a team of noobs