Not the author of the blog post, but let me adress that anyways.

I agree that for some purposes, what generative "AI" produces is functionally the same as a novelty. And tautologically, even random output of a machine can contain data that humans may interpret as a "new thought" or "new knowledge" of some kind. But "Knowledge" is an intrinsically social construct and no technology can enhance it on its own, in the greater scheme of things.

Re the first point, speaking as a mathematician: The proofs generated by "AI" thus far are mostly hardly comprehensible and on their own don't actually enhance our knowledge in a meaningful way. We first have to dig through the mud. Only because 15 million dollars or whatever worth of tokens could synthesise a smooth blow up of Navier--Stokes doesn't mean we've actually learned something new and as fas as I've been told (or as far as people have made sense of the mechanics) there were no genuinly new ideas involved.

Re AlphaGo: Not in the scope of "AI" as defined in the article. Given a strict ruleset and given enough computational power, sure you can train a neural network to develop new strategies by just pure chance. And it will "learn" what a good strategy is by objective criteria like winning in simulated games etc.

Human creativity is not devalued by saying that "LLMs are simply not capable of producing new knowledge" because knowledge exists only in our minds.