GPT is basically a machine that builds machines.

We understand how the machine builder works. We don't understand how the output of the machine works. Like mechanically we understand each individual sub-component works e.g. a single matrix multiplication, but we don't understand why specific weights are the value they are, or the broader macrostructure.

I don't think you realize that acting like "No we actually do understand how it works" just reveals that you don't understand the question, it does not show that the question has an answer.