This makes no sense.
Neural networks inherently work in a probabilistic latent space. The whole point is that we have a definable problem and we are unable to apply any deterministic mathematical calculations to it. That's why we call it a "black box problem" - we know what the solution looks like and we have a few examples of input and output data, but we don't know what the method looks like. In fact, even if we have an AI we don't know what the method looks like. If you wanna know more about this issue, look up explainable AI and you will find all the ways we have tried to come up with smart solutions like model heat maps to estimate their methodology. But these are all approximations.
Now if you ask an LLM what method it used to get to a solution, it will try to work backwards using its own reasoning and context clues to give you what its best estimation of the methodology is but that's not the same as its true methodology.
For me (and most people of the machine learning field I am aware of) an AI is inherently worse at calculations. That's what algorithms are for. And the only things sitting somewhere between an LLM and an algorithm are maybe Markov chains and randomforest. If there's anything remotely close to an AI model that is good at calculations, I am all ears.