Right I appreciate your insight. I'm not heavy in coding myself, I brush up against code and use it for some automation and assistance but I've also seen it write literal nonsense for code.

That rub is so real. I asked the thing to write some code for me for a Sailpoint workflow and I wasn't familiar with the syntax so figured it'd be a good place to use AI. That thing hallucinated extremely convincing code except it just wholesale made up how it worked and of course it looks fine to my eyes but then go to run it and error -> work to fix that error and then -> error, etc etc.

After chasing my tail for an hour or two I finally stopped and went to pull up some official documentation and that's when I figured out the AI was just hallucinating like a motherfucker. So I went back and did it the "old fashioned" way, but at least the AI did give me an idea of what I had to do.

Anyways, I could see someone who isn't me just running around in circles for days because they believe the LLMs we call AI are giving them actual answers.

I blame some of it on how AI got mixed into web search. Traditionally, the resource from the web would at least be written by a human and usually relevant. Generally semi trustworthy and not made up. Now people see search results with AI summary/answer at the top and that is not vetted, could be a straight up hallucination*, or even just a total misinterpretation of your search phrase so totally irrelevant.

*I wish this term was different because it has an implication that the AI can and cannot hallucinate when the truth is the AI is always hallucinating. Put another way, to the LLM there is no such concept as truth. It annoys me that people seem to believe that the AI "knows" stuff. It does not. Stop anthropomorphizing the fancy robot.