40
Report: AI Mistake Nearly Led U.S. Military to Board a Chinese Ship (maritime-executive.com)
submitted 4 days ago* by to c/techtakes@awful.systems
2 comments fedilink hide all child comments
 

The report, however, was wrong from the start. An analyst had used an AI chatbot to ask about some intelligence data related to the suspect ship's cargo manifest. The chatbot drew on classified data and open-source intelligence, then made up a conclusion: it decided that the vessel was carrying components for a nuclear weapons program, CNN's sources said. This was false, but the analyst did not detect that; instead, they packaged it up for formal use - again using AI for formatting - and sent it up the chain of command.

I really really hate bullshitters and this system is clearly optimized for them.

all 2 comments

sorted by: hot top controversial new old
[–] 5 points 4 days ago (1 child)

Are chatbots tuned for user engagement and not accuracy? As in the AI weighted the user engagement response for nuclear weapons higher than that of mundane commerce, and thats the reason it returned the inaccurate answers?

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] [S] 9 points 4 days ago* (last edited 4 days ago)

    There is no training chatbots for accuracy or precision, they pattern match information in whatever way is most closely alligned to the training data aesthetically, meaning or accuracy don't come into it but yes absolutely there is a "this must be the main plotline to the story we are in!" bias to chatbots, they whisper "this is it" into your ear in whatever way is most likely to send an excited shiver down your spine, hallucinating a high stakes Tom Clancy novel out of nothing is 100% in the wheelhouse of this shitty broken tech...

    ...after all the people who created and use this tool certainly almost by definition want to be a main character in a Tom Clancy novel...even if it ends badly or they are the "good guy" villain in it.

    In a way this is the oldest type of human delusion and this type of technology is the farthest thing from "new" possible, we have already seen this story a million times and it never ends any differently no matter how "new" or "unprecedented" AI is on the face of it.

  • permalink
  • fedilink
  • source
  • parent