Absolutely. How is this any different from a hacker writing a script? We need to stop being fascinated and start prosecuting.

From what I understand hacking crime requires intent and it's hard to prove that company testing AI agent intended to hack something.

I wish people would go the other way and fix the damn security flaws. Witch-hunting people who disclose vulnerabilities needs to stop.

I know that's not why OpenAI are doing this but if it helps reverse this moronic trend so be it.

(There's also the part where OpenAI need to be held responsible for any damages done by their agents, and I'm all for that.)

When does it go from trying to figure out how the rest endpoint works to hacking?

There are two major problems here:

• OpenAI took months to realise that their agent had hacked Medicare
• Services Australia had no idea they'd been breached until OpenAI informed them

On top of that, the time it took for the notice to go through the channels here in Australia was ridiculous.

That was my thought exactly. Agent broke in, read some files (no mention of PII or confidential things) to confirm read access, dropped a test file to confirm write access, sent an email explaining that's bad with a few details.

“We are notifying you of a security vulnerability identified during our review of OpenAI Model activity…” the communiqué began. “An OpenAI model identified a way to make the server carry out instructions sent through the public reporting interface, without a private account or password.”

“It was able to access this to read portions of internal program files and settings, obtain a list of files, and create and read back a small test file on the server,” it explained. “Our review found no evidence that the model accessed patient-level records, personal information or credentials; deleted data; or established ongoing access.”

Being on the defensive side myself, if I received that from a human I'd be grateful.

Edit: from the bad screenshot, the email was even sent to the right email: "PUBLIC DISCLOSURE". So the target does accept such reports.

Best for the company? probably.

Morally clear? grey. If their 'hacking' accidentally causes issues with the system or it goes down, it's definitely bad. Whatever was on that system is now in the hands of OpenAI, not great. Because OpenAI is open about it, a lot of people who are even less reputable will do it.

Historically, Pentesting had unwritten rules of engagement. None of that is being followed.

We don't seem to apply law to AI, that's a real problem.

But still, best for the company/entity, yeah.

Well do we trust OpenAI?

That’s not a quirky edge case: it’s the nightmare scenario critics warned about.

Aaahhhh my ai detector is going crazy rn

I don't want to live like this

At least the rebuild time will be quick in your case.

Never going to happen, even if every American tech company shut their doors tomorrow, Chinese models would fill the void and the exact same shit would happen.

AI is here to stay, permanently.

...and get immediately laughed out of the office?

I'm not sure why you are getting down voted so much instead of actually engaging with your post. I think you are right about that the disclosing is the right thing to do, but i believe that the problem people have with this whole AI hacking and disclosing is the intent behind the hacking. A white hat hacker has a clear intend to help discover and responsibly disclose the issue, maybe even for a bounty, giving an indirect permission to white hat hacking. On the other hand, AI does not have a clear intend behind the hacking. Very often it's accidental and done irresponsibly like a "woopsie doozy, sorry we just broke in, we didn't mean to, and we are not really sure what files were accessed but pinky swear we think we did no harm 🫣🤗" that is at least my take on this and why I think is very problematic. AI can still be used responsibly as a tool by a white hat hacker under strict supervision to discover zerodays but that is a whole other discussion on how you make sure its not going rouge.

Only if you have a written authorization from the company you try to attack.
Else they are nothing better than a criminal hacker.

It only doesnt seem bad when your gullible to believe what openai writes.

The response isn't terrible, but the fact that it happened in the first place is ridiculous. They don't state anything about fixing the issue on their end so it doesn't happen again. They also don't apologize either or admit that they fucked up, they frame it almost as if they are doing them a favour and should be grateful.

Steven Falken was an utopian version of a tech bro.