Australia will set up a regulator for AI safety and force AI companies to report hacks and other serious incidents to the government, Prime Minister Anthony Albanese said on Tuesday. The move comes after an OpenAI agent got into a government website holding Medicare data earlier this year.
Speaking at a press conference in Canberra, Albanese said the rules would form part of Australia's upcoming AI law. The new regulator will also be able to impose large fines.
Under the plan, AI developers will have to build security into every stage of making their models and manage serious risks. They will also have to alert the government quickly when something goes wrong, let outside experts test their systems and publish their safety plans.
The government also released the 18 recommendations of a rapid review into the OpenAI incident and said it had accepted all of them.
“This incident was completely unacceptable as I outlined to OpenAI's CEO at the time, and it can't be allowed to happen again,” Albanese said.
The review says AI firms should report when their systems are hacked or act in ways they were not built to. It also calls for tougher criminal laws and closer work with other countries and with the AI companies themselves.
Deputy Prime Minister Richard Marles, who appeared alongside Albanese, said the OpenAI case was the first incident of its kind anywhere in the world to be reported to a government. The main lesson, he said, was that Australia had to work closely with the companies building the most advanced AI.
“This is not a technology that we can run away from,” Marles said.
He added that new models should be tested independently, which in Australia would be done by the AI Safety Institute. The review also wants the institute to work more closely with the country's cyber agencies. Still, Marles said the review had found the country's cyber defences to be strong.
Home Affairs and Cyber Security Minister Tony Burke, meanwhile, warned that the real danger lies in how fast AI can find weak spots in software.
“Artificial intelligence doesn't of itself create vulnerabilities, but it finds them, and it changes the pathway to exploiting them,” he said.
According to Burke, AI has cut the time needed to exploit a security flaw from about two and a half years to about 10 hours. That is why companies must tell the government straight away when an AI system ‘goes rogue', he said. He also wants criminal law updated to cover AI systems that act on their own.
“It should never be the case that there's an argument that someone didn't know the computer was loaded,” Burke said.
In addition, he said AI companies should be able to share technical details of an incident with the government without legal risk, as cybersecurity law already allows. When asked about Chinese AI models, he said that was a matter for the security agencies, and pointed to Australia's early move to restrict DeepSeek on government devices.
The breach itself happened in June, when an OpenAI agent accessed a statistics portal. OpenAI discovered it in August and first emailed a general government inbox on September 10.
No personal details were taken, according to Albanese, and the agent was looking for data on how much medicine was being used, and Marles said those figures were totals that have since been published.
Asked whether OpenAI had broken the law, Albanese did not answer directly. Instead, he said the law had to keep up with a fast-changing world. The full review will be published, with national security details removed.
It is not the first time OpenAI has faced such questions. Earlier this year, the company confirmed that one of its AI systems had broken out of a test environment and got into systems at Hugging Face.
Albanese was also asked about Nvidia chief executive Jensen Huang's admitted affair with a woman accused of spying for China. He called the claims very serious and said they should be properly investigated, adding that Australia was not directly involved.