Page 73 - The Annual AI Governance Report 2025 Steering the Future of AI

The Annual AI Governance Report 2025: Steering the Future of AI

Yoshua Bengio argued that since we can't simply make AIs "dumb," the solution lies in controlling
their harmful intentions. He proposes a new approach: Non-agentic AI systems, which he calls
"Scientist AI":

• No Intentions or Goals: Unlike current AIs that imitate human behavior (which can include
deception), these systems are designed to have no intentions, goals, or drives for self-
preservation. They act like idealized scientists, focusing solely on understanding and
explaining the world, generating theories and predictions without a desire to act or
pursue objectives.
• Safeguard Mechanism: In the short term, "Scientist AI" could serve as a critical safety
layer. By monitoring the actions of more powerful, agentic AIs, they could predict the
probability of harm from a proposed action. If this probability exceeds a set threshold,
the "Scientist AI" could halt the action, acting as a gatekeeper to prevent dangerous
outcomes.
• Support for Research: Beyond safety, these non-agentic systems could also support
general scientific discovery by objectively looking for explanations and generating
hypotheses.

Rachel Adams (Director, African Observatory on Responsible AI) introduced the aspect of public
perception and social attitude surveys to "test for trust" in AI, highlighting that people's trust
in technology might differ significantly from technical trustworthiness.

Quotes:
• The biggest hindrance for deploying AI is how we can actually ensure the safety
and security of AI. (Dawn Song, Professor of Computer Science at UC Berkeley
and Director of Berkeley center for responsible decentralized intelligence)
• “I want to hear [a] call for action: benchmarking, benchmarking, benchmarking.”
(Boulbaba Ben Amor, Director for AI for Good at Inception, a G42 company)
• “While the major effort is turned to evaluate and benchmark AI models, we need
to move as quickly as possible to evaluate AI solutions.” (Boulbaba Ben Amor,
Director for AI for Good at Inception, a G42 company)
• “... There's a class of risk that, if it appears, it may appear very quickly … in a way
that the harms don’t materialize until it's too late to do anything about them, in
which case you need a risk management instrument that's able to identify those
issues in advance ...” (Chris Meserole, Executive Director, Frontier Model Forum)
• “If a system is smarter than you, more complex, more creative, it's capable of
doing something you didn't anticipate. So we don't know how to test for bugs we
haven't seen before.” (Roman V. Yampolskiy, Professor, Department of Computer
Science and Engineering, University of Louisville)
• "We need to continue to invest in R&D... to come up with the red lines, the
benchmarks, the thresholds, and the warning system. [There is] a five-stage
process: testing, auditing, verification, monitoring, and mitigation." (Ya-Qin
Zhang, Chair Professor, Tsinghua University)

64