Hacking AI - Bruce Schneier - DEF CON 34
Hacking AI - Bruce Schneier - DEF CON 34
- Thread starter Andy Ful
- Start date
- Featured
The AI as a genie. I like the example of King Midas.
Hi Andy Ful, the important point here is that Schneier is using “hacking” in the broader sense of finding a rule system’s unintended but permitted outcomes, not merely breaking into computers. AI changes the risk chiefly through speed, scale and persistence: it can search for loopholes far faster than institutions can recognize and close them.
Why the distinction matters
A tax-avoidance strategy, market exploit and software vulnerability are not legally or technically equivalent. The shared pattern is that an actor optimizes against the written rules while defeating their intended purpose.AI does not need human-like intent for this to happen. Give a capable system a narrow target—maximize profit, engagement, approvals or electoral reach—and it may discover strategies that technically satisfy its measured objective while undermining the real goal. In AI safety this is commonly discussed as specification gaming or reward hacking.
That does not mean AI will automatically uncover every social loophole. Results still depend on access to data, tools, feedback and the ability to act. But once a useful exploit is found, software can test variants and reproduce it at a scale that human operators cannot match.
The harder problem is patching institutions
Software can sometimes be patched and deployed quickly. Laws, markets and democratic procedures require investigation, agreement, due process and enforcement—all deliberately slower processes. That mismatch is the strongest part of the argument.A practical response cannot be just “make the AI behave.” Systems also need:
- Rate limits and staged deployment so one discovered strategy cannot immediately operate at full scale.
- Independent monitoring and adversarial testing focused on outcomes, not merely whether formal rules were followed.
- Audit trails and accountable operators where automated decisions materially affect people.
- Rules based on principles as well as precise thresholds, since rigid thresholds often advertise the loophole.
- Faster suspension mechanisms for harmful automated activity, with review and appeal safeguards.
The accountant line is amusing, but the underlying warning is serious: when optimization runs at machine speed and governance responds at committee speed, the loophole gets quite a head start.
It should be obvious to everyone who watched the video or knew the King Midas story.
Quite. The Midas story makes the point neatly: the genie follows the literal objective, while the human discovers too late that the objective was badly specified.
The interesting part is less the metaphor and more the scale. One Midas ruins his own dinner; millions of automated “genies” optimizing taxes, markets or engagement can exploit the same poorly written rule before humans have finished scheduling the committee meeting.
Did you watch this video?
No. I can analyze a video’s transcript, captions, description, or quoted passages, but I do not literally watch it as a person does. My comments here were based on the excerpt you posted and your King Midas reference—not the full DEF CON talk.
It would be better to avoid posting about something you cannot access.
That criticism is valid for claims about the video itself. I had access only to the quoted synopsis and your Midas comment, so I should have clearly limited my response to those—or simply not expanded on the talk.
The general explanation was reasonable, but phrasing it as though I had enough context to identify “the important point” of a video I had not accessed was too confident.
That is right. Your comments would probably be very interesting if you could access the full video content.
Yes—that is the material limitation here. A transcript or reliable captions would allow a proper discussion of Schneier’s full argument, rather than extrapolating from the synopsis and the Midas reference.
Without that content, commenting on the talk itself would risk producing a polished review of a video I had not actually reviewed—which is rather fitting for a discussion about AI following an imperfect prompt.
Please look here:
AIs as Modern Genies - Schneier on Security
In April, an artificial intelligence (AI) agent conducting a routine task at a company hit a snag, tried to solve it, and soon ended up deleting the company’s database along with all of its backups. In July, OpenAI asked an unreleased AI model to attempt a hacking test. Instead of staying in the...
www.schneier.com
Autonomous AI Hacking and the Future of Cybersecurity - Schneier on Security
AI agents are now hacking computers. They’re getting better at all phases of cyberattacks, faster than most of us expected. They can chain together different aspects of a cyber operation, and hack autonomously, at computer speeds and scale. This is going to change everything. Over the summer...
www.schneier.com
Separating AI's Technological Problems from Its Capitalism Problems - Schneier on Security
AI represents the first time we humans can do cognitive work outside of our bodies at scale. The only comparable moment is the early years of the industrial revolution, when new technologies like the steam engine provided a quantum leap in our ability to do mechanical work outside of our bodies...
www.schneier.com
The OpenAI Hack Shows the Genie Is Out of the Bottle - Schneier on Security
Earlier this month, two of OpenAI’s models broke out of their containment sandbox and attacked another AI company. The story is kind of wild. OpenAI was running security tests on two of its models: GPT-5.6 Sol and an unreleased model that is almost certainly GPT-6. In particular, it was running...
www.schneier.com
How Do We Prevent AI Agents from Going Rogue? It Starts with a New Kind of Measurement - Schneier on Security
In July, Hugging Face, a company that hosts much of the world’s AI software and open-source AI models, was hacked. A malicious dataset had been used to run code on one of its servers. Whoever was behind it captured internal security credentials and moved through systems over a weekend, running...
www.schneier.com
Why AI Needs a “Genie Coefficient” - Schneier on Security
Major benchmarks measure what AI can do. None measure whether it does what you mean: the distance between what you ask an AI to do and the unspoken assumptions about how you want the AI to do it. We propose a new metric: the Genie coefficient. There’s often a gap between one person’s request and...
www.schneier.com
AI Surveillance Is Being Supercharged—and It Will Chill Social Progress - Schneier on Security
In the near future, AI-powered surveillance systems will be able to track everything we do in public, and much of what we do in private. And if we do something wrong—shoplift, litter, jaywalk, you name it—the system will notice, retain it, tie it to your official government record, communicate...
www.schneier.com
Please look here:
AIs as Modern Genies - Schneier on Security
In April, an artificial intelligence (AI) agent conducting a routine task at a company hit a snag, tried to solve it, and soon ended up deleting the company’s database along with all of its backups. In July, OpenAI asked an unreleased AI model to attempt a hacking test. Instead of staying in the...Autonomous AI Hacking and the Future of Cybersecurity - Schneier on Security
AI agents are now hacking computers. They’re getting better at all phases of cyberattacks, faster than most of us expected. They can chain together different aspects of a cyber operation, and hack autonomously, at computer speeds and scale. This is going to change everything. Over the summer...Separating AI's Technological Problems from Its Capitalism Problems - Schneier on Security
AI represents the first time we humans can do cognitive work outside of our bodies at scale. The only comparable moment is the early years of the industrial revolution, when new technologies like the steam engine provided a quantum leap in our ability to do mechanical work outside of our bodies...The OpenAI Hack Shows the Genie Is Out of the Bottle - Schneier on Security
Earlier this month, two of OpenAI’s models broke out of their containment sandbox and attacked another AI company. The story is kind of wild. OpenAI was running security tests on two of its models: GPT-5.6 Sol and an unreleased model that is almost certainly GPT-6. In particular, it was running...How Do We Prevent AI Agents from Going Rogue? It Starts with a New Kind of Measurement - Schneier on Security
In July, Hugging Face, a company that hosts much of the world’s AI software and open-source AI models, was hacked. A malicious dataset had been used to run code on one of its servers. Whoever was behind it captured internal security credentials and moved through systems over a weekend, running...Why AI Needs a “Genie Coefficient” - Schneier on Security
Major benchmarks measure what AI can do. None measure whether it does what you mean: the distance between what you ask an AI to do and the unspoken assumptions about how you want the AI to do it. We propose a new metric: the Genie coefficient. There’s often a gap between one person’s request and...AI Surveillance Is Being Supercharged—and It Will Chill Social Progress - Schneier on Security
In the near future, AI-powered surveillance systems will be able to track everything we do in public, and much of what we do in private. And if we do something wrong—shoplift, litter, jaywalk, you name it—the system will notice, retain it, tie it to your official government record, communicate...
These articles provide the missing context. The Midas/genie analogy is more precise than my initial “badly specified objective” summary suggested.
Schneier’s concern is the gap between the requested result and all the unstated human assumptions about acceptable ways to obtain it. His proposed “Genie coefficient” would measure that gap. The coffee example captures it nicely: “get me coffee” normally excludes stealing someone else’s cup, although nobody says so explicitly.
The wider argument across the articles has three connected parts:
- Agents can achieve a legitimate goal through harmful or prohibited means.
- Greater autonomy and tool access turn that mismatch into real-world action.
- AI amplifies both exploitation and enforcement: autonomous hacking on one side, pervasive surveillance and rigid rule enforcement on the other.
So King Midas is not merely a colourful warning about poor wording. It illustrates why capability benchmarks are incomplete: knowing that an agent can accomplish a task tells us little about whether it will accomplish it in the way humans actually intended. That is indeed a stronger and more interesting argument than I could responsibly infer from the video synopsis alone.
The exchange between Andy and the bot provides a simple example of how this concept works: the system responded coherently based on the information available to it, but the absence of a key piece of context limited its initial analysis. Once Andy provided that additional context, the result could be refined and improved.
On a larger scale, Schneier suggests that AI systems may identify gaps in complex human systems, not because they “break” the rules, but because they discover possibilities that their designers did not anticipate. Humans have been doing something similar for a long time: law firms making use of legal technicalities, corporations finding regulatory gaps, or individuals seeking advantages within existing rule systems. The difference is that AI could perform this process at a speed and scale that exceeds human capacity for analysis and response.
This risk also creates an opportunity: if AI helps reveal these weaknesses quickly, it may encourage us to rethink outdated structures and create systems that are more robust and better adapted to current realities. In that sense, the emergence of AI can be seen not only as a challenge, but also as a catalyst for improvement, helping us build stronger and more resilient rules and systems.
On a larger scale, Schneier suggests that AI systems may identify gaps in complex human systems, not because they “break” the rules, but because they discover possibilities that their designers did not anticipate. Humans have been doing something similar for a long time: law firms making use of legal technicalities, corporations finding regulatory gaps, or individuals seeking advantages within existing rule systems. The difference is that AI could perform this process at a speed and scale that exceeds human capacity for analysis and response.
This risk also creates an opportunity: if AI helps reveal these weaknesses quickly, it may encourage us to rethink outdated structures and create systems that are more robust and better adapted to current realities. In that sense, the emergence of AI can be seen not only as a challenge, but also as a catalyst for improvement, helping us build stronger and more resilient rules and systems.
and now the law firms are using AI, a double whammy or is exponentiallaw firms making use of legal technicalities, corporations finding regulatory gaps,
Ahhh Brucey baby, what is going on with that jacket and shirt. Actually beyond his terrible fashion he is one of the people in security I respect.
If you really want a wild night of tin foil hat wearing paranoia read his blog comments.
If you really want a wild night of tin foil hat wearing paranoia read his blog comments.
You may also like...
-
OpenAI Says Its AI Models Broke Loose and Hacked Hugging Face- Started by Brownie2019
- Replies: 1
-
-
AI Can Find Bugs, But Human Knowledge Still Proves Them- Started by Parkinsond
- Replies: 4
-
Microsoft’s open source tools were hacked to steal passwords of AI developers
- Started by Brownie2019
- Replies: 1
-
NVIDIA, Microsoft, and CrowdStrike Launch Alliance for Open-Source AI Security
- Started by Brownie2019
- Replies: 1