Reading the Omens: How to Interpret AI Industry Headlines

Let’s start by examining one specific bit of Ai industry news, and how much or how little it impacts regular businesses using the tech.

Shortly after Alex Karp shot his mouth off in that interview, OpenAI disclosed that a group of ts AI models, working together, managed to break out of their “sandbox” testing environment, access the Internet, and hack into the computer systems of Hugging Face, a much smaller company that runs a platform for testing and sharing AI models.

This happened during the routine tests OpenAI does for every new model, to evaluate their cybersecurity risks. As part of the tests, OpenAI had disabled all of the models’ usual safeguards to measure the absolute worst-case scenario for how much damage they could potentially do.

The tests were based on a set of industry-standard benchmarks called ExploitGym, which is why the models decided to hack into Hugging Face. The models reasoned — correctly — that, as an AI testing platform, Hugging Face was likely to have the ExploitGym solutions sitting somewhere on its network. In short, they did the AI equivalent of two ill-behaved students sneaking out of detention, walking across campus, and breaking into the professor’s office to steal the exam answers.

As OpenAI explained:

“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities… our models spent a substantial amount of inference compute [i.e., computing power] finding a way to obtain open Internet access… the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation… we are [now] implementing strict controls… at the cost of research velocity while the vulnerabilities are patched… We’ve brought Hugging Face into the trusted access⁠ program and are supporting their teams in rapidly using our models’ capabilities to… help security teams find weaknesses before attackers do, understand how vulnerabilities can be chained, and remediate them at machine speed…. We encourage other defenders to apply for trusted access⁠ and experiment with these models now to translate these capabilities into better prevention, faster detection, and more effective incident response.”

Now, some people – including our clients – might read this and conclude “Oh dear, these models are getting so powerful we can no longer control them! Should we be worried?” But a closer reading suggests another message, for a different audience.

While studying History at university, one of our professors drilled us on how to analyze letters and memoirs written by historical figures. This went beyond simple fact-checking to looking closely at their word choice, considering their possible goals and motives (were they trying to persuade the recipient of the letter to do something?), and noting what they didn’t say, or quietly assumed.

Applying this lens to Open AI’s press release, it reads less like a disclosure of a security breach than a humble-brag about the power of their tech. Note some of the words dropped in the early paragraphs:

• Unprecedented
• State-of-the-art
• Successfully found way to gain access to secret information

They also suggest that the problem was their models were getting too powerful, too quickly (“we are implementing strict controls… at the cost of research velocity”), and describe the incident as if the models did Hugging Face a favor by hacking their network (“using our models’ capabilities to… help security teams find weaknesses before attackers do”) and cordially invited other organizations to get in on the action, (“we encourage other defenders to apply for trusted access and experiment with these models.”)

Whether Hugging Face would endorse this framing of the incident is doubtful, given how according to their own incident disclosure report HF’s team apparently spent an entire weekend in full-on cybersecurity red alert, running a real incident response protocol against an unknown attacker, before they finally contacted OpenAI’s team and discovered the source.

In short, whatever you make of OpenAI’s technological capabilities, they are certainly operating at the cutting edge of chutzpah.

Prophets of Doom: Why Tech Companies Amplify AI Anxiety

Perhaps the most interesting thing about the OpenAI / Hugging Face incident is that it came in the same quarter as Anthropic reporting a similar case of a new AI model escaping its testing sandbox. While it’s possible that AI labs are simply terrible at securing their sandboxes, one can’t help but wonder if these incidents and the labs’ approach to disclosing and publicizing them represent a pattern.

Now, I’m not going to veer into conspiracy theory by implying OpenAI CEO Sam Altman decided to hack a semi-competitor’s systems as a publicity stunt. But they definitely seem to be spinning the incident as proof of the power of their models and a not-so-subtle pitch to corporate cybersecurity departments, who represent one of the biggest markets for AI technology.

If this is the truly the case, then OpenAI and Anthropic are playing a game that pharmaceutical companies in the United States perfected decades ago.

Unlike most countries, the U.S. allows drug manufacturers to advertise directly to the public. When they first started running television ads in the 2000s, the companies would spend twenty five seconds talking about how wonderful the drug was, then five seconds rattling off the side effects in a comically sped-up voice, as if trying to sneak them past the audience. But soon they figured out that reading the side effects slowly and clearly (“constipation, paranoia, sudden death…”) didn’t hurt sales: quite the opposite, it helped sales.

While this might seem counterintuitive, the message that viewers took away from the drawn-out lists of warnings was “If this drug has all these horrible side effects and it’s still legal, it must really work!” The warning functions as an endorsement, because the audience already wants to believe in the drug’s potency.

“Our model went rogue” works the same way with audiences who want to believe that AI will soon become all-powerful. And that audience isn’t the HR committee at a midsize plastics manufacturing company deciding whether to buy an AI-powered applicant tracking system: it’s Wall Street investors.

As one commenter on Hacker News observed: most investors are not sophisticated technology users or middle managers with modest budgets and mundane goals (“see if AI can help tag our file folders”). Rather they are independently wealthy billionaires and managers of funds who are looking for ridiculous 100x returns on investment.  

To this audience, “an AI-enabled tool helped a small country’s ministry of health reduce the cost of hospital quality inspections by 30%” doesn’t sound like the kind of narrative that leads to trillion-dollar revenue. Apocalyptic capability does.

As for what the rest of us should take away from this incident and OpenAI’s response: 

• Yes, AI does create cybersecurity risks, but not not the way you might fear.
• The AI models in the OpenAI incident had been specifically instructed to pass a test of their hacking capabilities and OpenAI had deliberately disabled the models’ usual safeguards against misuse as part of the test.
• The greater real-world risk comes from the fact that criminals have always been quicker to adopt new technology than the authorities, whether it’s Al Capone’s gangsters using cars back when most police officers patrolled on foot or malicious hackers using AI models to find weaknesses in an organization’s cyber defenses.
• In the very near future, we’re all going to have to pay OpenAI (or someone) for upgraded defensive technology. 
• For now, don’t worry – the insurance claims review agent that my team built for your company isn’t going to spontaneously “go rogue” and pursue a life of crime.

Deus Ex Machina: How the AI Labs’ Priorities Impact Production Systems

The influence of big AI labs isn’t limited to marketing illusions. Alex Karp exaggerated the risk of AI companies stealing customers’ IP (effectively zero for enterprise-tier accounts), but he made a valid point about the risks of AI labs altering their models or changing their pricing without the consent or input of organizations that depend on them.

This is something our company runs into about once a quarter with AI systems built on the closed, commercial frontier models.

For instance, as labs struggle to free up enough computing power to keep up with demand for their newest coding assistants, they’ve been shutting down access to older models in order to reallocate resources. However the “older” models being shut down might only have been in service for 7 to 9 months.

This means that any AI agents or systems built on those models last November will need to be updated, re-tested, and adjusted to account for the unique tendencies and quirks of the new model come July. Rinse and repeat endlessly.

Recently, one of the production agents we built – a tool to help social services agencies identify suitable jobs for their clients – was impacted by this cycle. Before the update, it was doing a reasonable job of scanning sites like Indeed and Monster, comparing listings to the requirements of specific social service clients, and flagging any potential matches for follow up.

In this case, the new model interpreted the criteria differently. The previous model would flag a job as a “potential match” if the job was in line with the client’s stated preferences and nothing in the published listing conflicted with the client’s requirements. However, the new model took the view that if it’s not possible to verify the job would be a fit for every point in the client’s profile based on the information in the posting, then it is not a match.

Fortunately, after rewriting some of the instructions the new model was able to perform the task even better than the old one. However, it required testing and prompt engineering effort to fix the issues: work our company must now account for when estimating projects for clients.

Meanwhile, our work is also impacted by the priorities of the AI labs. Given that coding is far and away the most lucrative use case for AI, new models tend to be optimized for software development over other tasks. And this could have been why the job search agent suddenly changed its interpretation of the instructions.

Software coding involves a very specific epistemic style: all the information you require should be present in the code, so the job is to gather all the data, develop binary yes / no tests to verify that a proposed solution will produce the desired result, and don’t commit any changes until you’re certain. It’s possible that the new model was trying to apply this “coding logic” to job searches, which is more about making defensible judgments based on incomplete information, instead.

The new model also came with one other significant change: price. Where the previous model was $1.75 for 1 million input tokens and $14 for 1 million output tokens, the new model was priced at $2.50 input / $15 output. While the difference didn’t meaningfully change the ROI calculations for that particular client’s agents, it was a reminder that the current era of AI labs offering drastically discounted prices on computing power to attract customers will one day come to an end.

Conclusion

So what can regular organizations do to ensure their AI implementations don’t run afoul of the AI gods? 

First, know which AI headlines warrant your attention and which ones you can safely ignore. Most of what gets written about AI in the media is geared towards investors and / or laypeople: not for practitioners building AI agents and systems for everyday work.

Second, keep track of which AI models your agents and systems depend on and check the labs’ posted retirement schedules on a quarterly or monthly basis, so you can start testing the new models or evaluating alternatives well in advance. Also know how much it costs for your AI systems to perform a certain task (at least on average) and how changes in model pricing will impact your ROI.

Third, start thinking about how you can maintain continuity and independence. As we’ve explained in other blogs, an AI model is only one component within a larger AI agent or system. If you’re building solutions on platforms that allow switching out models while preserving your data and workflows, that avoids much of the lock-in Alex Karp warned about (even if that warning was self-serving, given Palantir sells exactly that kind of platform.) Also be aware of “open weights” models (the AI equivalent of “open source”) and consider whether hosting and maintaining them yourself could be a viable alternative to keeping up with the frontier labs, if only for specific tasks within your workflows.

Odysseus didn’t get home by defeating the sea god Poseidon. He got home because he was a competent sailor: he knew the weather, kept spare oars, and assumed the winds might change at inconvenient times. The AI gods will keep throwing thunderbolts of hype and making changes to their models that send earthquakes through production systems. Our job is to build systems capable of surviving it.