Old School vs. New School

Just as skateboarding has changed since my brief stint as a skater in the early 90s, the way we work with models has changed considerably in the two and a half years my team and I have been building and deploying AI systems for clients.

Back in 2024, model differences were real and they were about capability. You picked a model because it could or couldn’t do certain things. Claude was the best writer but couldn’t be trusted to follow instructions consistently. Early Gemini had to be prompted like you were writing an operating manual for a particularly complicated lawn mower, but at least executed consistently once you got the language right. ChatGPT was sort of the Goldilocks midpoint between the two.  

That world is mostly gone. The differences haven’t disappeared, but the baseline competence of the models has risen, and most of their quirks have been streamlined away like cars in a wind tunnel.  

Today, the differences between AI labs are more about operating characteristics: how much a model costs, how fast it answers, how much reasoning it does by default, how it handles tools, and how twitchy its guardrails are. And the most significant differences are between tiers (GPT Nano vs. GPT Chat, Sonnet versus Opus) less so, “which is better – ChatGPT vs. Claude?”

Prompting is a matter of clearly explaining tasks in manageable chunks and providing the necessary nuance and context, not guessing the correct synonym to make the model do something or magic phrase (“embody the world’s smartest lawyer”) to boost performance.

Vert vs. Street

The other thing to bear in mind is that the “best” AI model depends entirely on the context.

In skateboarding, there’s “vert” skating which involves going up and down a half pipe at high speed and doing elaborate spins in midair, then there’s “street” skating which is more about staying on the ground and doing grinds and slides on whatever curbs and railings happen to be nearby. Obviously, you want different setups for each purpose.

When our team builds “expert agents” to help salespeople write proposals, we don’t fuss over a $1.50 difference per proposal if a higher-quality model can increase the odds of winning a $250,000 contract even 1%. But when we’re doing large-scale process automation, where an AI agent might handle thousands of relatively simple tasks (e.g. rewrite a factory technician or social worker’s voice-dictated notes in a standardized format that can be used for reporting), reducing the per-task cost from 7.4 cents to 2.7 cents can be significant.

In other words, most of the industry press focuses on the performance ceiling for models, the AI equivalent of whether a skater can execute a 1260° rotation off a mega ramp. However, for many business applications, the cost floor is more important. We’re less concerned with “What cool things can the latest models do that weren’t possible last month?” versus “What’s the cheapest model that can do this particular task 99% consistently?”

Dialing In Your Setup

Recently I listened as someone made a strong argument that the new version of OpenAI Codex is better than the latest Claude Code release and that my development team should switch.

And the thing is, they might be right – but we’re still not going to switch.

This isn’t because our team is lazy or that we’re particularly loyal to Anthropic. It’s just that evaluating AI models and applications versus using them to ship software are almost opposite mindsets.

A person who blogs about the latest AI research has every incentive to notice a 7% improvement on benchmark performance. But a team actually working with AI tools has little incentive to burn three weeks retooling because some new model did a few pips better on a comparison chart. Claude Code is integrated into our systems and workflows, we know how it behaves, and most importantly, work is getting done on schedule and within budget to expected standards of quality, so at present our time is better spent on delivering more features with the same setup than migrating models. We don’t measure success in terms of “our AI tool is 12% smarter”, but rather “we saved our dev team from wasting an afternoon writing boilerplate code to change the sort order on a list, and instead had them work on more interesting features that will make our platform more competitive.”

The same goes for most of the AI agents we build for clients: if a Gemini-based agent is consistently providing sound advice for complex banking transactions, we’re not going to tinker with its core mechanics every three months just because a newer model scores two percent higher on an academic reasoning benchmark like GPQA Diamond, or claims marginal improvements on an agentic tool-use index like Tau-Bench.

And it’s also worth noting that AI systems are more than just the AI model.

New skaters obsess over decks (the large piece of wood that makes up most of a skateboard’s body) because decks are easy to compare, however experienced skaters will talk just as much about the equipment connected to the deck – truck geometry and wheel durometer and bushings – because that’s what determines how the board behaves over a real session. Likewise, while models get all the press, real AI developers will be just as concerned about the agentic harness and whether you’re using a knowledge graph or vector database in your context layer.

Just like in skating, a pro will swap out components for very specific reasons a novice wouldn’t have enough experience to appreciate. And they wouldn’t rebuild the whole setup every time they buy a new deck: they’d move the trucks over, perhaps tighten it up, adjust their stance and keep skating.

When to Bail

So what would motivate us to switch models or labs? Typically, one of three things:

Scenario 1: Our incumbent violates an assumption we’ve built a workflow around.

Sometimes, our current provider simply fouls up, forcing us to reevaluate our decision to use their models.

To give an example from outside AI, my company does a fair amount of video editing. For years, we did all our work in Apple’s Final Cut Pro software. On some level, all video editing software did the same basic tasks, but Apple offered a smoother user interface and required less fussing around with settings when importing different types of video.

But then Apple released “FCP X”, a major version update that took everything we liked about Final Cut Pro and blew it up. They simplified the interface to be more like their consumer iMovie product, which offered too little control for serious professional work. On top of that, they dropped other professional features like multi-cam editing (which was a deal breaker for us, since half our projects were talking head interviews filmed with a standard two-camera setup.) Then, to add insult to injury, they completely cut off support for projects created in previous versions.

Rivals like Adobe Premiere capitalized on the industry outrage, offering a 50% discount to editors looking to switch away from Final Cut, which our team did even though I’m still not in love with Premiere.

With AI, every so often a frontier lab will release a model that’s simply a dud: for instance, our team ran into all manner of issues with gpt-5.4-nano not following simple instructions that we didn’t have with plain old gpt-5-nano. We ran into a similar situation with our AI development work when Anthropic released a version of Claude 4.5 that took their “ethical AI” schtick to an extreme. Our agents were getting derailed by Claude injecting “don’t reproduce copyrighted material!” warnings simply because someone mentioned the model of a car. We also had cases of a Claude based virtual social worker getting squeamish and refusing to continue conversations when a client made offhand comments that offended the model’s sensibilities, even though the point was to help the client search for a new job, not correct anyone’s unenlightened attitudes. And it got bad enough that we were forced to move agents over to ChatGPT because of it.

It’s harder to anticipate this kind of thing with AI models versus traditional software. In fact, to call gpt-5.4-nano a “new version” of gpt-5-nano is a bit misleading since it’s not like a developer went in and made a couple of discrete tweaks to its code. Rather, the two versions are more like siblings: grown from the same genes in a similar petri dish, but there’s always a random element in AI model development that might cause it to pick up entirely new quirks nobody programmed on purpose. It’ll improve 90% of behaviors and surprise you with the other 10%. So when your agent suddenly gets chattier or more squeamish after a “minor” version update, that’s why.

Whatever the cause of these types of system-breaking issues – be it a rushed development schedule or a frontier lab’s ethical good intentions producing unintended consequences – they undermine our trust in the model to behave and do a job for clients. And that’s when we go shopping for alternatives.

Scenario 2: A rival becomes too good to ignore.

While most of this essay has advocated caution before migrating to the latest and greatest model, that doesn’t mean organizations shouldn’t experiment.

Our actual stance is quite the opposite: somebody in the organization should always be taking the newest models out for a spin to stay current with what’s possible. But this experimentation should happen in sandboxes, side projects, and new builds: NOT by dropping this month’s buzzy new model into a live instance of an insurance claims workflow. That can wait until after you’ve had a bit of time with the new model, learned its strengths and limitations, and decided whether the operational gains outweigh the migration pain (i.e. basic DevOps common sense.)

To give a recent example from my own work, I have a ridiculous personal project where four different AI personas act as if they are members of a 1960 / 70s pop band hanging out in a studio having random conversations (think The Monkees or The Partridge Family.) Each member of the band is an independent AI agent running on a different AI model, collaborating on song ideas which they eventually feed into ElevenLabs’ music generator.

Over time I noticed that the one band member running GLM-5, a model from up-and-coming Chinese lab Z.ai, was producing some of the funniest and smartest lines of dialogue even though GLM-5 costs about half as much as Sonnet.

After that, I started asking budget-conscious clients “Are you open to using Chinese models?” The Americans and Europeans mostly hemmed and hawed and opted to stick with the big labs, however my Middle Eastern manufacturing clients just laughed and said, “Everything else in our factories is Chinese — why not?” And now GLM-5 is my go-to recommendation for clients who’d appreciate slightly-lower-than-Sonnet performance at slightly-higher-than-Haiku prices.

Scenario 3: The AI lab retires (deprecates) a model.

For decades, technology platforms and application developers have been playing a game of tug-of-war. Tech platforms keep making improvements that require developers to update their applications, meanwhile developers expect tech platforms to continue to support apps built around the old architecture. Eventually it reaches a tipping point where the platform says “We’re not supporting the old way of doing things anymore: upgrade or leave” – and sometimes developers will comply and sometimes they’ll migrate to a competitor.

With traditional cloud and mobile software, Apple was often the most aggressive when it came to telling the developer community “The next version of iOS won’t support feature X anymore: deal with it.” Android, meanwhile, took a more forgiving stance, letting old code limp along for a few releases with plenty of advance warning. Microsoft often went too far in the other direction, letting SharePoint customers keep decade-old configurations for years, long past the point where organizations should have upgraded for their own good.

AI labs have been playing this same game on fast-forward, in part out of economic necessity.

Despite the explosion of data centers, the frontier labs have a limited amount of computing power available to run their models. Every GPU running last quarter’s flagship model is one not serving this quarter’s shiny new model, and the labs haven’t been shy about tossing users of last year’s models out on the street, like a bartender shouting “We don’t care where you go, but you can’t stay here!” at closing time.

Anthropic is the most like Apple in this respect, barely letting models run for a year before requiring updates (and not only that, but forcing developers to change key settings before migrating to the new model.) OpenAI’s been a bit more Android about it, slower to force migrations, perhaps because they’ve spent more money on data center capacity for existing models (even if it’s caused them to fall a bit behind in the arms race.) Meanwhile, Google has been slow to release new Gemini models to the point where, currently, some people are arguing they’re no longer a “frontier” lab.

What does this mean for those of us building systems on top of these models? Basically that, if you want to stay in the frontier fast lane, you’ll need to stay on top of all the announcements from the labs and be ready to recalibrate your agents two or three times per year.

Conclusion

Humans have been comparing gear probably since the days of stone tools. The epic poem Beowulf – one of the oldest surviving pieces of English literature – devotes entire stanzas to geeking out over gear: “I never heard before of a ship so well furbished with battle tackle, bladed weapons and coats of mail… Their mail-shirts glinted, hard and hand-linked; the high-gloss iron of their armour rang.”

That primordial love and respect for cool tech is alive and well today. Guitarists will obsess over what amp and pickups someone used in their rig. Gearheads will discuss a car’s suspension geometry for an hour. Photographers have their lenses and chefs have their knives. In the best case, none of them are really talking about equipment: they’re talking about the craft it enables.

Personally, I love gear talk… in the right context, with the right people, and in the right spirit. Put me in a room with other practitioners and I’ll burn an hour arguing about whether Claude’s new tokenizer gets better results or just lets Anthropic charge more. That’s the sort of “shop talk” that makes you sharper. 

But there’s a difference between a skater trying out new gear and practicing new tricks alone in an empty lot versus what you bust out during a competition in front of judges. In our world, a client’s system is the competition, not the practice session, and you don’t debut anything new there until you’re certain it’s going to work.

So what’s the takeaway here? 

Basically, if you’re just using AI for ad-hoc personal productivity, then by all means play around with different providers, try the different tiers, and go with whichever one feels right to you.

But if you’re building serious applications for high-stakes situations or enterprise scale, that’s when you need to put in the hours, sweat the details, and dial in your setup. Knowing which 2% benchmark bump matters and which is noise, which guardrail overreach will actually break a workflow versus which one you’re imagining: that only comes from watching real systems in production, not press releases and blogs.

Thinking back to skating with my friends in the parking lot, Edward didn’t ride that department-store board for a month because he stopped caring about gear. He rode it because that’s what he had, and he’d put in enough hours that the board was no longer the primary variable that mattered 

At some point, you simply have to shut up and skate.