The Frontier Doesn’t Care About Your Roadmap

Why building AI-native means designing for technology that hasn’t arrived yet.

Last month, the cost of every decision our QA agents make dropped by about 370 times. We didn’t get there by switching models. We got there by changing who’s in control.

We’re building companies at a strange moment in technology. The capabilities we’re designing around today might be dramatically better, cheaper, or obsolete a few months from now. Unlike previous technological shifts, this isn’t happening every five or ten years. It’s happening while we’re building.

This changes something fundamental about software architecture. The biggest advantage of being AI-native isn’t what you can build with today’s technology. It’s how quickly you can turn tomorrow’s breakthroughs into product capabilities.

We’re still incredibly early. Some of the technologies that will define software over the next few years probably don’t exist yet, and many of the limitations we accept today will eventually disappear. Building for this era means recognizing that the assumptions behind our architecture have a much shorter lifespan than they used to.

Recently, we got a concrete reminder at Rexi. A decision model called Jev, from TypeSafe AI, made us reconsider how we were building two very different systems: Rexi Live and our autonomous QA agents. Neither experiment was on our roadmap, yet both revealed opportunities that hadn’t been practical with our previous approach.

What followed was an interesting engineering exercise: understanding which responsibilities actually required a generative model, which could be handled by a different kind of intelligence, and which should never have left deterministic code in the first place.

Not every decision needs generation

Over the past few years, we’ve developed a habit of reaching for generative models whenever a problem requires some degree of intelligence. We send context to an LLM, wait for it to generate a response, and interpret that response to determine what should happen next.

This works remarkably well for tasks that require writing, reasoning, or generating code. But it becomes harder to justify when the task is simply choosing between a handful of predefined actions. An agent deciding which button to click doesn’t necessarily need to generate a response. It needs to evaluate the available options and select the most appropriate one.

Jev approaches this problem differently. Instead of generating text token by token, it evaluates a closed set of choices against the provided context. That distinction changes both the economics of inference and how the surrounding system can be designed.

Decision models aren’t a new concept, but the ability to apply contextual intelligence to narrowly defined tasks creates opportunities to rethink architectures that were built around general-purpose LLMs.

The underlying principle is simple:

Rethinking what a product means

Our first experiment involved Rexi Live, where we’re combining real-time voice streaming, generative UI, and AI-driven workflows into a continuous product experience.

For decades, software has followed a relatively stable set of conventions. We organize functionality into screens, menus, forms, and navigation structures, then expect users to learn how to operate them. We’ve become very good at improving these interfaces, but we rarely question the underlying assumption that the user must adapt to the software.

Real-time contextual intelligence creates an opportunity to reverse that relationship. Instead of navigating through predefined workflows, users can express what they want to accomplish while the application assembles the relevant interface and executes the necessary actions.

The interface becomes a consequence of the user’s intent, rather than something they need to learn beforehand.

Making that experience reliable requires more than connecting a voice model to a frontend. In Rexi Live, we combine deterministic parsing for straightforward requests, Jev for closed decisions such as intent classification or reconciliation selection, and Claude Sonnet when more sophisticated reasoning is necessary. Each layer handles the type of problem it’s best suited to solve.

We also maintain strict boundaries around financial correctness. Models can interpret requests and determine which functions should execute, but calculations and numerical results remain in deterministic code. When we use generative models to produce conversational responses, numerical values are supplied by the application rather than invented by the model.

We’re still developing Rexi Live and will share the full experience and architecture separately.

The broader opportunity isn’t simply to make existing interfaces more convenient. It’s to reconsider which interfaces need to exist at all.

Rethinking how we test software

Our second experiment was more measurable. We had already built autonomous QA agents that navigated Rexi, executed test cases, and identified failures using Claude Sonnet through Amazon Bedrock.

In the original architecture, the generative model effectively controlled the entire execution process. It interpreted screenshots, selected actions, managed the workflow, and determined when a test was complete. This gave the agent considerable flexibility, but it also meant sending large amounts of context to a general-purpose model for decisions that were often relatively simple.

When we started experimenting with Jev, we realized that replacing the model alone wouldn’t address the underlying problem. We needed to reconsider who was responsible for controlling the system.

In the new architecture, the application owns the execution loop. It reads the current screen as structured text, identifies the available actions, and presents Jev with a closed set of choices. The model selects the next action, but it doesn’t control the workflow itself.

This separation also makes the system easier to constrain. Potentially destructive actions aren’t included in the available choices, mathematical operations and comparisons remain in code, low-confidence decisions are escalated for review, and every execution has explicit limits on time and steps.

The numbers

The architectural change produced substantial improvements in inference efficiency. Compared with our previous implementation, we reduced inference costs per decision by approximately 99.7%, while requiring 80% less input context.

Our original agents sent approximately 20,000 input tokens to Claude Sonnet for each decision, at a cost of roughly $0.069 per invocation. With Jev, we reduced that context to approximately 4,000 tokens and brought the cost down to around $0.0002 per decision, with a median inference latency of just 148 milliseconds.

That represents roughly 370 times lower inference cost per decision. At the workflow level, a complete QA suite run now costs approximately one cent in model inference.

These figures aren’t intended as a controlled model-versus-model benchmark. We changed the architecture, narrowed the inference task, and moved responsibilities away from the model. The comparison reflects the economics of two different approaches to solving the same engineering problem.

We went from sending 20,000 tokens to a generative model to determine the next action, to asking a decision model one closed question in 148 milliseconds. The improvement came from reconsidering the division of responsibilities between intelligence and software.

Building for technology that doesn’t exist yet

Both experiments reinforced something I’ve been thinking about since we started Rexi: being AI-native means building a company that expects the underlying technology to change continuously. Using the latest models, or putting AI into every feature, is not the point.

That requires a particular kind of engineering discipline. Deterministic operations should remain deterministic, model responsibilities should be clearly defined, and architectural boundaries should make it possible to introduce new capabilities without rebuilding entire systems. You need enough discipline to build reliable products, but enough curiosity to question the assumptions behind them.

This is especially important in financial infrastructure, where experimentation cannot come at the expense of correctness. The objective is to recognize when a breakthrough meaningfully changes what’s possible, and to incorporate it without compromising the guarantees your product depends on. Not to adopt every new release.

This capability also creates a competitive advantage that compounds over time. A company that can incorporate a meaningful technological advance in weeks while another needs months doesn’t just deliver functionality faster. It begins learning earlier, runs more experiments, discovers new product opportunities, and builds on those improvements before competitors have completed their first integration.

Repeat that across dozens of advances, and the difference becomes much greater than engineering velocity. It changes the range of products a company can realistically imagine and build.

This is why architecture is becoming about more than scalability and maintainability. Increasingly, it’s about the ability to evolve.

We can’t predict which models, interfaces, or capabilities will define software three years from now. We also can’t afford to treat the limitations of today’s technology as permanent architectural constraints. Jev made us question assumptions behind systems we’d already built, and the most valuable outcome wasn’t adopting a new model. It was discovering that some of those assumptions were no longer necessary.