Teams adding on-device AI to embedded and mobile products routinely specify the intelligent path in full and leave the failure path to be discovered in testing. For a system that must operate without reliable connectivity, this ordering is backwards: the behavior of the product when inference is unavailable is not an edge case but the product’s guaranteed floor, and it determines whether the feature can be depended on at all. This article describes a four-rung inference fallback ladder terminating in a deterministic scripted responder, argues that the deterministic rung rather than the model is what makes such a system dependable, and sets out why a floor is far cheaper to specify before construction than to retrofit afterwards.
Every AI feature has a floor. Very few teams have decided where it is.
The usual sequence is familiar to anyone who has shipped one. The team selects a model, establishes that it fits within the device’s memory and thermal budget, builds the path from user input to model output, and then discovers during testing what happens when that path is unavailable. The answer is generally that the feature stops working, which is treated as a defect and patched.
For a connected consumer application this is survivable. For anything embedded, mobile or intermittently connected it is not, because unavailability is not a defect. It is a routine operating condition, and the system’s behavior under it is the only behavior that can be guaranteed.
Inverting the order costs very little at design time and is close to impossible to arrange later. The question to answer first is: what does this feature do when every clever component is unavailable, and is that acceptable?
In the assistant we built, the answer takes the form of a ladder. The default path is a language model running entirely on the device, which is what allows the feature to function with no network at all. Below it sits a private cloud inference rung, wired but currently defaulted off. Below that, a cloud model reached through our own backend gateway rather than a vendor endpoint, which keeps provider credentials off the device entirely. And below that, a deterministic scripted responder containing no inference of any kind.
That last rung is the one that matters, and it is the one most teams do not build.
It contains no intelligence. It cannot answer a novel question. What it can do is respond correctly to the narrow set of things a user most commonly asks, in a form that is accurate, bounded and instantaneous, and it can do this with no model, no network and no state beyond what the device already holds. It is unglamorous and it is the reason the assistant has never simply failed in front of a user.
There is a design principle here worth stating directly. “Always answers” is a substantially harder requirement than “answers well”, and it is the requirement that determines whether people trust the feature. A system that is excellent nine times in ten and silent on the tenth is experienced as unreliable. A system that is adequate nine times in ten and predictable on the tenth is experienced as dependable. Users calibrate against the worst case, not the average, and the worst case is the floor you either designed or did not.
The reason the floor cannot be retrofitted is structural rather than a matter of effort. A fallback ladder is only meaningful if every rung produces output in the same shape, so that the layers above cannot tell which rung answered. That constraint has to be imposed on the interface before any of the rungs exist. Teams that build the model path first almost always let the model’s response format become the system’s response format, at which point the deterministic rung has to imitate a language model rather than simply return a correct answer, and the abstraction leaks in both directions.
A second constraint follows and is worth the effort on its own merits. Fallback behavior is difficult to trigger deliberately on real hardware. Forcing a thermal throttle, a network partition and a provider timeout in a controlled sequence on a device is slow and unreliable, so in practice these paths get verified rarely and by hand, which is to say they do not get verified.
We addressed this by writing the conversation loop and the provider-selection logic as pure functions with no I/O, pushing the microphone and the network to the edges of the system. Every rung of the ladder, every degradation path and every policy branch is therefore exercisable without hardware. The engine layer currently carries 1,053 automated tests, and the great majority of them cover conditions that would be impractical to reproduce on a device. This is not a testing convenience. It is the only reason we have any evidence that the floor works, and a floor without evidence is an assumption.
None of this is free. A deterministic responder is a body of code that produces no visible feature and that stakeholders will reasonably ask about. It requires someone to decide, in advance and in writing, what the product’s minimum acceptable behavior is, which is a harder conversation than choosing a model. And it constrains the interface of everything above it.
The alternative is a product whose guaranteed behavior is whatever happens to occur when inference fails, discovered by users rather than chosen by engineers.
As on-device inference moves into more embedded and mobile products, the differentiator will not be which model a team selected. Model capability at the edge is converging and will continue to. What will distinguish dependable products from impressive demonstrations is whether anyone specified what happens when the model is not there and did so early enough for the answer to be designed rather than inherited.
Dr. Abtin Aghagolian is co-founder and CTO of Pikd, a London-based spatial computing company building geospatial augmented reality and on-device AI. He holds a doctorate in business administration and is a patent-pending inventor in privacy-preserving augmented reality.
