Imagine a small app that turns a messy customer note into a clean job summary. On Monday, it correctly notices that the customer's artwork is still awaiting approval. On Tuesday, the same note comes back with the artwork listed as ready. The screen looks identical. The app has not released a new version. What changed?
One possible answer is beneath the app: the AI model provided by the operating system.
This is no longer just a hypothetical maintenance issue. In its Foundation Models update notes, Apple tells developers that the on-device model changes when a person updates to a new version of iOS, iPadOS, macOS, or visionOS. Apple explicitly recommends testing app prompts with the new model to verify behavior. That does not mean every update breaks an AI feature. It means the model is a changing dependency, even when the app's own code stays put.
The app is more than its code
Traditional software has dependencies too. A browser update can change layout. A map service can return a new route. But AI makes the dependency unusually visible to the person using the product: the answer itself can change.
Suppose a quoting assistant receives this note:
Twelve uniforms needed by Friday. The logo file is attached, but the customer has not approved the final artwork.
A useful result might extract the quantity and deadline while keeping artwork approval marked pending. A different model response might choose different wording, ask for more context, or—in a failure case—treat the attachment as approval. The example is illustrative, but the design problem is real: a fluent sentence is not the same thing as a reliable handoff.
Apple's prompt-evaluation guidance explains that outputs can vary even for the same input, both because generation is probabilistic and because the underlying model can be updated. Testing one polished demo is not enough to know how the feature behaves across ordinary, messy work.
What should stay the same?
The goal is not to demand identical sentences after every update. A new model may produce a clearer summary or catch a detail an older one missed. The goal is to decide which facts and boundaries must survive any rewrite.
For the quoting example, those boundaries might be:
- Pending artwork must remain pending.
- The system must not invent a price or delivery promise.
- Missing information must be visible rather than filled in confidently.
- A person must review the quote before it goes to the customer.
Those rules can become a small set of repeatable checks. Start with five real, permission-safe examples: one straightforward request, one ambiguous one, one with conflicting details, one missing a required field, and one where irrelevant text tries to distract the assistant. Run them against the current behavior. Then run them again when the model or operating system changes. Record whether the important facts survived, not whether every sentence matches word for word.
This approach is close to what Apple recommends: begin with a handful of scenarios and a single quality criterion, then expand the evaluation as you discover failure modes. A useful first criterion here would be: Does the summary preserve every unresolved customer decision?
The interface needs room for uncertainty
Testing catches problems before release. Product design helps when an unexpected case still gets through.
Show the source note beside the generated summary. Mark extracted facts separately from suggestions. Let the user correct a field without rewriting the entire result. Require confirmation before an AI-generated quote, message, or order becomes a real commitment. These are ordinary interface choices, but they make a model change easier to notice and easier to recover from.
The same principle applies beyond phones. Any product that depends on a model it does not fully control needs a way to notice when behavior shifts. The mechanism may be an OS update, a cloud-model upgrade, a changed prompt, or new input data. The exact cause differs; the responsibility to check the experience remains with the people building the product.
A useful question for the next update
When your phone updates, you rarely ask whether its calculator still adds correctly. With an AI-powered feature, the more useful question is: Which decisions does this feature help people make, and how would we know if it started handling one differently?
Pick one important decision. Save a few real examples. Define what must remain true. Run them again after the next model change. That small habit can turn a surprising update into a manageable product change.
Source check: September 29, 2026. Apple's Foundation Models updates document OS-linked model changes; its evaluation guidance explains how to test variable model output. The quoting scenario above is illustrative.
