AI features that survive contact with users

An AI feature can be demonstrated in an afternoon and still be months from being usable. The gap is not model quality. It is everything around the model: what it reads, how you know it is right, and what happens when it is not.

Decide how you will measure it first

Before writing the feature, assemble a set of real inputs with known good outputs. A few dozen is enough to start. Without it, every change is judged by whether the output felt better, which is not a basis for shipping anything.

This is the step most often skipped and the one that most reliably separates features that improve from features that change.

The content is the hard dependency

Retrieval over documentation is only as good as the documentation. If a third of it is out of date, the system will answer confidently from the stale third, and confident wrongness is worse than an empty answer.

Auditing and fixing the source material is unglamorous and routinely the highest-value part of the project.

Narrow beats general

A feature that answers one category of question well is useful and evaluable. An assistant that answers anything is neither, because there is no way to tell whether it is getting better.

Pick the narrowest version that would genuinely help someone, and ship that.

Design for being wrong

The model will be confidently wrong sometimes; that is a property, not a bug to be eliminated. The interface has to make verification easy — show the source, make the uncertainty visible, and keep the human in the loop for anything consequential.

Budget for the running cost

Inference cost scales with use, unlike most features, which is a different financial shape to the one your planning is used to. Model the cost at ten times current usage before launch rather than after.

Be willing to stop

Set a threshold on the evaluation set in advance, below which the feature does not ship. Deciding that afterwards, with the work already done, almost never goes the honest way.