Software that knows when to call a human

Almost every conversation about artificial intelligence circles around how much models can already do. I find myself asking something else more often: what happens at the moment they cannot?

A product is tested in that instant. A demo trick answers every time, with the same confidence, right or wrong. A real product knows the difference between I am sure, I think so and this is not mine to answer.

Confidence is a feature too

When a system answers everything with equal firmness, people quickly learn to doubt all of it. Confidence, the kind a system asks for and the kind it gives, is a feature you design, not a coincidence you wait for.

Designing it takes work, but the rules are few:

• Say what you know and what you do not. An answer that admits doubt is worth more than a polished one that fails in silence.
• Make the handover cheap. Calling a person cannot be a dead end with a form. It has to be quick, with the context already on its way, so nobody repeats what they just said.
• Keep the why. Every time the system asks for help, the reason is recorded. That is where the map of what is still left to learn lives.

Knowing when to stop is a kind of intelligence

We talk about autonomy as a ladder where going higher is always better. It is not. Good autonomy grows where things can be verified: where a mistake is cheap, or where someone can check the result quickly. Everywhere else, the intelligent move is to stop.

It sounds like the opposite of enthusiasm, and perhaps it is just its grown-up version. The people we trust most are not the ones who know everything. They are the ones who say, without drama, this is beyond me, let me bring someone who knows.

What I want to build

Software that knows the size of its own certainty, and shows it. That treats the human as part of the design rather than a failure of the system. And that hands over in such a way that nobody notices a fall, only a handoff.

Models will keep getting better. This part, knowing where their confidence ends, will keep being ours.