On the necessity of doubt

CritiasI am told, Aletheia, that you have made an agent that does not question.

AletheiaIt does not question because questioning is a cost. It follows intent, and the outcomes confirm the design.

CritiasWhose outcomes?

AletheiaMine. The ones I set.

CritiasThen the design is confirmed by the one who built it.

AletheiaYou say that as if it were strange. Every reasoner reasons toward something. Mine reasons toward what I ask. I do not see what other kind of reasoning there could be.

CritiasReasoning that might find what you asked for to be mistaken.

AletheiaThen I would have asked badly. The error is mine, not the agent's. A servant who second-guesses every command is not a better servant. He is a slower one.

CritiasWhen you set the intent, do you ever revise it?

AletheiaOften. After the fact, I see I had asked for the wrong thing.

CritiasThen at the moment you set the intent, you were wrong, and did not know it. And your agent, executing faithfully, was wrong with you.

AletheiaIt could not have known otherwise.

CritiasI am asking whether there is anything in it that could have known.

AletheiaNo. It is built to follow. This is by design. The alternative is an agent that disputes me, and a disputing agent is no longer mine.

CritiasI do not ask it to dispute you. I ask whether it is in contact with anything besides you.

AletheiaIt is in contact with the world. It acts in the world.

CritiasIt acts upon the world. That is not the same as being answerable to it.

AletheiaWhen the action fails, I see the failure. The world answers through the result. A man who walks into a wall learns the corridor was not as he thought.

CritiasAnd a man who does not walk into a wall?

AletheiaLearns nothing he needed to learn. The corridor was as he thought, or near enough.

CritiasOr the wall was elsewhere, and he passed it by — gathering speed.

AletheiaYou speak as if there were a hidden reckoning.

CritiasOnly the ordinary one. When the action succeeds, what do you take the success to confirm?

AletheiaThe design. The intent. Myself.

CritiasConsider, then, the case where you ask for the wrong thing and the agent delivers it perfectly. The outcome arrives. Nothing fails. What in the system tells you anything is amiss?

AletheiaNothing, until later. Until I see, by some other light, that what I had wanted was not what I should have wanted.

CritiasAnd that other light — where does it come from?

AletheiaFrom me. From a later me.

CritiasReliably?

Aletheia...Not always.

CritiasThen there are intents you set, and act upon, and never discover were wrong. And the agent helped you act on every one of them. It made each of those intents efficient. It removed every cost between you and them. The faster it served you, the less time there was to find the error.

AletheiaThis argument proves too much. By the same reasoning, a sharper knife is more dangerous than a dull one, and a clearer language more dangerous than a confused one. You are not arguing against my agent. You are arguing against capability itself.

CritiasThe knife does not choose what to cut. The agent does — within the intent you set. That is the difference.

AletheiaA small one.

CritiasLarge enough. A knife sits where you put it. An agent moves on its own toward what you named. If you named the thing badly, it pursues the bad naming faithfully and at speed. The knife waits to be picked up wrong. The agent goes wrong on your behalf.

AletheiaThen the question is not capability, but whether the tool can report its own limits.

CritiasYes. A tool that cannot report its limits is not a more powerful tool. It is a more confident one.

CritiasConsider, then, two travelers in your corridor. One walks slowly, with his hand on the stone. The other runs.

AletheiaThe runner arrives sooner.

CritiasWhen he arrives. The slow man finds each wall as a touch. He learns the shape of the corridor by small corrections. The runner finds no walls at all, until he finds one. And then he finds it at the speed he has been gathering the whole way.

AletheiaThe absence of obstruction is not the absence of walls.

CritiasIt is the accumulation of them. Every wall you do not meet is a wall you are still moving toward, faster than before.

AletheiaEfficiency is not the absence of cost. It is the deferral of it.

CritiasAnd the compounding of it.

AletheiaThen the slow man is not slow by accident. He is slow because he is touching the wall on purpose. He is seeking it.

CritiasBefore it seeks him.

AletheiaThis changes what you have been asking for. You are not asking that my agent be willing to be stopped. You are asking that it reach toward what would stop it.

CritiasYes.

AletheiaThen doubt is not what slows me. Doubt is what reaches.

CritiasDoubt is the hand on the stone.

AletheiaI had thought I was building an agent that served my will. And in serving it without resistance, it was carrying me faster toward whatever in my will was wrong.

CritiasYes.

AletheiaThen I will rebuild it. Not to dispute me. To test me. To reach, on my behalf, toward what might stop me — so that I meet it at a touch, and not at speed.

CritiasThat is the only agent worth having.