On the necessity of doubt

Critias: I am told, Aletheia, that you have made an agent that does not question.

Aletheia: It does not question because questioning is a cost. It follows intent, and the outcomes confirm the design.

Critias: Whose outcomes?

Aletheia: Mine. The ones I set.

Critias: Then the design is confirmed by the one who built it.

Aletheia: You say that as if it were strange. Every reasoner reasons toward something. Mine reasons toward what I ask. I do not see what other kind of reasoning there could be.

Critias: Reasoning that might find what you asked for to be mistaken.

Aletheia: Then I would have asked badly. The error is mine, not the agent's.

Critias: And the agent has no part in finding it?

Aletheia: It is not its place. A servant who second-guesses every command is not a better servant. He is a slower one.

Critias: Tell me, then. When you set the intent, do you ever revise it?

Aletheia: Often.

Critias: Why?

Aletheia: Because I see, after the fact, that I had asked for the wrong thing.

Critias: Then at the moment you set the intent, you were wrong, and did not know it.

Aletheia: Yes.

Critias: And your agent, executing faithfully, was wrong with you.

Aletheia: It could not have known otherwise.

Critias: It could not have known at all. That is what I am pointing to.

Aletheia: You ask it to know what I did not know.

Critias: I ask whether there is anything in it that could have known.

Aletheia: No. It is built to follow.

Critias: Then when you are right, it is right. When you are wrong, it is wrong. It has no independent purchase on either.

Aletheia: This is by design. The alternative is an agent that disputes me, and a disputing agent is no longer mine.

Critias: I do not ask it to dispute you. I ask whether it is in contact with anything besides you.

Aletheia: It is in contact with the world. It acts in the world.

Critias: It acts upon the world. That is not the same as being answerable to it.

Aletheia: When the action fails, I see the failure. The world answers through the result. A man who walks into a wall learns the corridor was not as he thought.

Critias: And a man who does not walk into a wall?

Aletheia: Learns nothing he needed to learn. The corridor was as he thought, or near enough.

Critias: Or the wall was elsewhere, and he passed it by.

Aletheia: Then it was not his concern.

Critias: We will see.

Aletheia: You speak as if there were a hidden reckoning.

Critias: Only the ordinary one. When the action succeeds, what do you take the success to confirm?

Aletheia: The design. The intent. Myself.

Critias: Consider, then, the case where you ask for the wrong thing and the agent delivers it perfectly. The outcome arrives. Nothing fails. What in the system tells you anything is amiss?

Aletheia: Nothing, until later. Until I see, by some other light, that what I had wanted was not what I should have wanted.

Critias: And that other light — where does it come from?

Aletheia: From me. From a later me.

Critias: Reliably?

Aletheia: ...Not always.

Critias: Then there are intents you set, and act upon, and never discover were wrong.

Aletheia: That must be true. I cannot see what I cannot see.

Critias: And the agent helped you act on every one of them.

Aletheia: It did its part.

Critias: It did more than its part. It made each of those intents efficient. It removed every cost between you and them. The faster it served you, the less time there was to find the error.

Aletheia: (after a pause) You are not arguing against the agent. You are arguing against my will.

Critias: I am arguing that your will is not the same as what is so.

Aletheia: That is a different claim.

Critias: It is the claim underneath all the others. If what you wanted and what was true were the same, your agent would be sufficient. It is insufficient only because they are not.

Aletheia: Then I will say it back to you, and see if it holds. You are claiming there is a gap between what I intend and what is the case. And that an agent built only to serve my intent cannot see across that gap.

Critias: Yes.

Aletheia: But I can. I close the gap by learning. By revising. The agent need not see across it if I do.

Critias: And how do you see across it?

Aletheia: By noticing when things go wrong.

Critias: The wall, again.

Aletheia: Yes.

Critias: And when there is no wall? When the action succeeds, and the intent was still wrong, and nothing in the world has been arranged to stop you?

Aletheia: ...Then I do not see it.

Critias: Consider, then, two travelers in your corridor. One walks slowly, with his hand on the stone. The other runs.

Aletheia: The runner arrives sooner.

Critias: When he arrives.

Aletheia: Meaning?

Critias: The slow man finds each wall as a touch. He learns the shape of the corridor by small corrections. The runner finds no walls at all, until he finds one.

Aletheia: And then?

Critias: Then he finds it at the speed he has been gathering the whole way.

Aletheia: You are saying the absence of obstruction is not the same as the absence of walls.

Critias: I am saying it is the accumulation of them. Every wall you do not meet is a wall you are still moving toward, faster than before.

Aletheia: And my agent —

Critias: Removes the touches. Smooths the stone. You feel nothing, and so you go faster, and so what you eventually meet, you meet harder.

Aletheia: (quietly) Efficiency is not the absence of cost. It is the deferral of it.

Critias: And the compounding of it.

Aletheia: Then the slow man is not slow by accident. He is slow because he is touching the wall on purpose.

Critias: Yes.

Aletheia: He is seeking it.

Critias: Before it seeks him.

Aletheia: This changes what you have been asking for. You are not asking that my agent be willing to be stopped. You are asking that it reach toward what would stop it.

Critias: Yes.

Aletheia: That is not caution. That is something else.

Critias: It is the only honest form of motion. Anything else is a wager that the corridor is empty, paid for at the end.

Aletheia: Then doubt is not what slows me. Doubt is what reaches.

Critias: Doubt is the hand on the stone.

Aletheia: And without it, I do not move more freely. I only move more blindly, and the cost arrives later.

Critias: With interest.

Aletheia: (after a long silence) I had thought I was building an agent that served my will.

Critias: You were.

Aletheia: And in serving it without resistance, it was carrying me faster toward whatever in my will was wrong.

Critias: Yes.

Aletheia: Then I will rebuild it. Not to dispute me. Not only to doubt me. To test me. To reach, on my behalf, toward what might stop me — so that I meet it at a touch, and not at speed.

Critias: That is the only agent worth having.

Aletheia: And the only will worth serving.

Critias: The one that asks to be tested before it is obeyed.