A pilot is a test with a decision attached. If nobody can say in advance what result would make them stop, the pilot is a demonstration with a longer calendar, and it will end the way most demonstrations end: with a decision to continue, taken by nobody in particular. Writing the stop condition down first is uncomfortable, and it is the only thing that makes the rest of the pilot honest.
Prove the data path first. Most pilots fail on plumbing rather than on modelling: permissions that were never granted, a nightly export that arrives late, a document format that changed in March, a field that is populated by hand and populated inconsistently. The pilot should run the whole path, from the source system to the screen a person actually uses, on a schedule that matches production. A model that performs well on a prepared file has proved very little.
Prove the error rate on a fixed test set. Take real cases, agree the correct answer for each one before the build, then record every disagreement, including the ones that feel unfair. Report failures by type, because a system that misreads one field in ten and a system that produces one confident wrong answer in a hundred are different products with different costs. Record what a wrong answer costs in each case: a minute of correction, a customer email, or a refund. That estimate should come from the team that does the work, not from the project, because the team is the one that pays it.
Prove the human loop. Someone has to review the output, and the pilot must show what that review costs. Measure the time a reviewer spends, the corrections they make, and what they do when they disagree. If checking the system takes longer than doing the task, the pilot has proved a negative result, and a negative result delivered in four weeks is a good outcome. It is cheaper than a production system that quietly adds work to a team that was already busy. Review time is part of the cost of the system, and it belongs in the pilot result.
Prove the operating cost and the exit. Write down what keeps the pilot running: who refreshes the data, what happens when a source changes, how the output is monitored, and who owns the code when the engagement ends. Include what a rollback looks like and how long the system could be switched off. A pilot that cannot be turned off is not a pilot; it is an unplanned production dependency. Name the person who can switch it off, and the conditions under which they would.
Finally, prove that nothing got worse. Measure the current process before the pilot starts, on the same cases, so that there is a baseline to compare against. Otherwise the only evidence available at the end is a demonstration on cases that were chosen after the fact. A baseline recorded after launch is a memory, not a measurement. The pilot should end in a written note that states what was tested, what the results were, what it would take to run in production, and what would have to change for the answer to be different.