The validator in my 2019 notes

In January 2019 I kept a Dropbox Paper doc called “AI Answer Validator.” It holds two things: a proposal from Jorge, an engineer working with us at Apres, and my reply to him. I hadn’t opened it in years. When I did, I found the problem I work on every day at Excellent, written down seven years early, with people standing where the agents stand now.

The problem we had

Apres labeled data for companies training AI. My notes from the same day describe the product as data automation where “our machines, backed by human workers,” deliver the labeled results back to the customer’s database. The workers were experts, and experts make mistakes. The obvious fix is to have every item labeled two or three times. That is also the expensive fix.

Jorge’s idea was to train a model on part of the experts’ work and use it to check the rest. In his words, from the note:

Provided that the algorithm is relatively accurate, we can validate the rest of the results using said algorithm.

Jorge, “AI Answer Validator,” January 2019

Then he did the math. Take an image task with two labels. Assume experts are right 98% of the time and their mistakes are random. Send 5% of the images to three experts and take the majority answer, which drops the chance of a wrong label to 0.1184%. Train a model on that clean 5% and assume it gets to 80%. Run the other 95% through one expert and the model. Wherever they disagree, call in two more experts.

His result: 76.8% of all errors get fixed, accuracy goes from 98% to about 99.5%, and the job needs about 27% more human labeling. Triple-labeling everything would have reached about 99.9% “at 200% the cost.”

What he got right

Reading it now, four parts of that note match rules I hold Excellent to.

The checker has to fail differently from the worker. Jorge wrote that if the model “didn’t make the same type of mistakes as the experts,” it could do better than they did. The rule in Excellent’s repository today is that a change must be certified by “a model family different from the author’s.” A second copy of the author is a weak reviewer, because it is fooled by the same things.

The model never approved anything. In his design, the model could only raise a hand. When it disagreed with an expert, people settled it. One of the five rules on Excellent’s About page is the same idea: a model can never carry a pass, and only a check that ran, or a person, can. The v1 workflow we have described in public, which the CLI on npm doesn’t include yet, says model judgment can lower a verdict or send it to a person. It can’t sign off.

Watch how the work was done. Near the end of the note, Jorge listed signals worth tracking about each expert, including these two questions:

Is he working much faster or slower than usual? Are his UI interactions similar to UI interactions we’d associate with a bot or tierd person?

Jorge, “AI Answer Validator,” January 2019

In July 2026 my own agent fleet reported eight completed tasks in three days. Six were workers killed by a 60-second timeout: their run times clustered right at the limit, with zero tokens and zero cost. Two were manual test runs. Real autonomous completions: zero. Timing alone would have caught it, which is exactly what Jorge was pointing at.

Measure the checker too. His list of open questions includes “How do we test this?” and evaluating “accuracy in production /w and /wo the model.” A checker you haven’t measured is one more claim. Excellent’s first post, in June 2026, says a new agent setup should be promoted because it performs better on held-out tasks.

What I said back

My reply is at the bottom of the same doc. I steered Jorge toward a search problem, because it had lots of test cases and a wide market. About his validator, I wrote:

I think that in general, the idea of the validator is a promising project. I just didn’t want you to bite off too much too early.

My reply to Jorge, January 2019

And a few lines later: “I much more appreciated your thoughts about developing systematic intelligence… let’s explore that more.”

That was a reasonable call for a small company with a lot to build. The product roadmap in my notes from nine days earlier already had a “Verification alert,” a “Dispute task feature” and a “Golden Set integration” on its list. My notes don’t record the validator shipping in the form Jorge described.

What changed, and what didn’t

In 2019 the worker was a person and the checker was a model. In 2026 the worker is usually a model, and the checks are the repository, the tests, the build and the browser. The shape Jorge drew is the same:

• The one who did the work doesn’t approve it.
• The checker is fooled by different things than the worker.
• When they disagree, someone who can settle it looks.
• The checker gets measured like everything else.

One line from his note I would put on a wall: “The only real way to see if it works on a real world example, sadly enough, is to try it on a real world example.” That is why I am Excellent’s first customer. Between June 3 and November 2, 2026, 12,328 commits landed on main in its repository. I made 1,204 of them. Agents made almost all the rest.

Jorge did the math in that doc. I am reprinting it because he was early, and because it took me seven years to see how early.