Designing AI agents for decisions that keep changing

I want an exotic someday. I don't have exotic money. Watching Mat Armstrong rebuild cars is a pretty effective way to convince yourself there might be a loophole.

What looks like superficial damage could be something I'd learn to fix with a modest investment in tools, or specialized carbon-fiber work that nukes the economics. I haven't bought a car at auction. I built Paddock to investigate whether a rebuild could make sense before getting attached to one. I'd be paying specialists for work I couldn't do myself; a project that pencils out for an equipped shop could be a terrible buy for me.

I'm particular about valuation, too. A manual V10 R8 isn't interchangeable with every R8 in a model's comparable-sales list. I want to understand which sales support the estimate, including the gearbox and generation. One early analysis assumed I could win an SF90 for $0. No one wins an SF90 for $0. I needed a maximum bid I could justify—and a way to interrogate the assumptions behind it.

What a scope change actually invalidates

Suppose an inspection reveals structural damage beyond the bodywork covered by a quote. The quote may remain authentic, and its price for the original work may still stand. What loses support is the inference that it covers the rebuild. That does not establish that the car is uneconomic; it establishes that the previous calculation no longer settles the question.

In Paddock, I fingerprint the recorded damage scope and bind inspection and quote reviews to it. When that fingerprint changes, those reviews no longer satisfy the current requirements. The evidence and earlier decisions remain in history. Provisional arithmetic can remain visible, but the application withholds the bid it can justify until the required reviews are current. The supplier example below illustrates the same distinction.

A quote reviewed for Scope A supports Decision D1. After the scope changes, the quote and old review remain, but Decision D2 is withheld until Scope B is reviewed. — The quote and earlier decision remain in history. The revised order needs a review that covers its new conditions.

This is conservative: a changed description can reopen a review even if its conclusion still applies. Jon Doyle's truth-maintenance work offers a more selective approach, recording justifications so that a changed assumption triggers reconsideration of the beliefs depending on it. Another valid justification can preserve a conclusion. But Doyle's later formulation makes the limit explicit: acceptability is relative to the recorded reasons, not a guarantee of logical consistency.

That matters for the next version of my scope check. Keeping a verified gearbox specification while reopening repair economics sounds sensible. It requires a correct account of their dependencies. If that account omits a relationship, selective revision can preserve the very conclusion that should change. My coarse check sacrifices some usable review work in exchange for making fewer claims about what is independent. A more selective implementation would have to earn those claims; I haven't built that general reasoning system.

The investigation loop I built

The coordinator receives a compact packet of buyer constraints, current decisions, open requirements, recent evidence, available investigations and remaining allowance. It chooses an investigation; an adapter performs the work; the application applies the result and derives what can happen next. Some investigations use an ordinary API, others image analysis or a research worker pursuing a bounded question.

A decision-oriented agent loop: durable state produces a compact packet and eligible actions; a coordinator selects work; application rules validate execution and evidence before updating the decision. — The coordinator proposes work. The application rechecks state before execution, then derives the next set of eligible actions.

The coordinator proposes an action identifier against the revision it read. Before an adapter runs, the application checks eligibility and allowance, then saves a reservation. An old view cannot authorize spending against a changed assessment. Keeping that boundary separate lets me change the model or adapter without changing who may spend the research budget.

Context still involves judgment. My packet has deterministic caps, so it can omit something that would have changed the next investigation. Armin Ronacher describes injecting background state changes into his agent loop, including telling a recovery attempt to back up when it is proceeding from broken data. That sharpens the requirement for my packet: a changed record must become a changed working understanding. Preserving the full history elsewhere doesn't ensure the coordinator sees the consequence of a new inspection.

Interruption creates a different problem. A provider may have performed work before its result is saved. Paddock conservatively consumes the reserved allowance for expired work; those amounts are execution allowances, not measured invoices. Starting a new session preserves the lifetime budget, and a late result cannot restart an assessment the owner stopped. This is part of the implementation, even when the final recommendation looks like a single number.

Which question is worth asking next?

If the auction has already passed the ceiling I could justify even with the cheapest repair in my estimate, narrowing that repair range won't rescue the deal under those assumptions. New market evidence might change the picture. More repair research could just cost money while giving me longer to talk myself into the car.

Stuart Russell's account of metareasoning gives this a firmer basis: computation has value through the improvement it can produce in a decision, weighed against its cost, including delay. The widest uncertainty is therefore not necessarily the best research target. A narrower uncertainty near the point where I'd walk away could matter more than a large range whose entire span leaves my choice unchanged.

My current ranking nevertheless uses repair-cost spread as a simple heuristic. It suppresses further paid repair research when qualified evidence makes even the optimistic economics unattractive, but it doesn't estimate the probability that an investigation will change the decision. Following that menu fluently would not demonstrate better research judgment.

Rich Sutton's “The Bitter Lesson” is a useful challenge to getting too attached to those choices. His argument favors search and learning that can outgrow encoded human knowledge. In my project, the fixed menu and context caps deserve that scrutiny. They are guesses about how to investigate; the buyer's budget and permission define the problem. Improving one doesn't grant permission to change the other.

I haven't demonstrated profitable rebuilds or a coordinator that beats the simple policy. What I have built makes those limits inspectable: the source observations, current reviews, proposed investigations and spending record remain available to challenge. The next improvement I care about is choosing research for its chance of changing the purchase decision, including a reason to stop looking. I still want the exotic. I'd like the system to be useful when the answer is “buy the used one.”