Designing AI agents for decisions that keep changing

I want an exotic someday. I don't have exotic money. Watching Mat Armstrong rebuild cars is a pretty effective way to convince yourself there might be a loophole.

What looks like superficial damage could be something I'd learn to fix with a modest investment in tools, or it could turn out to be specialized carbon-fiber work that nukes the economics. I haven't bought a car at auction, but I wanted to understand whether the rebuild route could make sense for me before getting attached to the idea. That became a personal project: building an agent to help me investigate a rebuild before bidding.

I'd be paying a specialist for any work I couldn't do myself, so I wanted those costs visible before they got buried in a bid ceiling. A rebuild that pencils out for a shop with the equipment already paid for could still be a terrible buy for me.

I'm just as particular about the valuation. If I'm looking at a manual V10 R8, I want to see which sales support the number and how the comparison accounts for the gearbox and facelift. I'd be looking through forums and ownership accounts to understand what people value about that version. I don't want to reverse-engineer the model's estimate only to discover it treated cars I'd never consider interchangeable as equivalent comps.

One early analysis assumed I could win an SF90 for $0. No one wins an SF90 for $0. What I needed was a maximum bid I could justify, with enough of the reasoning exposed to interrogate it. If that ceiling was zero, explain why.

I wanted the agent to investigate the assumptions that could change that number. That meant its recommendation would have to survive new information about the car, the buyer and the market—or be withdrawn when it no longer held up.

Consider the same problem in a purchasing workflow. A supplier quotes a price for a particular quantity and delivery date, someone reviews it, and an agent recommends the purchase. If the order subsequently changes, the quote may still be authentic and correctly extracted while its approval covers an order nobody is placing anymore.

Evidence can remain true while losing its ability to justify a decision. Keeping the source document is only part of the work; the system also has to retain what was reviewed and establish whether that review applies to the purchase being proposed now.

For a rebuild, discovering additional damage can leave you with a perfectly genuine quote that no longer covers the work. The application binds inspection and quote reviews to the recorded repair scope, so a change reopens those requirements and withholds the maximum bid it can justify. The earlier evidence remains in the record.

At present, this works by comparing a fingerprint of the scope. It is a conservative implementation: a change can force another review even when a particular quote would still apply. I'd like to reduce that burden, but a more selective approach would need to demonstrate that it catches the changes that invalidate the earlier bid.

In the agent loop, the coordinator chooses an investigation from the work currently available. An adapter performs that investigation, after which the application evaluates the evidence, updates the decision record and determines what can happen next.

Some investigations need an ordinary API call; others need image analysis or a research worker pursuing a bounded question. I want to be able to improve any of those implementations without also changing the rules for accepting evidence or authorizing another paid call.

The coordinator gets a compact view of the buyer's constraints, the unresolved questions, the evidence already available and the remaining research allowance. A worker investigating one repair gets the context relevant to that repair. The application retains the fuller history, including the reviews and assumptions behind earlier decisions.

When the supplier's delivery date changes, the coordinator's next view should reflect that the old approval no longer applies and that purchasing is unavailable. Making the model reconstruct those consequences from conversation history would put too much weight on what it happened to remember. There is a cost to compacting context, though: I currently cap evidence summaries deterministically, and those limits can omit information that would have changed the next investigation.

OpenAI's function-calling API lets an application declare its tools while restricting calls to an allowed subset. I apply a similar idea to individual investigations: each has an identifier, a reason and an execution allowance. The coordinator selects one against the revision it read, then the application rechecks eligibility and reserves the allowance before execution. A choice made against an old view doesn't get to authorize spending against a changed record.

Whether an action is permitted and whether it is worth doing are separate questions. I want the model making judgments about which investigation to pursue, with ownership, spending authority and stale-state checks enforced by the application.

Suppose the auction has already passed the maximum bid you could justify even with the cheapest repair in your estimate. Narrowing that repair range won't rescue the deal under those assumptions. New market evidence might change the picture, but another repair search could just cost money while giving you more time to talk yourself into the car.

An investigation earns its place through the prospect of changing the decision, taking into account how likely it is to yield useful evidence and what it costs to obtain. The widest uncertainty range doesn't tell us all of that. The current implementation nevertheless uses the spread between high and low repair estimates as a simple ranking heuristic, and removes paid repair research when qualified market evidence shows that even the optimistic economics cannot justify the observed bid.

Before crediting a model with better research judgment, I'd want to see it beat that policy. Following the existing ranking and explaining each choice fluently wouldn't establish an improvement.

I also wouldn't freeze the architecture around the limitations of the current model. Anthropic describes a context-reset intervention in its Managed Agents architecture that helped one model and became unnecessary for a later one. A better coordinator might warrant a different context policy or the ability to propose investigations the current menu doesn't express, while still working within the buyer's authority and remaining allowance.

Keeping those rules outside the conversation also matters when work is interrupted. OpenAI's Conversations API can persist messages, tool calls and outputs across sessions, while the application still has to account for what was spent and who can authorize further work. If a paid investigation was interrupted before its result was saved, it may already have incurred a cost. Starting a fresh conversation shouldn't silently buy it again, nor should a late result revive a task the owner stopped.

I can challenge the application rules directly with stale revisions, attempts to promote model claims into verified source facts, and new sessions trying to reset the task budget. I want to inspect the resulting record, particularly when the agent's final message says it handled a change successfully.

The coordinator needs a comparison against a simple policy using the same evidence environment, available actions and allowance. I'd measure how far each gets toward a justified decision, along with spend, latency and unnecessary work. The scripted coordinator I use in runtime tests takes the first offered action and exercises the runtime; optional live model trials use local test evidence. Those checks don't yet demonstrate better research selection.

Useful buying decisions require another standard entirely. For a rebuild, I'd need to compare eventual costs and outcomes with estimates recorded before the outcome was known. Synthetic cases can't establish that calibration, and I haven't demonstrated profitable rebuilds with this system.

I'd also test the agent while someone else changes the requirements. In Sierra and its collaborators' τ²-bench, both the agent and a simulated user can act on the telecom environment, and success depends on the resulting state. For this application, I'd use that idea to test whether an agent pursuing a recommendation notices when another actor has changed the conditions that made it valid.

Withdrawing a recommendation after discovering more damage could be the most useful result of the investigation. Asking for an inspection might be the only responsible next step, whereas exhausting the research budget tells us little about the car itself. An evaluation that lumps those outcomes together as failures to finish would give me a poor basis for deciding whether the system helps.

What I'd like to get out of this project is a clearer view of which rebuilds deserve a closer look, what I'd have to establish before bidding, and where the numbers stop working for my circumstances. The architecture should let us measure whether a better model gets there with less wasted investigation and fewer unsupported assumptions.

I still want the exotic. I'd also like the system to be useful when the answer is "buy the used one."