When people and AI disagree: resolve the disputed decision

An AI system recommends postponing a planned launch. The responsible manager considers the date realistic. Both assessments sound plausible. What is missing is a procedure for resolving the difference. Asking for another generated answer may leave the organisation with three opinions and the same unresolved decision.

I would treat this disagreement as an assignment: which specific claim separates the two assessments, and what information could resolve it? The sequence below is my own organisational proposal. It concerns bounded administrative and business decisions. It is not a validated procedure for diagnosis or treatment. The continuing example of a medical technology department is fictional.

What research contributes to intervention

In 2021, Buçinca and colleagues reported an experiment with 199 analysed participants. The task involved substituting food ingredients, and the AI assistance was simulated. Designs requiring independent thought reduced acceptance of incorrect recommendations compared with simple explanatory assistance, without eliminating it. Particularly effective variants received less favourable ratings. The finding concerns that experimental setting and establishes no clinical safety benefit. [1]

Poursabzi-Sangdeh and colleagues conducted four preregistered experiments involving 3,800 people predicting apartment prices. Greater model transparency sometimes improved understanding of its predictions, but could impede correction of large mistakes on unusual cases. A targeted message about possible outliers changed that result in the fourth experiment. The work, published in 2021, does not examine explanations produced by today's language models. [2]

These bounded findings lead me to a design question: how can the disputed matter become sufficiently clear for the responsible person to investigate it? The amount of text on the screen is an unconvincing measure by itself. The proposed sequence needs testing within each organisation and adjustment to make its workload proportionate.

1. State the disagreement in one sentence

In the fictional example, a department is preparing to introduce an internal document repository. The AI summarises approved project records and recommends postponement because training has supposedly not taken place. The project leader remembers a completed training session. The testable disagreement is: was the training required for this launch completed by the agreed cutoff date?

This formulation specifies the subject, condition and time. It keeps a debate about the overall capability of AI from displacing the particular investigation. The participants might mean different groups. A different training version might apply. The system might lack a record. Each possibility requires a different next step.

2. Record the basis for the human assessment

Before generating further explanations, the project leader briefly records the basis for her assessment: invitation date, remembered participant group and presumed completion record. Her uncertainty belongs in the note too. Remembering a session, for example, does not establish complete attendance. The note makes visible which parts of her own position subsequently need confirmation or correction.

For routine cases, this can be one sentence. A decision with larger consequences may require a second responsible person. What matters is giving additional scrutiny a clear subject. Requiring lengthy explanations for every small difference would create considerable work, and its benefit would first need to be demonstrated.

3. Check the authoritative information independently

In this example, the approved training record is opened through the known internal repository. The check covers the participant group, training version and date. A location supplied by the AI may provide a search lead; its existence and content need independent verification. Existing access permissions apply. Resolving disagreement provides no reason to transfer protected records into an additional, unapproved system.

The record shows that the project leader correctly remembers training for the first user group. A second group has since been added to the scope of the launch. Its completion record is missing. Both original statements were too broad. The relevant decision now concerns the launch scope and the outstanding prerequisite for that group.

4. Separate the established facts from the decision

Establishing a fact often resolves only part of a decision. The department could maintain the date for the first group, include the second group later, or postpone the whole launch. Whether an option is acceptable depends on dependencies, commitments and applicable internal requirements. That assessment remains with the responsible function.

The decision owner therefore documents two brief statements: what was established, and what action follows from it, with the reason. With different evidence, the same procedure could confirm the original AI recommendation. Its purpose is a reasoned decision. A high rate of human corrections would provide no more evidence of quality by itself than a high agreement rate.

5. Allow missing information to remain an outcome

Sometimes the decisive record cannot be found. Identify the unresolved prerequisite, the person who can clarify it and an appropriate time for the next decision. If a decision is needed sooner, the designated responsibilities and rules for handling uncertainty apply. Another generated formulation cannot supply the missing information.

I would initially treat a statement such as “I am 95 per cent confident” as part of the answer too. Such a number deserves weight in a decision only when its origin is understood and its relationship with actual correctness has been checked on comparable, independently assessed cases. This is my proposed assessment rule. The two cited studies do not measure the calibration of current language models.

6. Record the cause for future cases

After the decision, a brief cause statement is sufficient in this example: the launch scope was inconsistent across the records. This differs from a missing source, an inaccurate summary or an outdated human recollection. Distinguishing these possibilities helps identify an appropriate change. Additional training, for example, would do little to directly resolve contradictory project scope.

For recurring cases, the organisation can collect a few permitted details: type of disagreement, basis for checking, decision, remaining uncertainty and required follow-up work. The collection needs defined retention and restricted access. Personal information or trade secrets belong there only when their processing for this purpose has been authorised. An abstract description of the error type is often enough.

A short worksheet for the next case

The worksheet can contain six fields: disputed claim, human reasoning, authoritative information, established facts, accountable decision and required change. It stays blank until an actual disagreement justifies using it. Its purpose is to facilitate an investigation, and it can be shortened or refined after initial permitted trials.

What interests me especially is the opportunity for organisational learning. A properly resolved disagreement can reveal imprecise terms, missing sources or unclear responsibilities. Investigating the particular case makes these differences visible. A troubling AI answer can then become a traceable reason to improve the shared basis for decisions.