A timeout after dispatch does not establish whether a tool's side effect committed. A payment service may have accepted a charge even though the caller never received its response. Retrying because the call looked unsuccessful can repeat work that already happened.

When an agent can reissue side-effecting tool calls, treat that as an at-least-once execution problem: duplicate attempts are possible, while successful completion is not guaranteed. Reliability controls belong in the orchestration layer and at the tool boundary, not in model output. The model may propose another attempt; code must decide whether that attempt represents an existing operation and whether it is safe to dispatch.

Consider an order workflow that reads stock, reserves an item, charges the customer, and requests fulfillment. Assume fulfillment becomes noncancellable once accepted. This example spans independently committed operations, with no transaction covering the whole workflow.

Why the loop retries

Separate three triggers rather than translating every disappointing result into another mutation:

• Timeout or transport loss. The caller lacks a usable response. Unless rejection before execution is established, the outcome may be unknown.
• Malformed or misunderstood output. The effect may have committed, but response parsing or interpretation failed. Prefer repairing or rereading the result over repeating the effect.
• Model judgment. A model may consider a valid result inadequate and request another call. That request is not evidence that the previous operation failed.

Each framework and application can handle these triggers differently. Client-tool interfaces can leave execution with application code while the model selects calls and consumes results. That boundary is where retry policy can be enforced.

Distinguish these triggers from known rejection. Throttling may justify waiting and retrying under the service's contract; unchanged validation failures usually require correction, not repetition. An orchestrator restart is another reason to recover an existing operation rather than create a replacement. Backoff controls retry timing and load. It does not establish whether a mutation already happened.

Idempotency must reach the service that owns the effect

Sending email, charging a card, and filing a ticket can create additional effects when repeated. Idempotency makes repeated attempts at one logical operation produce no additional intended effect. It does not guarantee identical response bytes. This is established distributed-systems practice that also applies to agent tools.

For the order workflow, the desired guarantee is one charge for the approved payment operation, regardless of how many attempts reach the payment tool. A durable orchestration ledger can record that intention and its observed outcome. That record alone cannot prevent the payment service from charging twice.

The service that owns the effect must enforce the guarantee. Where a mutation and its deduplication record share a transactional store, commit them atomically. Otherwise, a crash between changing state and recording completion leaves a gap. AWS's idempotency guidance explicitly identifies this atomicity requirement.

The implication for an external provider is important: a local transaction does not include that provider's commit. Carry the operation's identity through to a supported downstream idempotency contract, or leave unresolved outcomes for reconciliation. A wrapper that remembers successful responses cannot by itself close that gap.

Generate keys from durable execution steps

For agent orchestration, apply the caller-generated identity contract outside model output. The orchestration layer must generate the idempotency key, derived from the durable execution step, before dispatch. The orchestrator must persist its association with the logical operation and approved arguments and reuse it across transport retries, model-requested retries, and recovery after a restart.

Do not ask the model to generate this key. A newly sampled value can identify the retry as a different operation, defeating deduplication. Likewise, a fresh tool-call identifier is not automatically a stable business-operation identifier. Attach the key outside model-controlled arguments.

Here, step means a durable operation occurrence with an identity independent of its position in a newly generated plan. The charge for this order needs one identity; a separately approved second purchase needs another. Identical arguments alone do not establish identical intent. AWS documents this distinction using callers that legitimately want two identical resources.

Changed arguments need separate handling. If the model changes the amount while the original charge is unresolved, do not silently create a new key and dispatch. Both amounts could otherwise be charged for the same order. Resolve the original operation, then evaluate whether the changed request is authorized.

At the tool boundary, workers that can receive attempts for the same operation need coordinated deduplication state. A process-local cache is insufficient if another worker or a restarted process can receive the retry. Coordination must cover in-progress attempts as well as completed results so two workers cannot both decide they are first.

Check retention and payload-conflict rules. Stripe documents that keys can be pruned after they are at least 24 hours old, reused pruned keys create new requests, and mismatched parameters produce errors. A workflow resuming later cannot assume its old key still prevents duplication.

Classify tools by replay safety and reversibility

Use four categories to choose a default policy, then document exceptions in each tool's contract. Read-only and idempotent describe execution behavior; compensatable and irreversible describe recovery options. These are independent properties, not four mutually exclusive types.

Tool category	Order example	Retry policy
Read-only, repeatable	Read stock	Retry within limits; reassess freshness.
Idempotent write	Ensure one reservation exists for this order	Retry while the tool's idempotency contract holds.
Non-idempotent side effect, compensatable	Charge, with a separate refund operation	Require deduplication or reconcile before reissuing.
Irreversible action	Request noncancellable fulfillment	Require human confirmation plus duplicate prevention; reconcile unknown outcomes.

These classifications are assumptions about the example's tools, not guarantees conferred by their names. A stock read can return a different answer on the next attempt. Replaying a reservation must not decrement inventory again, and a nominally idempotent update must not overwrite an intervening change without the intended concurrency checks.

An irreversible action can still have an idempotent interface. Confirmation authorizes an operation; deduplication prevents additional execution; reconciliation establishes its outcome; compensation addresses completed effects where compensation is possible. Use them together where their risks overlap.

Resolve ambiguous success before taking another action

Suppose the charge commits but its response is lost. The payment record should mark the outcome as unknown. If the provider's contract permits safe replay, the orchestrator can retry the same operation with the same key. Otherwise, it needs to query authoritative status using the operation's durable reference. An empty or stale lookup is not proof that an in-flight charge will never commit.

For this workflow, automatic retries favor progress when duplicate suppression covers the actual effect and the full retry window. Stopping for reconciliation favors safety when those guarantees are missing, but delays completion and requires an operational owner. In the order workflow, that can mean delaying fulfillment while the payment is reconciled. The provider's contract determines which path is safe; model confidence cannot establish whether the effect occurred.

Do not issue a refund merely because the charge timed out. Payment must first be confirmed. If the order must then be cancelled, a refund is a new operation with its own failure modes and idempotency identity. It compensates for a completed charge; it does not erase the original transaction.

For the irreversible fulfillment step, require human confirmation before dispatch. Show the exact order, items, quantities, destination, and noncancellable consequence. Approval must be bound to those immutable parameters and the logical operation. If material parameters change, require new approval. Enforce the gate in code rather than accepting a model's claim that approval was obtained.

Approval does not prove that an earlier attempt failed. Once the approved fulfillment request has an unknown outcome, preserve its identity and reconcile it; another confirmation must not become permission to duplicate it. Human review addresses intent and consequences, while the tool boundary remains responsible for replay safety.

Recover partial trajectories without assuming rollback

If six steps complete and step seven fails permanently, the preceding effects remain committed. Ending the agent run or replacing its plan does not roll them back. In a workflow spanning independent services, recovery requires deciding what to retain, what to compensate, and what remains unresolved.

Durable per-operation records need to include approved inputs, keys, status, and external references, distinguishing completed, rejected, and unknown outcomes. After a restart, those records allow the orchestrator to resume or reconcile existing operations instead of replaying the trajectory from its first mutation. Compensation progress needs its own record so recovery itself can resume.

In the order example, confirmed payment followed by confirmed fulfillment rejection could justify refunding the charge and releasing the reservation, according to the order's rules. An unknown fulfillment outcome should instead block an automatic refund or stock release until dispatch status is resolved. Otherwise, recovery could mark inventory available while shipment is already underway.

If fulfillment succeeded and a later administrative step failed, retain the fulfillment result and repair the remaining step. Do not repeat the order merely to obtain a cleaner execution history. Compensation can also fail, so preserve its outstanding work and hand unresolved cases to an operator with the relevant operation references.

When duplicate suppression or authoritative outcome lookup cannot be established, keep automatic mutation retries disabled for that tool. Require reconciliation and an explicit recovery decision instead of asking the model to try again.

Caller-generated idempotency keys and compensating actions are established distributed-systems practices that also apply to model-triggered retries. Use the four categories to classify your tool inventory and decide which calls may be reissued unattended. For related practice from Formation, take the sub-challenge and read the Aug 19 split-brain post.