Building Idempotency Into AI Workflows From Day One

A reliable automation is one you can run twice by accident and nothing bad happens. That's it. That's the property. Most teams discover they don't have it the hard way: a Zapier retry sends the same invoice three times, an AI agent re-emails the same lead, a webhook fires twice and your CRM ends up with duplicate contacts that took six hours to clean up.

Idempotency is the property that running an operation N times produces the same result as running it once. It sounds academic. In practice, it's the difference between an automation you trust and one you babysit. If you're building anything with AI agents, webhooks, queues, or retries — and you are — this needs to be designed in, not bolted on after the second outage.

Why retries are not optional

Every serious automation platform retries failed steps. n8n retries. Make retries. Zapier retries. OpenAI's API client libraries retry. Your webhook provider retries when your endpoint times out. AWS Lambda retries on failure. This is not a bug — partial failures are the norm in distributed systems, and retries are how you recover.

The consequence: your workflow will execute the same step more than once. Sometimes twice. Occasionally five times. If the step is "send a payment," "create a Stripe charge," "post a Slack message," or "email the customer," you have a problem unless you've designed for it.

The naive fix is to disable retries. Don't do this. You'll trade duplicate sends for silent failures, which is worse — at least duplicates are visible. The right fix is to make every side-effecting step safe to repeat.

The deduplication key pattern

The core technique: before doing anything that has an external effect, compute a stable key that uniquely identifies this specific operation, and check whether you've already done it.

A stable key is deterministic. Given the same input, you always get the same key. Bad keys: timestamps, random UUIDs generated inside the workflow, anything based on now(). Good keys: the source event ID, a hash of the input payload, a business identifier like invoiceid + actiontype.

Concrete example. You have a workflow that takes new Typeform submissions, runs them through an AI classifier, and creates a HubSpot contact. The dedup key for the HubSpot step might be typeformresponseid. Before creating the contact, you check: has a contact been created with this typeformresponseid in a custom property? If yes, skip. If no, create it and write the ID into that property atomically.

For steps where you control the destination system (Postgres, your own API), use a unique constraint on the dedup key column. The database rejects duplicates for you. For steps where you don't (Stripe, HubSpot, Slack), most major APIs support an Idempotency-Key header — Stripe especially. Pass your dedup key there and the provider handles it.

The "did this already happen" check

For systems without idempotency key support, you need a lookup step before the action step. The pattern is:

Compute dedup key from the trigger payload

Query the destination: does a record with this key already exist?

If yes, exit cleanly (log it, but don't error)

If no, perform the action and write the key into the destination

This only works if step 4 is atomic — meaning the action and the key-write happen together, or not at all. If you create the Slack message first and then try to record that you did it, a crash between the two leaves you with a sent message and no record. Next retry: duplicate.

When the destination doesn't support atomic writes, use a separate ledger. A simple Postgres table with (workflowname, dedupkey, status, createdat) and a unique constraint on (workflowname, dedupkey) works. Before the action, insert the row with status pending. After the action succeeds, update to done. On retry, the insert fails because the key exists, and you check the status — if done, skip; if pending, the previous attempt crashed mid-action and you need a recovery decision.

Where this matters most

Money movement is the obvious one. Charging cards twice destroys customer trust faster than almost anything else. Use Stripe's Idempotency-Key header, always, with no exceptions.

Email and notifications are the underrated one. An AI workflow that emails leads is one network blip away from sending the same nurture sequence three times. We've seen a client whose RAG-powered support bot retried on a 504 and sent the same apology email to a customer four times in a row. The customer was not impressed.

AI generation calls deserve a mention too. They're expensive and non-deterministic, so retrying them is doubly bad: you pay twice and get two different outputs. Cache LLM responses by a hash of the prompt for the duration of a workflow run. Most orchestration tools support this with one config line.

Third-party CRM writes are where most teams get burned. Duplicate contacts, duplicate deals, duplicate tasks assigned to sales reps who now have to figure out which one is real. Use external IDs aggressively — every modern CRM supports them.

Common pitfalls

Using timestamps as dedup keys. A retry one second later has a different timestamp and bypasses your check. Use the source event's ID, not the time you processed it.

Dedup keys that are too narrow. If your key is just customerid, you'll dedup operations that should happen twice — like two separate orders from the same customer. Include the action and the specific event: customerid + orderid + action.

Dedup keys that are too broad. Hashing the entire payload sounds safe until you realize one whitespace difference produces a different hash. Normalize first, or use stable business identifiers.

Forgetting the AI step is non-deterministic. If your workflow uses an LLM to extract fields and then writes those fields to a CRM, two runs produce two slightly different field values, and your downstream dedup check based on those fields fails. Dedup on the input to the AI step, not the output.

No expiry on dedup records. Your ledger table grows forever. Set a TTL based on how long retries can realistically happen — 7 to 30 days is usually enough.

A practical checklist

For every new workflow, before you ship it, answer these:

• What is the stable dedup key for each side-effecting step?
• Does the destination system support idempotency keys natively? If yes, use them.
• If no, where is the ledger that records what's been done?
• What happens if the workflow crashes between the action and the ledger write?
• Is every retry safe, or does it produce a new side effect?
• Are AI generation calls cached within a single workflow run?

If you can't answer all six for a workflow that touches money, customers, or external systems, it's not production-ready. Fix that before launch, not after the first incident.

Idempotency is unglamorous. No one demos it. No one writes a LinkedIn post about the duplicate Stripe charge they prevented. But it's the property that separates automations you can leave alone from ones that need a human watching them. Build it in from day one and you'll save yourself the rewrite.

If you're standing up AI workflows that touch customers or money and want them designed to survive retries, partial failures, and the messy reality of production, see how we build them.