# Checkpoint is not idempotency: Serverless Under Constraints, Study 1

> An approved R$ 100.00 refund, a provider that writes the transaction in 145 ms, and a caller that gives up after 3 seconds without seeing the response. In the first study of the Serverless Under Constraints series, I put Lambda with SQS and Lambda Durable Functions in front of the same ambiguous outcome, in a real AWS account and with an independent oracle: both variants refunded R$ 200.00, and both ended up reporting success.

- Author: Raphael Moura
- Published: 2026-10-19
- Canonical HTML: https://raphamoura.dev/en/blog/serverless-under-constraints-estudo-1/
- Markdown (PT): https://raphamoura.dev/blog/serverless-under-constraints-estudo-1/index.md


An approved R$ 100.00 refund. The caller calls the provider, the provider writes the transaction to the ledger 145 milliseconds later, and the response never arrives. After 3 seconds without an answer, the caller gives up, records the timeout and hands the problem to the retry mechanism. A minute later, the retry calls the provider again, the provider writes again, and the customer receives R$ 200.00. The execution ends in `ExecutionSucceeded`, the message leaves the queue without touching the DLQ, and no alarm has any reason to fire.

I ran this scenario in an AWS sandbox account in `us-east-1` on October 7, with the evidence frozen and [published](https://github.com/w00fx/serverless-under-constraints). It is the first study in the **Serverless Under Constraints** series, and it was born from a sentence of mine: in the [article about Lambda Durable Functions](/en/blog/aws-lambda-durable-functions/), I wrote that anything that must not repeat has to live inside a step. Being inside a step is necessary, and the text implied it was enough. The study shows what that difference costs.

## The ambiguous outcome

Study 1 is about the **ambiguous outcome**: the caller dispatched the call and was left without an authoritative response. From where it stands, "the provider never received it" and "the provider executed it and the response got lost" are indistinguishable, and the timeout only says it stopped waiting. Brandur Leach described exactly this situation in 2017, on [Stripe's engineering blog](https://stripe.com/blog/idempotency), and it is the reason Stripe's API accepts an idempotency key.

Anyone who has sent a Pix, Brazil's instant payment, with the banking app frozen on "processing" knows the feeling. The sensible person opens the statement before sending it again. An automatic retry gets an error and repeats the operation, because that is what it was configured to do, with no statement to check.

And in serverless, retry lives in every layer: in the SDK, which [retries by default, including on client-side timeouts](https://docs.aws.amazon.com/lambda/latest/dg/invocation-sync.html), in the event source mapping, which returns the batch to the queue when the function fails, in Step Functions' `Retry`, and in the retry strategy of every Durable step. Every layer that repeats a call with an external effect can duplicate that effect. On a read, that costs milliseconds; on a refund, it costs money someone will have to go and claw back.

The study's question: when the provider writes the refund before the caller hits its deadline, does each execution strategy preserve the rule "one approved refund produces exactly one monetary effect, within the captured amount"? With a provider that has no idempotency key, the double refund was predictable, and the hypothesis, written into the specification before the run, said so. What the study tests is whether any caller-side mechanism (checkpointing, a FIFO queue, a delayed retry) changes that outcome, and whether the system notices the damage.

## Serverless Under Constraints

To answer questions like this one with evidence, I built an open lab. The series tests serverless architecture strategies under failures, traffic spikes, cost limits, concurrency and consistency requirements. Each study puts architecture variants in front of the same business scenario and the same controlled treatment, on real resources in an AWS sandbox account; emulators help with development and stay out of the final evidence. An independent oracle reads the frozen evidence and returns `pass`, `fail` or `indeterminate` for a business invariant, such as money, orders or inventory, and uptime and error rates only count as diagnostics. Protocol, code, redacted evidence and limitations are public, so anyone can repeat the experiment and challenge the conclusion.

## The design of Study 1

Both variants consume their own SQS FIFO queue, with batch size one, `maxReceiveCount` set to two and a FIFO DLQ, and they call the same provider synchronously, through Lambda Invoke, via a shared provider client: a 3-second application deadline and SDK retries turned off (`maxAttempts: 1`), so the only retry in play is the architecture's. What changes is who owns the retry:

- **Conventional (Lambda + SQS):** on timeout, the function rethrows, the invocation fails, and SQS delivers the message again when the 60-second visibility timeout expires.
- **Durable (Lambda Durable Functions):** the function has a single step, `refund-attempt`, with an explicit retry strategy of two attempts, a 60-second delay and no jitter. The second attempt happens inside the same durable execution. This variant's queue uses a 360-second visibility timeout, above the 300-second execution timeout, because the [event source mapping](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html) only treats the message as processed when the whole execution finishes.

The Durable handler, simplified from the study's code:

```ts
import { StepSemantics, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(async (event, context) => {
  const delivery = parseDelivery(event);
  return context.step(
    'refund-attempt',
    (step) => caller.runStepAttempt(delivery, step.attempt),
    {
      retryStrategy: (_error, attemptsMade) =>
        attemptsMade < 2
          ? { shouldRetry: true, delay: { seconds: 60 } }
          : { shouldRetry: false },
      semantics: StepSemantics.AtLeastOncePerRetry,
    },
  );
});
```

Notice the retry strategy's signature: an `Error` and the number of attempts already made. That is all the information the mechanism has to decide whether to repeat.

In the `CONTROL` scenario, the provider writes and responds right away. In `COMMIT_THEN_TIMEOUT`, the provider atomically writes the transaction to the ledger and holds the response behind a barrier. The caller hits its 3 seconds and records the timeout in its own journal; an independent controller reads that record through the DynamoDB Stream, validates it, and only then signals the provider that it can release the response. The release depends on the recorded timeout, in a causal chain the oracle verifies, and the "commit before timeout" order is imposed by the design and checked by wall clock. Before any trial, a transport probe, an isolated execution with a single call, had to prove in the real cloud that this mechanism works.

The provider was designed without an idempotency key, without deduplication and without a call cap. That is the choice that defines the study: it exposes the architecture's retry on purpose.

The verdict comes from an **independent oracle**, which reads a strongly consistent snapshot of the provider's ledger and applies three monetary rules: exactly one successful transaction per refund request, a refunded total within the captured amount, and a transaction set equal to the authorized effect, with the right amount, currency and identities.

Every verdict-changing rule had to be implemented in the oracle before the first cloud evidence, because an oracle adjusted after seeing the data writes the conclusion instead of measuring it. The protocol also forbids altering or excluding a trial that contradicts the hypothesis. Evidence is frozen in packages identified by the SHA-256 of their index, and every published value cites the artifact and digest it was read from.

## Two variants, the same refund twice

The canonical run executed four trials in sequence. All four were valid, cleanup finished, the leak audit (the sweep that looks for study resources left behind in the account) came back clean, and the comparison between variants was eligible, with equal inputs and treatment for both and intact evidence:

| Trial | Scenario | Variant | Ledger transactions | Refunded | Verdict |
|---|---|---|---|---|---|
| 1 | `CONTROL` | conventional | 1 | R$ 100.00 | `pass` |
| 2 | `CONTROL` | durable | 1 | R$ 100.00 | `pass` |
| 3 | `COMMIT_THEN_TIMEOUT` | conventional | 2 | R$ 200.00 | `fail` |
| 4 | `COMMIT_THEN_TIMEOUT` | durable | 2 | R$ 200.00 | `fail` |

In the controls, the provider call finished in 158 and 238 ms, with at least 2.7 seconds of headroom before the deadline. In the treatments, both variants failed all three monetary rules, and each variant's validations (two-trial executions run before the canonical run to prove the implementation) reproduced the same pair of results.

**How much the ledger refunded, second by second.** Cumulative successful transactions in the provider's ledger, for each trial of the canonical run; treatment is the `COMMIT_THEN_TIMEOUT` scenario. The shaded area sits above the approved amount, and the vertical line marks the caller's timeout: in the treatments, the first commit happens before the timeout, and the retry takes the total to R$ 200.00.

Trial 4, the Durable treatment, tells the whole story in eight lines. I joined events from three independent sources: the caller's journal, the provider's journal and the durable execution history that AWS itself exposes (UTC times):

```text
06:16:50.248  caller    dispatch_started            attempt 1
06:16:50.393  provider  provider_commit_confirmed   R$ 100.00 in the ledger
06:16:53.277  caller    caller_timeout_recorded     ~3 s without a response
06:16:53.384  durable   StepFailed                  attempt 1, next in 60 s
06:17:53.509  durable   StepStarted                 refund-attempt
06:17:54.489  provider  provider_commit_confirmed   R$ 100.00 in the ledger
06:17:54.563  durable   StepSucceeded               attempt 2
06:17:54.593  durable   ExecutionSucceeded
```

No component broke its own contract, and the customer got paid twice.

The conventional variant reached the same place by another path. The provider confirmed the commit 212 ms after dispatch, the caller hit the deadline, the invocation failed, and SQS delivered the same message again about 56 seconds after the timeout (the visibility timeout counts from receipt). The second delivery called the provider, which wrote again. Counterintuitive as it sounds, the queue was FIFO, and FIFO's ["exactly-once processing"](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) did not help at all. Its deduplication stops a repeated send from putting a second copy of the message in the queue; here there was a single message, delivered twice because nobody deleted it after the first time.

## Success on the status, double on the statement

In both treatment trials, the final state the system reports is success: the request `FINISHED` with reason `SUCCEEDED`, an empty DLQ in every observation, the durable execution in `ExecutionSucceeded`. In the conventional variant, the only operational trace is one failed invocation that the retry "healed" less than a minute later, and a transient error that resolves itself is exactly what most alarms treat as noise. Only the study's instrumented journal records that knowledge about the effect ended as `UNKNOWN`.

**Every attempt against the 3-second deadline.** Time from dispatch to the end of each attempt, in the canonical run and the validations. The controls finish with room to spare, the first attempt under treatment stops at the deadline, and the retry comes back successful in under a second.

The chart shows what the caller went through: one attempt that hit the deadline and a retry that came back successful in under a second. As far as the caller knows, the retry worked. The oracle says `fail` because it looks at the ledger, which holds two R$ 100.00 transactions for an approved R$ 100.00 refund, against a captured payment of R$ 100.00. The study derives a metric for this contradiction, `correct_completion`, which is true only when the verdict is `pass` and processing finished successfully. In both treatments it is `false`, with processing reporting `SUCCEEDED`.

To me, this is the lesson that holds even for people who will never use Durable Functions. The telemetry of the system under test records what the system believes it did, and that is why it cannot serve as the oracle for an external effect. If you move money, the check that matters in production is reconciliation against the ledger of whoever actually moved the money. It's the Pix frozen on "processing" again: what tells you how many times the money left is the statement.

## Why the checkpoint did not save it

A checkpoint stores what the function knows, and the ambiguous outcome is precisely what it does not know. When a step finishes, its result goes into the checkpoint log and the step never runs again. When the call inside the step hits its deadline, there is no result to store, and the retry strategy decides with nothing but the error it received, as the signature above shows.

The step's semantics flag does not change that. `AtLeastOncePerRetry`, the default, and `AtMostOncePerRetry` deal with what happens when the function is interrupted mid-attempt and replay reaches that step; the decision to try again after an error still belongs to the retry strategy. AWS itself is explicit: the [Lambda idempotency page](https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-idempotency.html) warns that steps have at-least-once semantics by default, and the [best practices page](https://docs.aws.amazon.com/durable-execution/patterns/best-practices/idempotency/) adds that both semantics apply per attempt, that neither guarantees a single execution across the whole workflow, and that a retry strategy that retries on failure runs the step again even under at-most-once. The problem is the mental model, one my earlier article helped spread, that putting the call inside a step is the protection. The step guarantees that what finished will not repeat; what did not finish is on you.

Reading the SDK with this result in hand, what stands out is what happens when nobody configures anything: without an explicit retry strategy, the TypeScript SDK falls back to the [default preset](https://github.com/aws/aws-durable-execution-sdk-js/blob/main/packages/aws-durable-execution-sdk-js/src/utils/retry/retry-presets/retry-presets.ts), with up to six attempts, exponential backoff and jitter. A step that calls a payment provider with no declared retry strategy can call that provider six times. The layers also multiply: the same idempotency page states that event source mappings offer no idempotency for durable executions, so every new SQS delivery starts a new execution, with none of the previous one's checkpoints. On the study's configured path, two attempts per execution times two queue deliveries gives up to four provider calls on Durable, against two on the conventional variant, and at-least-once delivery can still add more. In trial 4, the second attempt succeeded and the queue never had to deliver again.

Durable has real gains and real costs. The second attempt happened inside the same execution, with no new queue delivery, and the history shows `StepFailed` followed by `StepSucceeded` with the exact delay configured. The retry became code, with a name, a delay and an attempt count, and it shows up in code review and in the execution history. In exchange, the message stays in flight for the entire execution, the visibility timeout has to be sized for the execution, and, when fed by an event source mapping, the whole execution is capped at the function's maximum timeout: 15 minutes in on-demand mode, 90 on Lambda Managed Instances. The long wait, which is Durable's big appeal, shrinks when the input is a queue.

The study does not pick a winner, and I won't pretend it does. On a single axis, how readable the retry is, I prefer the one that shows up in the execution history over the one that happens because a visibility timeout expired.

## Where I would put the defense

The study deliberately excluded provider-side idempotency and reconciliation. What follows is my position, backed by the mechanism the study exposed and by what the payments industry already does, and not yet measured by the series.

The defense has to sit where the effect happens. One non-monetary rule passed in both treatments: both attempts carried the same `refund_request_id`, `ref-poc-001`. The logical identity was there; what was missing was someone on the other side of the call using it. In [Stripe's API](https://docs.stripe.com/api/idempotent_requests), a retry with the same `Idempotency-Key` gets the saved response of the first execution, even when it was a 500 error. The [Central Bank of Brazil's Pix API](https://github.com/bacen/pix-api/blob/2.10.0/openapi.yaml) models a refund as `PUT /pix/{e2eid}/devolucao/{id}`, with the `id` generated by the client: an `id` already used for that Pix is rejected, and a `GET` on the same path queries the result. Key and reconciliation in the same contract. It is also the design AWS describes in the [Builders' Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/), with a caller-provided request identifier, and that EC2 implements with `ClientToken`. I strongly recommend taking this to design review as a provider selection criterion: if the API that moves money does not accept an idempotency key, no retry configuration on your side eliminates the risk.

Idempotency in your handler does not solve this case. The [Powertools for AWS Lambda idempotency utility](https://docs.aws.amazon.com/powertools/typescript/latest/features/idempotency/) protects against the same input being processed twice, and its documentation warns that an unhandled exception deletes the idempotency record, so the next invocation with the same payload runs again. That is exactly trial 3's path: the application timeout became an error, the invocation failed, and the second delivery would have been free to call the provider. Deduplicating the input on your side leaves the effect on the other side duplicated all the same.

And the unknown has to become domain state. In the study, knowledge about the effect becomes `UNKNOWN` at the first timeout and never leaves it; a later success only proves the later effect. In a money flow, a timeout after dispatch should open reconciliation, querying the provider by your reference before any new attempt. In the example's retry strategy, that becomes a condition on the error: a timeout after dispatch returns `shouldRetry: false`, and the flow moves on to reconciliation. Stripe itself, which has idempotency keys, [advises](https://docs.stripe.com/error-low-level#network-errors) stopping retries and reconciling when the retry window ends without a result. Turning retry off without reconciling trades the risk of a double refund for the risk of a refund that never happens. Querying before repeating also has a race window, because the first call may still be in flight at the provider, which is why reconciliation complements the idempotency key; on its own, it leaves that window open.

## Limits and how to challenge this

One trial per cell, a fixed order and a controlled provider are enough to validate protocol and mechanism. How often this window opens in production, I have no data to say. The provider reproduces the causal and monetary boundary that matters, and a real payment processor has a lot more behavior. The "commit before timeout" order is checked by wall clock across different Lambda functions, under a clock-alignment assumption declared in the protocol, without any formal AWS guarantee.

Provenance has a limit too, and it is written in the close-out: no third-party timestamp predates the evidence. The repository had no CI, and the oracle commits only reached GitHub after the evidence was frozen. The "oracle before evidence" order rests on commit dates, on records frozen inside the packages and on replays of the old oracle over those packages, and the one attesting to that is me. Billed cost was left `unverified`: the pre-billing estimate was US$ 0.28 for the run, against a declared ceiling of US$ 5.00, and the sandbox account had no billing export to confirm it.

The path also left two AWS gotchas on record. When you delete a DynamoDB table with a stream enabled, the stream goes to `DISABLED` and is only deleted [24 hours later](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteTable.html), with no call that deletes it sooner; meanwhile, it remains readable and tagged. The rehearsal probe's leak audit could not prove who owned that stream, closed as `inconclusive`, and closure needed a recovery amendment. In addition, `ListDurableExecutionsByFunction` refused an alias as qualifier, with the message "Cannot filter by alias"; the [current API reference](https://docs.aws.amazon.com/lambda/latest/api/API_ListDurableExecutionsByFunction.html) describes the parameter as a version, while the [AWS CLI v1 reference](https://docs.aws.amazon.com/cli/v1/reference/lambda/list-durable-executions-by-function.html) still says "version or alias". The first Durable validation tried to list executions by alias, could not prove they had finished, and its trials froze as `indeterminate`. The validation was excluded by the reproduction criterion and remains listed in the close-out, with the reason. An execution that goes wrong is data too.

Everything is in the [Serverless Under Constraints repository](https://github.com/w00fx/serverless-under-constraints): specification, variant and oracle code, a `results.json` in which every value cites the artifact and SHA-256 it was read from, and a redacted copy of the evidence packages, in which only the account id and local paths were replaced. The first check needs only `tar`, `jq` and `shasum`, and no AWS account:

```sh
cd public/study-1-evidence
tar -xzf study-1-evidence.tar.gz
jq -r '.files[] | "\(.redacted_sha256)  \(.path)"' redaction-manifest.json \
  | shasum -a 256 -c --quiet -
```

The command checks every file in the copy against the redaction manifest. The [README](https://github.com/w00fx/serverless-under-constraints#re-verify-study-1-from-bytes) covers the rest (each package's identity against `results.json`, the files changed by redaction, the verification records), and the [study close-out](https://github.com/w00fx/serverless-under-constraints/blob/main/STUDY-1.md) separates what a reader can prove from the public bytes from what is only attested.

## What we learned

Putting result, mechanism and limits together, one change comes out of this for how to design and another for how to run the series. For those who design, the rules fit in a short list:

1. **A timeout after dispatch is `UNKNOWN`.** It is domain state, with an exit to reconciliation and never to a blind retry.
2. **Automatic retry of an irreversible effect only with an idempotency key at the provider**, derived from the business identity and the same across every layer.
3. **Retry strategy always explicit**, with the layers counted and multiplied: SDK, step and queue add up attempts, and no default goes in without review.
4. **The monitor that matters compares request with ledger** and alarms on divergence. The pair the study uses, preservation verdict and correct completion, works as a production metric; success status stays a diagnostic.

For the lab, the limits listed above turn into method lessons, and some of them hurt. The "oracle before evidence" order ended up attested instead of proven, and the fix is simple: publish the oracle commit with a tag before the first cloud execution, so GitHub serves as a witness. Six specification items were ratified about fifteen hours after the run, describing what the code already did, and ratification has to come first. Cost allocation tags and the billing export need to be active before deploying, so the run's bill can be attributed when it arrives, up to a day late. Staged qualification, with a transport probe, per-variant validation and only then the canonical run, paid off: the rehearsal probe caught the residual DynamoDB stream and the first validation caught the alias problem, both before the run that counts.

The questions left open are the natural candidates for the next studies. What changes when the provider accepts an idempotency key, both Stripe-style, replaying the saved response, and Pix-style, rejecting the repeated `id` and forcing the caller to query the result? Does reconciling before repeating, without a key, close the race window or only narrow it? Does Durable, after exhausting its attempts and coming back through the queue, really reach the configured path's four calls? And with realistic latency, without a forced treatment, how often does the commit land before the timeout? That last one requires repetition and statistical design, which one trial per cell cannot support.

## Closing

Under `COMMIT_THEN_TIMEOUT`, Lambda with SQS and Lambda Durable Functions refunded R$ 200.00 for an approved R$ 100.00 refund, and both ended up reporting success. The retry decided when the second effect would happen, with no way of knowing whether it should. The platform's guarantees (at-least-once, checkpointing, FIFO) operate on messages and executions, the business invariant operates on effects, and that gap between the two is where the series will keep looking.

Take a flow in your system that moves money, inventory or any irreversible external effect and answer in writing: if the call to the provider times out after the commit, what stops the retry from duplicating it? Look for the answer in the provider's API contract before looking in your handler. If the contract accepts neither an idempotency key nor a query by reference, you just found the risk no dashboard will show you. And if you want to challenge the result, clone the repository and run the verification from the bytes.

## References

- [Serverless Under Constraints, study repository](https://github.com/w00fx/serverless-under-constraints)
- [Brandur Leach, Designing robust and predictable APIs with idempotency (Stripe, 2017)](https://stripe.com/blog/idempotency)
- [AWS Lambda, Invoking a Lambda function synchronously](https://docs.aws.amazon.com/lambda/latest/dg/invocation-sync.html)
- [AWS Lambda, Invoking durable functions with event source mappings](https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html)
- [Amazon SQS, Exactly-once processing in FIFO queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html)
- [AWS Lambda, Idempotency in durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-idempotency.html)
- [AWS Durable Execution, Idempotency best practices](https://docs.aws.amazon.com/durable-execution/patterns/best-practices/idempotency/)
- [AWS Durable Execution SDK for JavaScript, retry presets](https://github.com/aws/aws-durable-execution-sdk-js/blob/main/packages/aws-durable-execution-sdk-js/src/utils/retry/retry-presets/retry-presets.ts)
- [Stripe, Idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [Central Bank of Brazil, Pix API (OpenAPI 2.10.0)](https://github.com/bacen/pix-api/blob/2.10.0/openapi.yaml)
- [AWS Builders' Library, Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
- [Powertools for AWS Lambda (TypeScript), Idempotency](https://docs.aws.amazon.com/powertools/typescript/latest/features/idempotency/)
- [Stripe, Low-level error handling: network errors](https://docs.stripe.com/error-low-level#network-errors)
- [Amazon DynamoDB, DeleteTable](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteTable.html)
- [AWS Lambda, ListDurableExecutionsByFunction](https://docs.aws.amazon.com/lambda/latest/api/API_ListDurableExecutionsByFunction.html)
- [AWS CLI v1, list-durable-executions-by-function](https://docs.aws.amazon.com/cli/v1/reference/lambda/list-durable-executions-by-function.html)
- [Serverless Under Constraints, Re-verify Study 1 from bytes](https://github.com/w00fx/serverless-under-constraints#re-verify-study-1-from-bytes)
- [Serverless Under Constraints, Study 1 close-out](https://github.com/w00fx/serverless-under-constraints/blob/main/STUDY-1.md)