Building an execution engine that can fail safely

Retry is the most common recovery strategy and also the easiest way to create duplicate side effects.

Lease-based ownership, an event log, and idempotency keys turn failure into a state problem instead of a hope problem.

If you cannot answer who owns a job and whether a side effect already happened, you do not have recovery. You have roulette.