Webhook Deduplication vs Idempotency
Deduplication and idempotency both address duplicate events but operate at different levels and serve different purposes.
Deduplication (System-Level)
Deduplication detects and prevents duplicate events at the system level. The ingester checks incoming events against a deduplication store using a configurable key (event ID, correlation ID). If a match is found within the time window, the duplicate is dropped.
| Property | Deduplication |
|---|---|
| Where | Ingester (system level) |
| When | Before delivery |
| Mechanism | Key-based lookup in deduplication store |
| Scope | Cross-session, within time window |
| Persistence | Currently in-memory; persistent store planned |
Idempotency (Consumer-Side)
Idempotency enables consumers to safely process the same event multiple times. Events carry an idempotency key that consumers can use to check whether they have already processed the event. This is essential in at-least-once delivery systems.
| Property | Idempotency |
|---|---|
| Where | Consumer (application level) |
| When | During processing |
| Mechanism | Idempotency key tracking by consumer |
| Scope | Per-consumer |
| Persistence | Consumer-managed |
How They Work Together
Event arrives → System dedup check → Delivery → Consumer idempotency check → Processing
- The system checks for duplicates to avoid redundant delivery
- The consumer checks for idempotency to handle any duplicates that do arrive
- Together, they minimize duplicate processing while allowing for recovery and replay