Events, Ingress & Notifications

Section 1

The event spine in one view

Ummah never asks Adyen "what happened?" — Adyen tells it. A dedicated ingress service (ummah-platform-webhook) receives every inbound webhook, verifies its HMAC signature, persists it as a row with a unique dedupe key, and enqueues a BullMQ job for the worker fleet (ummah-platform-worker) to process. The same persist-then-enqueue shape serves Sumsub KYC events. On the outbound side, the backend renders every operational email into a Notification row and hands only its id to SQS for the notification service to deliver.

%%{init:{"theme":"base","themeVariables":{"fontFamily":"Bricolage Grotesque, Inter, Segoe UI, sans-serif","fontSize":"13px","primaryColor":"#FBE5D6","primaryTextColor":"#00273A","primaryBorderColor":"#E86C2B","lineColor":"#8195A1","actorBkg":"#FFFFFF","actorBorder":"#D2D8DE","actorTextColor":"#00273A","actorLineColor":"#D2D8DE","signalColor":"#33505F","signalTextColor":"#33505F","activationBkgColor":"#FBE5D6","activationBorderColor":"#E86C2B","noteBkgColor":"#FEF3DC","noteBorderColor":"#E8D5A8","noteTextColor":"#00273A","labelBoxBkgColor":"#F1F4F8","labelBoxBorderColor":"#D2D8DE","loopTextColor":"#33505F"},"sequence":{"mirrorActors":false,"actorMargin":46,"messageMargin":30,"boxMargin":8}}}%%
sequenceDiagram
  participant AD as Adyen
  participant WH as Webhook service
  participant PG as Postgres
  participant MQ as Redis · BullMQ
  participant WK as Worker dispatcher

  AD->>WH: POST /webhooks/adyen
  WH->>WH: HMAC verify — HmacSignature or per-item
  WH->>PG: persist AdyenWebhookEvent + dedupeKey
  alt row already has processedAt
    WH-->>AD: ack — redelivery dropped
  else fresh event
    WH->>MQ: enqueue webhook-in.adyen, jobId = sha256(dedupeKey)
    WH-->>AD: ack
    MQ->>WK: job {eventId, format}
    WK->>PG: re-check processedAt
    WK->>WK: dispatch / dispatchBalancePlatform
    WK->>PG: stamp processedAt
    Note over MQ,WK: throw → BullMQ exponential backoff, 3 attempts
  end

ummah-platform-webhook/src/webhooks/ingress.service.ts:18-144 · ummah-platform-worker/src/workers/webhook-in-adyen.processor.ts:15-86 · ummah-platform-backend/prisma/schema.prisma:1273, 752, 2168

Section 2

The vocabulary

Twelve terms carry this whole page. Everything else is composition.

Section 3

Ingress & verification

All Adyen traffic — every product, every event family — lands on a single endpoint: POST /webhooks/adyen in ummah-platform-webhook. The controller distinguishes two wire formats and verifies each differently:

• BP-native ({type, environment, data}): the raw request body is HMAC-signed by Adyen, with the signature in the HmacSignature header. The ingress computes the HMAC against three configured keys — hmacKey (the Configuration webhook), transferWebhookHmacKey (ADYEN_TRANSFER_WEBHOOK_HMAC) and balanceWebhookHmacKey (ADYEN_BALANCE_WEBHOOK_HMAC) — and accepts on any match. Adyen configures Configuration, Transfer and Balance webhooks as separate subscriptions, each with its own secret; pointing all three at one endpoint and trying each key avoids running three routes.
• Classic (notificationItems[]): each item is verified individually by recomputing the HMAC over the item's canonical field projection, against hmacKey and paymentHmacKey (ADYEN_PAYMENT_HMAC_KEY).

Sumsub has a twin door: its own controller in the same service (sumsub-webhook.controller.ts) verifies the Sumsub HMAC, persists a SumsubWebhookEvent row and enqueues to the worker's Sumsub processor, which runs the merchant KYC state machine — a GREEN review answer enqueues Adyen provisioning, RED fails the merchant. That downstream story lives on the onboarding page of this series; here it matters only that Sumsub events ride the identical persist–dedupe–stamp spine.

ummah-platform-webhook/src/webhooks/adyen-webhook.controller.ts:42-119 · ummah-platform-webhook/src/webhooks/ingress.service.ts:50-107 · ummah-platform-webhook/src/config/configuration.ts:202-205 · ummah-platform-webhook/src/webhooks/sumsub-webhook.controller.ts · ummah-platform-worker/src/workers/webhook-in-sumsub.processor.ts:1-16

Section 4

Dedupe & at-least-once delivery

Adyen and Sumsub both redeliver webhooks until acknowledged, and BullMQ retries failed jobs — so the same event will arrive more than once. The pipeline never promises exactly-once delivery; instead it makes duplicates cheap at three separate layers and requires every handler to be idempotent.

%%{init:{"theme":"base","themeVariables":{"fontFamily":"Bricolage Grotesque, Inter, Segoe UI, sans-serif","fontSize":"13.5px","primaryColor":"#FFFFFF","primaryTextColor":"#00273A","primaryBorderColor":"#D2D8DE","lineColor":"#8195A1","secondaryColor":"#F1F4F8","tertiaryColor":"#F7F8FA","clusterBkg":"#F7F8FA","clusterBorder":"#E5E8EC","edgeLabelBackground":"#FFFFFF"},"flowchart":{"curve":"basis","nodeSpacing":36,"rankSpacing":48,"padding":10}}}%%
flowchart TD
  RX["Verified event arrives"]:::adyen --> KEY["Compute dedupeKey
bp: + sha256 raw · or eventCode:psp:success"]:::ummah
  KEY --> SEEN{"Row exists with
processedAt stamped?"}:::plain
  SEEN -->|"yes"| DROP["Drop the redelivery"]:::plain
  SEEN -->|"no"| ENQ["Persist event row +
enqueue jobId = sha256 of dedupeKey"]:::ummah
  ENQ --> RECHK{"Worker re-checks
processedAt"}:::plain
  RECHK -->|"stamped meanwhile"| DROP
  RECHK -->|"fresh"| RUN["Dispatch to handler
idempotent upsert or state machine"]:::ummah
  RUN -->|"success"| STAMP["Stamp processedAt"]:::sub
  RUN -->|"throw"| RETRY["BullMQ exponential backoff ×3
then PSP redelivery re-enters"]:::warn
  RETRY -.-> RECHK
  classDef ummah fill:#FBE5D6,stroke:#E86C2B,stroke-width:1.5px,color:#00273A;
  classDef liable fill:#E86C2B,stroke:#B84E1F,stroke-width:1.5px,color:#FFFFFF;
  classDef adyen fill:#00273A,stroke:#00273A,color:#FFFFFF;
  classDef client fill:#E8F0FE,stroke:#1A73B8,stroke-width:1.5px,color:#00273A;
  classDef sub fill:#E6F4EA,stroke:#1E8E3E,stroke-width:1.5px,color:#00273A;
  classDef warn fill:#FEF3DC,stroke:#B26A00,stroke-width:1.5px,color:#00273A;
  classDef danger fill:#FCE8E6,stroke:#C5221F,stroke-width:1.5px,color:#00273A;
  classDef plain fill:#FFFFFF,stroke:#D2D8DE,stroke-width:1.5px,color:#00273A;

The retry ladder is worth stating precisely. A handler that throws rides BullMQ's exponential backoff for three attempts. Beyond that, the event row exists but stays unstamped, and Adyen's or Sumsub's own redelivery schedule re-enters the pipeline at ingress — the unstamped row means the redelivery is not dropped, and a fresh job is enqueued. The system therefore converges as long as the underlying fault clears, without any manual replay machinery.

ummah-platform-webhook/src/webhooks/ingress.service.ts:18-144 · ummah-platform-worker/src/workers/webhook-in-adyen.processor.ts:56-62 · ummah-platform-backend/prisma/schema.prisma:1273 (AdyenWebhookEvent dedupeKey unique)

Section 5

The event-code inventory

The worker's dispatcher (AdyenWebhookService.dispatch for classic, dispatchBalancePlatform for BP-native) is the single routing table for everything Adyen says. Below is the complete handled set and what each code changes locally. Every classic payment event additionally writes a PaymentEvent timeline row carrying the raw payload, and fans out the corresponding merchant webhook.

5.1 · Classic payment family

5.2 · Dispute family

Every dispute code first runs handlePayment — Transaction → DISPUTED, with won/reversed outcomes restoring SETTLED — then DisputesIntakeService.ingest, whose applyDisputeEvent dedupes repeats. The full handled set:

• Opening: CHARGEBACK, NOTIFICATION_OF_CHARGEBACK — dispute row created (anchored on the payment's pspReference), local earmark taken in BalanceSnapshot.disputeReservedMinor, chargeback fee booked, dispute.opened webhook + email.
• Evidence: REQUEST_FOR_INFORMATION, INFORMATION_SUPPLIED — drive the RFI stage and the dispute.evidence_required event.
• Escalation: SECOND_CHARGEBACK, PREARBITRATION_WON/LOST, SCHEME_ARBITRATION_WON/LOST.
• Deadlines and outcomes: CHARGEBACK_REVERSED, ISSUER_RESPONSE_TIMEFRAME_EXPIRED, DISPUTE_DEFENSE_PERIOD_ENDED — settle the dispute WON/LOST with dispute.won/dispute.lost events and emails; a loss triggers the recovery waterfall and, if the platform absorbs the remainder, the recovery.written_off email.

5.3 · Balance Platform family

5.4 · Settlement is derived from transfer legs

There is no settlement webhook — no SETTLED event code is consumed anywhere. Instead, balancePlatform.transfer.* events with category platformPayment are the booked split legs, upserted into AdyenTransferLeg keyed by categoryData.pspPaymentReference. reconcileSplits buckets the incoming BalanceAccount, Commission and AdditionalCommission legs (statuses booked/captured, never outgoing legs) and waits until they sum exactly to the captured or authorised amount — only then does the Transaction promote to SETTLED, with splitsReconciledAt stamped and the rolling reserve accrued. The economics of those legs — who gets what and why — are covered on Pricings, Splits & Fees; this page only cares that the legs arrive as events and are reconciled.

%%{init:{"theme":"base","themeVariables":{"fontFamily":"Bricolage Grotesque, Inter, Segoe UI, sans-serif","fontSize":"13.5px","primaryColor":"#FFFFFF","primaryTextColor":"#00273A","primaryBorderColor":"#D2D8DE","lineColor":"#8195A1","secondaryColor":"#F1F4F8","tertiaryColor":"#F7F8FA","clusterBkg":"#F7F8FA","clusterBorder":"#E5E8EC","edgeLabelBackground":"#FFFFFF"},"flowchart":{"curve":"basis","nodeSpacing":36,"rankSpacing":48,"padding":10}}}%%
flowchart LR
  TR["balancePlatform.transfer.*
category platformPayment"]:::adyen --> LEG["Upsert AdyenTransferLeg
key categoryData.pspPaymentReference"]:::ummah
  PF["PaymentFee 0-marker legs"]:::plain -.->|"excluded"| BUCKET
  LEG --> BUCKET["Bucket incoming booked legs
BalanceAccount · Commission · AdditionalCommission"]:::ummah
  BUCKET --> SUM{"Legs sum equals
captured amount?"}:::plain
  SUM -->|"exact"| SET["Transaction SETTLED
splitsReconciledAt + reserve accrual"]:::sub
  SUM -->|"divergence"| MIS["splitsMismatch recorded
+ split.mismatch alert"]:::danger
  SUM -->|"FX leg currency differs"| FX["SETTLED unverified
reserve accrual skipped"]:::warn
  classDef ummah fill:#FBE5D6,stroke:#E86C2B,stroke-width:1.5px,color:#00273A;
  classDef liable fill:#E86C2B,stroke:#B84E1F,stroke-width:1.5px,color:#FFFFFF;
  classDef adyen fill:#00273A,stroke:#00273A,color:#FFFFFF;
  classDef client fill:#E8F0FE,stroke:#1A73B8,stroke-width:1.5px,color:#00273A;
  classDef sub fill:#E6F4EA,stroke:#1E8E3E,stroke-width:1.5px,color:#00273A;
  classDef warn fill:#FEF3DC,stroke:#B26A00,stroke-width:1.5px,color:#00273A;
  classDef danger fill:#FCE8E6,stroke:#C5221F,stroke-width:1.5px,color:#00273A;
  classDef plain fill:#FFFFFF,stroke:#D2D8DE,stroke-width:1.5px,color:#00273A;

ummah-platform-worker/src/modules/webhooks-in/adyen-webhook.service.ts:72-191, 210-348, 350-408 · ummah-platform-worker/src/modules/webhooks-in/transfer-event.service.ts:45-89, 229-311, 313-412 · ummah-platform-worker/src/modules/balances/balance-snapshot.service.ts:42-70 · ummah-platform-backend/prisma/schema.prisma:1477-1485

Section 6

The email pipeline

The backend is the single template authority: it renders all fifteen notification types into a Notification row — type, recipient, subject, HTML and text bodies — and posts a message containing only {notificationId} to SQS. Because the row is complete before queueing, it serves as the delivery job and the audit log in one; nothing about the email's content depends on the consumer.

ummah-platform-notification long-polls the queue (WaitTimeSeconds 20, batch size 10), loads each row by id, sends it through the SMTP mailer, and marks it SENT with sentAt. A failed send is simply not deleted from the queue: the message reappears after the visibility timeout, up to NOTIFY_MAX_ATTEMPTS (default 5), after which the row is marked FAILED. A direct-SMTP fallback path in the backend shares the exact same templates, so a queue outage degrades delivery, never content. Donor-facing invoice download emails from the invoices module ride this same pipeline.

%%{init:{"theme":"base","themeVariables":{"fontFamily":"Bricolage Grotesque, Inter, Segoe UI, sans-serif","fontSize":"13px","primaryColor":"#FBE5D6","primaryTextColor":"#00273A","primaryBorderColor":"#E86C2B","lineColor":"#8195A1","actorBkg":"#FFFFFF","actorBorder":"#D2D8DE","actorTextColor":"#00273A","actorLineColor":"#D2D8DE","signalColor":"#33505F","signalTextColor":"#33505F","activationBkgColor":"#FBE5D6","activationBorderColor":"#E86C2B","noteBkgColor":"#FEF3DC","noteBorderColor":"#E8D5A8","noteTextColor":"#00273A","labelBoxBkgColor":"#F1F4F8","labelBoxBorderColor":"#D2D8DE","loopTextColor":"#33505F"},"sequence":{"mirrorActors":false,"actorMargin":46,"messageMargin":30,"boxMargin":8}}}%%
sequenceDiagram
  participant BE as Backend
  participant PG as Notification row
  participant SQ as SQS
  participant NS as Notification service
  participant SM as SMTP mailer

  BE->>PG: render template — type, to, subject, html, text
  BE->>SQ: publish {notificationId}
  NS->>SQ: long-poll WaitTimeSeconds 20, batch 10
  SQ-->>NS: up to 10 messages
  NS->>PG: load row by notificationId
  NS->>SM: send email
  alt send succeeds
    NS->>PG: status SENT + sentAt
    NS->>SQ: delete message
  else send fails
    Note over SQ,NS: message reappears after visibility timeout
NOTIFY_MAX_ATTEMPTS 5 → row FAILED
  end
  Note over BE,SM: direct-SMTP fallback in the backend shares the same templates

ummah-platform-backend/src/core/notifications/notification-templates.ts:238-253 · ummah-platform-notification/src/consumer/notifications.consumer.ts:7-145 · ummah-platform-backend/prisma/schema.prisma:2168 (Notification), 2162 (NotificationStatus)

Section 7

Staff surfaces

Because every inbound event and every rendered email is a database row, observability is a read model, not a logging exercise.

ummah-platform-backend/src/modules/events/events.controller.ts:24-160 · ummah-platform-backend/src/core/notifications/admin-notifications.controller.ts · ummah-platform-backoffice/app/(app)/ (events, notifications screens)

Section 8

Implementation notes

Data model

• AdyenWebhookEvent (schema:1273) — unique dedupeKey, rawPayload, processedAt; SumsubWebhookEvent (schema:752) is its Sumsub twin.
• AdyenTransferLeg (schema:1477-1485) — unique adyenTransferId, category, platformPaymentType (BalanceAccount|Commission|AdditionalCommission|PaymentFee), pspPaymentReference, amount, balance account, direction, status.
• Notification (schema:2168) with NotificationStatus (schema:2162) — the rendered email, its recipient and delivery state; PaymentEvent rows form the per-transaction timeline.

Section 9

Gaps & recommendations

The spine is honest about its own limits — two of the four gaps below are flagged in code comments. Priorities: p0 correctness/money-risk, p1 commercial/ops friction, p2 polish.