44. Design a Payment System

Tests: idempotency, distributed transactions, integration with PSPs, ledger, fraud, regulatory compliance.

44.1 Requirements

Functional

• Charge a customer (one-time or recurring).
• Refund.
• Payouts to merchants.
• Multi-currency.
• Multiple PSPs (Stripe, Adyen, PayPal).
• Dispute / chargeback handling.

Non-functional

• Strong consistency: never lose or double-charge.
• High availability (downtime = lost revenue).
• Compliance: PCI-DSS (don't store card numbers), KYC/AML.
• Regional constraints: GDPR, currency rules.
• Audit trail: every action logged immutably.

44.2 Architecture

[ Web/Mobile ] (collect card via tokenizer)
       │
[ Payment API ]──→[ Order svc ]
       │              │
       │     Kafka(PaymentRequested)
       ↓              │
[ Payment Processor ]──Saga──→[ PSP adapter ]──→[ Stripe / Adyen / ... ]
       │
[ Ledger ] (double-entry accounting)
       │
[ Dispute svc ] [ Reconciliation svc ] [ Payouts svc ] [ Fraud svc ]
       │
[ Reporting / data warehouse ]

44.3 Tokenization (PCI scope reduction)

Don't touch raw card numbers. Use the PSP's client SDK:

• Card data goes directly from browser to PSP (Stripe.js, Adyen Drop-in).
• PSP returns a token (e.g., tok_abc...).
• Your server only stores tokens.

This keeps you out of PCI scope (or reduces it to SAQ-A).

44.4 Charge flow (the core)

1. Order svc → Payment API: POST /charges
   { amount, currency, source: token, idempotency_key, customer_id }

2. Payment API:
   a. Idempotency check: if key seen, return prior result.
   b. Create Charge record (status=pending).
   c. Decide PSP (routing rules: country, currency, A/B).
   d. Call PSP via adapter.
   e. PSP returns success / decline / processing.
   f. Update Charge record.
   g. Emit ChargeCompleted event.
   h. Return result.

3. Ledger svc consumes ChargeCompleted:
   a. Debit customer; credit merchant; add fees.
   b. Atomic in ledger DB.

44.5 Idempotency

The single most important property.

• Client sends Idempotency-Key per logical operation (Stripe convention).
• Server stores (key, request_hash, response, status, created_at).
• On repeat: return cached response if request matches; reject if request differs.
• TTL: typically 24h; long enough for retries.

Without this, retries (network blips, client retries) cause double-charges.

44.6 Ledger

The source of truth for money. Double-entry: every transaction is paired debits + credits that sum to zero.

ledger_entries (
  txn_id, account_id, amount (signed), currency, ts, ref
)

For a $10 charge:

• credit customer:42 by -$10
• debit merchant:99 by +$9.71
• debit fees:stripe by +$0.29

Sum = 0. Always.

Account balance = sum of entries. Materialize for fast lookup; recompute periodically for verification.

Why double-entry

• Self-checking: every entry must balance.
• Audit-friendly.
• Standard accounting practice.
• Reconciliation against PSP statements becomes math, not heuristics.

Ledger DB

• Postgres / Spanner: ACID critical.
• Schema enforces constraints (sum-zero check via app).
• Append-only; never UPDATE / DELETE entries — corrections are reversing entries.

Performance

Ledger is write-heavy. Sharding by account_id helps; per-account balance updated atomically.

44.7 PSP integration

Each PSP has different APIs. Adapter pattern:

• Common interface (charge, refund, void, getStatus).
• Per-PSP adapter implements it.
• Routing service picks PSP per (currency, country, merchant config, A/B test).

Multi-PSP for redundancy (Stripe down → fall back to Adyen) and cost optimization.

44.8 Async / webhooks

Some PSP responses are async ("payment is processing; we'll let you know"):

• Initial response: status=processing.
• PSP webhook later: success/failure.
• Webhook signature verification (HMAC).
• Idempotent processing: webhook may retry on our error.

Outbox pattern (chapter 22): we write status + emit event in one transaction.

44.9 Refunds

POST /refunds { charge_id, amount, idempotency_key, reason }

Steps:

• Look up original charge.
• Validate (refundable amount, time window).
• Call PSP refund API.
• On success: ledger reversing entries.
• Emit RefundCompleted.

Partial refunds: ledger entries proportional.

44.10 Recurring / subscriptions

Subscription state machine: trialing → active → past_due → canceled.

Scheduler:

• Periodically charges renewals.
• Saved payment method (stored token).
• Failure → retry per schedule (dunning); cancel after N attempts.

Subscription engine often a separate microservice.

44.11 Payouts

Money from your account to merchants:

• Schedule based on merchant terms (daily, weekly, instant).
• Aggregate ledger entries since last payout.
• Initiate PSP payout (Stripe Connect) or bank ACH/SEPA.
• Update ledger; reconcile with bank statement.

44.12 Disputes / chargebacks

Customer disputes; bank initiates chargeback.

• PSP webhook notifies dispute.
• Hold the disputed amount.
• Workflow to gather evidence (delivery receipts, emails).
• Submit to PSP / bank; wait for ruling.
• Ledger reflects all stages.

Some disputes auto-respond (low value, recurring legitimate). Others need manual.

44.13 Reconciliation

Daily PSP settlement files. Match every transaction in ledger to PSP statement.

• Discrepancies → investigation queue.
• Sources of mismatch: webhooks lost, fees changed, FX rounding.
• This is where bugs become visible; a robust recon process is the safety net.

44.14 Fraud detection

• ML model scores transactions in real-time.
• Features: amount, geo, device fingerprint, velocity (recent activity), historical patterns.
• High score → step-up auth (3DS), decline, or manual review.
• Feedback loop: chargebacks → labeled training data.

3DS (3D Secure): bank pushes auth challenge to cardholder. Liability shift to bank if completed.

44.15 Multi-currency

• Store amounts in smallest unit (cents, paise).
• FX rates from a quoted feed; lock rate at transaction time.
• Settlement currency may differ; track both.

44.16 Compliance

• PCI-DSS: tokenize, don't store cards.
• KYC (Know Your Customer): for marketplaces / payouts.
• AML (Anti-Money Laundering): monitor large or suspicious transactions; report.
• GDPR: PII handling.
• PSD2 (EU): SCA (Strong Customer Authentication).
• Regional: different per country.

44.17 Audit log

Every state change recorded:

• What changed, who, when, why.
• Immutable, append-only.
• Retention per regulatory rules (7+ years often).

44.18 Failure modes

• PSP down: fall back to alternate; otherwise queue + retry; surface "try again later" to user.
• Webhook lost: idempotent retry; reconciliation catches missed updates.
• Ledger inconsistency: alert; halt new charges until investigated.
• Double-charge (severe): refund automatically + alert; root-cause urgently.

44.19 Sketch

[Client + tokenizer]──token──→[Payment API]──→[Idempotency check]──→[PSP routing]
                                                                          │
                                                                  [PSP adapters]──→PSPs
                                                                          │
                                                            outbox + Kafka(charge events)
                                                                          │
                                                                   [Ledger svc]
                                                                          │
                                                            [Reconciliation, payouts, disputes]
[Webhooks from PSPs]──→[Webhook receiver]──verify signature──→Kafka
[Fraud svc]──→inline scoring + offline training
[Audit log]

44.20 What an interviewer wants

• Idempotency keys everywhere.
• Tokenization to scope out PCI.
• Double-entry ledger as source of truth.
• Saga / outbox for distributed coordination.
• Webhooks signed and replayed.
• Reconciliation as a backstop.
• Fraud detection in the loop.

44.21 Common pitfalls

• Storing card numbers → catastrophic PCI scope.
• No idempotency → double-charges.
• No ledger / single-entry → can't reconcile.
• Sync provider call without retry/queue → outages cascade.
• No reconciliation → divergence undetected.

Key takeaways

• Tokenize cards; never touch raw PANs.
• Idempotency keys end-to-end.
• Double-entry ledger as source of truth; sum-zero invariant.
• Multi-PSP via adapter; automatic failover.
• Webhooks + reconciliation handle async settlements.
• Fraud + dispute + payouts are major sub-systems.