Notification system

Step 1 · Clarify — 5 minutes

• Which channels? — email, SMS, push, and “we might add WhatsApp”. That last clause is the interviewer handing you the extensibility question. Take it.
• Can a user turn things off? — yes, per event type and per channel, plus quiet hours. This is the difference between a fan-out toy and a notification system.
• What happens when a provider is down? — say the word retry and then immediately say bounded, backed off, jittered, dead-lettered. Four words, and you have covered the whole second half of the hour.
• Is delivery synchronous? — no. Checkout must not wait on Twilio. Enqueue and return.
• Do we need delivery receipts? — yes for the design, because “did it arrive?” is a support question, so notifications carry a status.
• Out of scope: writing the actual SMTP/APNs client, the templating language itself, user auth, and the analytics warehouse. Say it in one sentence and move on.

Step 2 · An event is not a notification

This is the single distinction the whole design hangs on. An event is a fact about the world: order ORD-91 shipped, for user U-2. A notification is a decision about a person: send U-2 this exact SMS body. The order service is qualified to state the fact. It is not qualified to make the decision, and it must not try.

// The publisher. It states a fact and stops.
class OrderService {
    private final NotificationService notifier;

    void markShipped(Order order) {
        shipments.record(order);
        notifier.publish(new Event(
            EventType.ORDER_SHIPPED,
            order.userId(),
            order.id(),                       // the entity, for the idempotency key
            Map.of("orderId", order.id())));  // template placeholders, nothing more
    }
}

// The decision maker. It is the ONLY class that knows channels exist.
class NotificationService {
    List<Notification> publish(Event event) {
        List<Notification> out = new ArrayList<>();
        for (ChannelType type : preferences.channelsFor(event.userId(), event.type())) {
            String body = templates.render(event.type(), type, event.data());
            Notification n = new Notification(
                event.userId() + ":" + event.type() + ":" + event.entityId(),  // dedup key
                event.userId(), type, body, Priority.of(event.type()));
            if (dedup.seen(n.key())) continue;   // at-least-once needs this
            queue.offer(n);                      // enqueue — do NOT send here
            out.add(n);
        }
        return out;
    }
}

Step 3 · Preferences decide who, and whether

One event does not mean one notification. It means zero or more, and the number is decided by data. The lookup is a three-part key — user, event type, channel — and the answer is a boolean.

• Default to sensible, not to silent. A missing row should mean on for transactional and off for promotional. Say which default you picked and why — it is a real product decision and interviewers notice when you make it deliberately.
• A global unsubscribe outranks everything. One flag on the user that short-circuits the whole matrix, checked first. Legally you want exactly one place that can be wrong.
• Quiet hours are per user timezone, not server time. One extra field, and forgetting it is how you wake somebody at 3am.
• Never let quiet hours suppress an OTP. Category matters: transactional bypasses, promotional does not. Encode the category on the event type, not in an if in the service.

Step 4 · Templates are data too — and they are per channel

The same fact reads completely differently in an inbox and on a lock screen. An email can be four paragraphs with a tracking link. An SMS gets 160 characters and costs real money per segment beyond that. A push notification is a title plus about forty characters of body before the phone truncates it for you. So the template key is not eventType — it is (eventType, channel, locale).

Step 5 · Channels behind one interface

Channel.send(Notification) → DeliveryResult. Four implementations, one method. Adding WhatsApp is a new class and some preference rows — the service does not change, because the service never names a channel. This is Strategy, and the tell that you got it right is that there is no switch on a channel string anywhere: the registry is a map from ChannelType to Channel, populated once at startup.

The thing beginners miss is that channels are not interchangeable in behaviour, only in shape. SMS has a hard 160-character segment and a per-message cost. Push needs a device token that goes stale the moment the user reinstalls the app. Email bounces — sometimes because the mailbox is full and sometimes because the address does not exist, and those two are not the same event. So a boolean return type throws away the only information the caller needs.

sealed interface DeliveryResult {
    record Success(String providerId) implements DeliveryResult {}
    record Retryable(String reason) implements DeliveryResult {}   // timeout, 503, 429
    record Permanent(String reason) implements DeliveryResult {}   // bad number, unsubscribed
}

class SmsChannel implements Channel {
    public DeliveryResult send(Notification n) {
        if (!isValidNumber(n.destination()))
            return new DeliveryResult.Permanent("invalid number");   // NEVER retry this
        String body = n.body();
        if (body.length() > 160) body = body.substring(0, 157) + "...";  // 1 segment, 1 charge
        try {
            return new DeliveryResult.Success(provider.send(n.destination(), body));
        } catch (RateLimitedException | TimeoutException | ServerBusyException e) {
            return new DeliveryResult.Retryable(e.getMessage());     // the world may recover
        }
    }
}

Step 6 · Retry, backoff, jitter, dead-letter — the second act

A send fails. You do not fail the caller — the caller was a warehouse service that finished its job twenty seconds ago. You put the message back on the queue with a later ready time, and you make each wait longer than the last: 1s, 2s, 4s, 8s. That is exponential backoff, and most candidates stop there. Stopping there is how you take down your own provider.

Here is why. A provider blips and a thousand messages fail in the same second. With pure exponential backoff, all thousand wait exactly one second and all thousand retry in the same instant — a synchronised wall of traffic hitting a service that was already struggling. It fails again, and now all thousand wait two seconds together. You have built a metronome that hammers the provider harder each round. Jitter is the fix: randomise each delay inside a window so the retries spread out instead of arriving as a spike.

class RetryPolicy {
    private final long baseMillis, maxMillis;
    private final int maxAttempts;
    private final Random random;          // injected → tests are deterministic

    /** Equal jitter: half the delay is fixed, half is random. Never zero, never synchronised. */
    long delayFor(int attempt) {
        long base = Math.min(maxMillis, baseMillis * (1L << (attempt - 1)));  // 1s, 2s, 4s, 8s
        return base / 2 + (long) (random.nextDouble() * (base / 2.0));        // [0.5b, b]
    }
    boolean exhausted(int attempt) { return attempt >= maxAttempts; }
}

class Dispatcher {
    void runOnce(long now) {
        Notification n = queue.poll(now);            // respects priority AND readyAt
        if (n == null) return;

        DeliveryResult result = channels.get(n.channel()).send(n);

        if (result instanceof DeliveryResult.Success s) {
            n.markSent(s.providerId());
        } else if (result instanceof DeliveryResult.Permanent p) {
            deadLetters.add(n, p.reason());          // ZERO retries — it can never succeed
        } else if (result instanceof DeliveryResult.Retryable r) {
            if (policy.exhausted(n.attempt())) {
                deadLetters.add(n, "exhausted after " + n.attempt() + ": " + r.reason());
            } else {
                n.nextAttempt(now + policy.delayFor(n.attempt()));
                queue.offer(n);                      // back on the queue, later
            }
        }
    }
}

Idempotency — the sentence to say out loud

“At-least-once delivery plus idempotency.” Retries mean a message can go out twice. You cannot prevent that at the network level, so you make the duplicate harmless: every notification carries a stable key — userId + eventType + entityId — and the send path checks it before doing anything. Same order, same user, same event, same key, no second SMS.

Step 7 · Enqueue, never send inline

publish() must not call channel.send(). If it does, a slow SMS provider is now inside your checkout request, and a five-second timeout at Twilio becomes a five-second checkout. publish() resolves, renders, dedupes and offers to a queue; a pool of workers drains it. That is Producer–Consumer with a very concrete payoff, and the retry machinery in Step 6 only works at all because there is a queue to put things back on. Size the pool with Thread pools & Executors, not with a thread per message.

Priority: an OTP must not queue behind a marketing blast

Marketing schedules 200,000 promos at 09:00. At 09:00:02 somebody tries to log in and needs an OTP. If both land in one FIFO queue, that OTP is delivered some time after lunch and the user is locked out. The rule is short: transactional traffic never shares a lane with promotional traffic.

• Separate queues per class (transactional / transactional-bulk / promotional) with dedicated workers is the simplest answer and the one that cannot starve — each lane has guaranteed capacity.
• A single priority queue ordered by (priority, readyAt) is fine at interview scale and is what the prototype does, but say the word starvation unprompted: an endless stream of high-priority work never lets the low lane run. The fix is a floor — reserve some workers for the low lane, or age priority upward.
• Priority comes from the event type, not the caller. Priority.of(OTP) = HIGH lives in one table. Letting each publisher declare its own priority means everything is urgent within a month.
• Retries keep their original priority but go to the back of their own lane — a failing OTP still outranks a fresh promo.

Rate limiting, batching, and digests

• Per-user throttling is a safety net against your own bugs: a loop that fires 500 pushes at one person is a support incident and an app uninstall. Cap it — “at most N notifications per user per hour, per category” — and drop or fold the excess. Do not re-teach the algorithm in the round; say “a token bucket per user” and cross-reference Rate Limiter.
• Provider-side limits are different and are a Retryable result, not a drop: a 429 from the provider means back off, not discard.
• Batching / digests turn twelve notifications into “you have 12 new messages”. Implement it as a window: hold low-priority notifications for a user for N minutes, then flush one summary. It is a scheduler plus a per-user buffer, and it is the single best answer to “how do you stop being annoying?”
• Digests must never batch transactional traffic. An OTP in a 15-minute digest is an OTP that has already expired.

Status tracking — because “did it arrive?” is a support question

The 60 minutes

The follow-ups

• “Twilio is down. Fall back to a second SMS provider.” → wrap, don't branch. A FailoverChannel holding a primary and a secondary implements Channel and tries the second only on a Retryable result. That is Decorator / Chain of Responsibility, and it keeps the failover policy out of every channel's send(). An if (provider == twilio) inside SmsChannel is the answer they are hoping you do not give.
• “Scheduled and delayed notifications.” → already free: the queue orders by readyAt, so a scheduled send is a notification enqueued with a future ready time. Same mechanism as a retry. Say that out loud — reusing one mechanism for two features is worth a point.
• “Per-locale templates.” → the template key already has a locale component; the user's locale comes from the preference row. A new language is rows, not a release.
• “How would you test this?” → a FakeChannel you construct with a script of outcomes: fail retryable twice, then succeed, or fail permanently. Inject it and inject a fixed-seed Random into the RetryPolicy, and the whole retry ladder becomes a deterministic unit test — the same trick as the injectable dice in Snake & Ladder. Assert on the dead-letter queue's contents. This is Dependency Injection & IoC earning its keep.
• “How do you know it is working in production?” → delivery rate and dead-letter rate per channel, retry counts, and time from publish to delivered at p99. A DLQ that is growing is the alarm; a DLQ nobody looks at is a bug factory.
• “A million notifications for one campaign.” → the design does not change shape, it changes deployment: the queue becomes a real broker, the workers scale horizontally, and the dedup set becomes Redis. Say that the classes survive and only the infrastructure moves — that is the answer they want.
• “What if the user changes preferences while messages are queued?” → decide and say it: resolve preferences at publish time (fast, may be stale) or at send time (fresher, more lookups). For an unsubscribe, re-check at send time — sending after an opt-out is the one that gets a complaint.

How this round is lost

• OrderService calls an email client. Everything else you say afterwards is decoration on a design that already failed the one question the problem asks.
• A boolean DeliveryResult. You now cannot tell “try again in two seconds” from “this number will never work”, and every retry strategy built on top is guessing.
• Retrying a permanent failure. Unbounded retries against an invalid number: a real incident, an easy one to avoid, and the interviewer is watching for it.
• Retrying with no jitter. “Exponential backoff” said and then stopped. A thousand synchronised retries is a self-inflicted denial of service on your own provider.
• No dedup key. At-least-once delivery with no idempotency means every retry storm double-sends, and users get the same OTP four times.
• Sending inside the request. Checkout latency now depends on an SMS provider in another country.
• A switch on a channel string. Every new channel edits that switch, and the Strategy seam you drew on the whiteboard is not actually there in the code.

The shape you just learned

Strip the notifications away and this is an unreliable side effect, decoupled from the thing that caused it. A caller states a fact; a separate pipeline decides what to do about it and owns the consequences of failing. Once you see that shape you will find it everywhere, and the same four moves apply every time: a three-way result, bounded jittered retry, a dead-letter queue, and an idempotency key.

• Webhook delivery — you POST to a customer's URL. It times out, or it returns 410 Gone. Identical problem, identical answer, and the dead-letter queue becomes a page in their dashboard.
• Payment capture and refunds — at-least-once with an idempotency key is not optional there; it is the difference between one charge and three.
• Search and cache invalidation — index writes fail, and retrying them forever against a document that no longer exists is the same permanent-failure bug.
• Audit and analytics event shipping — the producer must not block on the sink, which is why the queue comes first, exactly as in Producer–Consumer.
• Any outbound integration at all. The moment your code calls something you do not operate, you own retry classification, backoff and a place for the corpses.

Where this design stops working

• When ordering matters. “Shipped” must not arrive after “delivered”. A priority queue plus jittered retries actively reorders things. If order matters you need a per-user or per-entity sequence and a channel that respects it — which is a real cost, so only pay it where it is genuinely required.
• When the in-process queue dies with the process. Everything still pending is gone. At interview scale say so out loud and name the fix: persist the notification row before enqueuing, or use a durable broker. The classes do not change; the queue's implementation does.
• When exactly-once is actually required. It does not exist end to end — you cannot stop a phone network from delivering an SMS twice. The honest answer is at-least-once plus idempotency at every consumer, and pretending otherwise is a worse answer than admitting it.
• When the user is offline for a week. Retrying a push for seven days is pointless; the notification should expire. Add a expiresAt and drop instead of dead-lettering — an expired promo is not an incident.
• When one user has millions of events. Per-user rate limiting and digesting stop being a nicety and become the primary design constraint, and the whole thing tilts toward the batching model instead of the per-event one.