Every send respects consent, across tens of millions of customers

What it is

A consent platform, running in production for tens of millions of customers. Up front, a preference center your customers reach from a link in any email, with no password. Behind it, one source of truth for who wants what, rebuilt every night into the lists your sending reads. Plus a small backoffice where your team loads suppression files from other channels, exports lists, and looks up the history of any email.

The problem it solves

At tens of millions of customers, consent is a data problem, not a page. A customer’s choice lands in one system while your campaigns read another. A request that came in by phone sits in a spreadsheet until someone loads it by hand. Every miss turns into a complaint, a spam report that hurts the delivery of every other email you send, and, in regulated industries, a compliance question nobody wants to answer.

It costs time inside too. Marketing cannot see who is suppressed or why, support cannot say when someone opted out, and the old system everyone avoids touching holds the only copy of the truth.

What you get

• Opt-outs that hold: a customer’s choice reaches every campaign without anyone copying it by hand.
• Suppression requests from phone, branches or partners loaded from a file in minutes, with every invalid line reported back.
• A clear answer to “when and why was this email suppressed?” for any customer.
• Lists for sending that are always complete: a rebuild either finishes or the previous one stays.
• Off the legacy database without losing history: every past opt-out and bounce comes across.

How we deliver it

We start from what you have: where preferences live today, who reads them, and what the sending side needs. We migrate the existing data into the new system, run both side by side and compare them list by list, and switch over when the differences are explained. We stay on for the first weeks of real use.

Is it for you

It fits when you send to lists in the tens of millions, have a legacy system holding consent data, and need to prove what happened to any one customer. It is not for you if your email tool already handles preferences well and nothing else reads them, or if lists must update in real time across several tools: here, sending lists refresh daily and on demand.

Design goals

1. A customer’s choice is never overwritten by a batch process.
2. Sending reads a complete list or the previous one, never half of one.
3. Moving off the legacy database loses no opt-out and no bounce.
4. Every change to a person’s consent can be traced: what, when, from where, by whom.
5. Nothing runs, and almost nothing costs, while nobody is using it.

Architecture

1. People and inputs

   CustomersLink in any email, no login

   AdminsInvite-only backoffice

   Campaign platformCategories, read-only token

2. Apps

   Preference center APIServerless, signed link

   BackofficeServer-side sign-in

   JobsContainers, on demand

3. Source of truth

   PreferencesOne record per email

   Audit logEvery change, searchable

4. Sending

   SnapshotsOne file per list

   Query tablesSQL over the files

   Sending sideFilters every campaign

Four pieces, each sized for its traffic. The preference center API is serverless: customers arrive in bursts after a campaign, then nothing for hours. A signed, expiring code in the email link identifies the person, so there is no password and no email address in the URL. Categories come from the campaign platform marketing already edits, through a token that can only list them, cached in two layers so an outage there does not take this page down.

One record per email is the source of truth: the categories chosen, whether the person is on the global suppression list, and whether their address bounced. At this size (tens of millions of customers, millions of them on the global list, and far fewer category subscriptions) a key-value table with pay-per-request billing handles customer saves and bulk jobs without capacity planning.

Bulk work runs as containers started on demand: imports, exports, the nightly rebuild, the migration. An hours-long migration does not fit a function’s time limit, and nothing should bill between jobs.

Sending reads files, not the live table. Every night, and on any export, the records are rebuilt into one file per list, with SQL tables on top. Campaign filtering becomes a query over those tables, and the live table never sees a full-table read during business hours.

How the hard parts work

What happens when a customer saves

Every save sends every checkbox, and the audit line records what changed and where the person ended up. One rule needed real judgment: someone on the global list who now checks a category has given consent again, so they leave the list. A bounced address is the exception. It stays suppressed whatever is checked, because sending to a dead mailbox hurts delivery for everyone else. The bounce is a permanent field on the record, never inferred again from an old row.

Importing a file from another channel

Requests from phone or branches arrive as a plain file of emails. The browser uploads it straight to storage through a short-lived link, so the API never handles the bytes. The job starts only once, and only if the file actually arrived. It reports duplicates and invalid lines with their line numbers instead of failing the whole file. The action applies to one list (the global list, or a category), and people already in the requested state are left alone.

Migrating off the legacy database

1. Read the legacy sideRead-only session, read in pages
2. One record per emailKeeps the original rows
3. Respect new choicesAlready saved here? Keep it
4. Write and compareRebuilt files match the old tables

Stops before it can do damage. If a re-run would delete more than a small share of what it imported, it fails and deletes nothing.

The legacy side is read over a read-only session. Category tables fit in memory; the global list is paged by email at a steady, throttled rate. Each record keeps its original legacy rows, so rebuilt files match the old tables column for column and can be compared before switching. The migration can be re-run until the cutover, and two guards make that safe: anyone who already saved in the new center keeps their choices, and if a re-run would delete more than a small share of what it imported, it stops and deletes nothing.

Publishing the lists

The rebuild streams rows to temporary files, so memory does not grow with the table. The query tables are repointed only after every file is uploaded, and the “latest” pointer is written last. A failed upload leaves readers on the previous snapshot.

Decisions

Question	Options	We chose	Why
Where bulk jobs run	Functions; always-on workers; containers on demand	Containers on demand	Hours-long jobs, nothing running between them
How sending reads the lists	Live table; table format with row updates; plain files + SQL tables	Plain files, tables repointed per snapshot	Full rebuilds need no row-level updates; repointing is atomic
Where categories come from	Copy of the campaign database; its API	Its API with a read-only token and a cache	One source the marketing team already edits
Admin sign-in	Hosted login pages; tokens in the browser; server-side sign-in	Server-side, session in HttpOnly cookies	No token reachable from page scripts
History of a person	History table; audit log lines	Audit log lines, searchable for a set retention period	Enough to answer “what happened to this email”, nothing new to run
Abuse protection	Web application firewall; limits in the app	Per-email and per-IP limits, concurrency cap	Proportionate for a small internal tool

Guardrails

• Security. Invite-only backoffice with two roles. Server-side sign-in, session in HttpOnly, Secure, SameSite=Strict cookies, a custom header on every change, a strict content security policy. Failed sign-ins are limited per email and per IP; the CDN’s client-IP header is trusted only with a secret the CDN adds.
• Least privilege. The token used to read categories can do nothing else. Jobs can update only their own tables. The legacy database is only ever read.
• Cost. Pay-per-request storage and functions, containers only while a job runs, old snapshots expire, and the backoffice has a hard cap on concurrent executions, so a flood gets errors instead of a bill.
• Reliability. Conditional writes protect customer choices, every job is safe to re-run, mass deletes are refused, and the table keeps point-in-time recovery.
• User data. Audit search is for administrators only. The customer page never shows or accepts an email address in the URL.

Failure modes

What breaks	What happens
Campaign platform down	The last cached category tree is served; saves keep working
A job dies mid-run	Its record stays “running”; re-running is safe
An upload fails while publishing	Readers keep the previous snapshot
Legacy database read comes back incomplete	The re-run refuses a mass delete and fails loudly
Someone floods the backoffice	Concurrency is capped; extra requests get 429
A deploy changes category ids	Cached copies must be cleared before and after, or the old tree is served for a few minutes

Trade-offs, and when not to use this

Sending lists refresh daily and on every export, not in real time; if several tools need a choice within seconds, push changes to them instead. Per-category opt-outs are recorded as absence from a list, matching how the legacy tables worked; a separate “unsubscribed from X” list is not built. New bounces after the migration are not fed in yet (planned). Without a firewall, slow distributed password guessing stays under the limits; for a large public-facing admin tool, add one.

What we would measure

• Global-list size and category subscriptions after migration.
• Migration throughput, sustained.
• Nightly rebuild time at full volume.
• Export time, from request to download, at full volume.
• Preference center save latency (p95).
• Monthly running cost.
• Differences between the new lists and the legacy tables at cutover.