Dispatch email campaigns to millions, at a pace your domain can take

What it is

A massive campaign dispatcher, running in production, that sends email campaigns to millions of recipients at a pace your sending domain can take. It runs in your own cloud account: your team uploads a contact list and an HTML template, with placeholders filled from any column of the list, chooses when to send, and the platform does the rest. Every email is tagged with its campaign, so deliveries, opens, clicks, bounces and complaints come back as numbers per campaign and as a report per recipient.

The problem it solves

At millions of recipients, sending a campaign is an engineering problem, not a button. Send it all at once and mailbox providers see a sudden flood from your domain: a good share lands in spam, and the domain’s reputation suffers for every campaign after it. Send it from a marketing suite priced per contact and large lists get expensive fast, so they get trimmed to save money.

Then the questions start. Did this customer get it? Why did that campaign underperform? The data about who received what lives in someone else’s system, and every new business unit means another contract.

What you get

• Campaigns to millions of recipients, sent reliably: a list of any size streams through the sender, and a campaign that crashes is never resent on its own.
• Sending that protects your domain: a steady, throttled pace you raise as your reputation warms up, without a deploy.
• An answer to “did this customer get it?”: a per-recipient report with every event, next to the columns of your own list.
• Campaigns that run on time without anyone watching: schedule a send and it starts when it is due.
• A cost that follows what you send: you pay the cloud email service per message, not a suite per contact, and almost nothing between campaigns.
• One design for every business unit: each gets its own account and its own sender identity, deployed from the same code.

How we deliver it

As a Systems Integration engagement. We deploy the platform into your cloud account, verify your sending domains, connect email events to the reporting archive, and agree on a warm-up plan that raises the sending rate step by step. Your campaign tool or internal front end talks to it through a small API. We need access to the account, control of your DNS records, and a sample of your list format. After launch, an Engineering Retainer covers new business units, rate changes and new reports.

Is it for you

It fits when you send large campaigns from lists you already own, want the data and the bill in your own cloud account, and need to prove what happened to any one email. It is not for you if your team needs a drag-and-drop editor, customer journeys or A/B testing: this sends prepared campaigns from a file. It is also not for transactional email triggered one by one, and at small volumes a hosted marketing tool is simpler.

Design goals

1. Sending never outpaces what the domain’s reputation can take.
2. Lists and templates never pass through the API; they go straight to storage.
3. A send that crashes is never resumed or repeated on its own; a person decides.
4. Every email carries its campaign’s tag, so every event traces back to it.
5. Heavy work runs only while there is a campaign to prepare or a report to build.

Architecture

1. People and triggers

   Marketing teamUploads list and template

   SchedulerDue campaigns, metrics refresh

2. Apps

   Campaign APIServerless, expiring upload links

   Preparation jobsContainers, started on demand

   Send workerReads queue, throttled sending

3. State

   Campaign recordsState, schedule, metrics

   FilesLists, templates, reports

   QueueCampaigns ready to send

4. Sending and tracking

   Email serviceSends, emits every event

   Event archivePartitioned by day

   Query engineSQL over events and lists

Each piece is sized for how it is used. The campaign API is a single serverless function behind an HTTP gateway: it is called a handful of times per campaign and sits idle the rest of the day. Creating a campaign returns two expiring upload links, one for the list and one for the template, so the browser sends the files straight to storage and the API never holds them.

Campaign records live in a key-value table with pay-per-request billing, with indexes by state and by creation date. The state field (created, scheduled, enqueued, dispatching, completed, aborted, failed) is what the scheduler, the worker and the reports all read, so there is one place to see where any campaign stands.

Preparation and report jobs run as containers started on demand. Getting a large list ready, or building a per-recipient report, can take longer than a function is allowed to run, and nothing should bill between campaigns.

The send worker is a small container fed by a queue. It scales on queue depth with a ceiling of one, so two campaigns never compete for the same sending rate.

Tracking runs on files, not on a database of events. The email service publishes every event for the campaign’s configuration to an archive partitioned by day, and a query engine runs SQL over it.

How the hard parts work

Sending at a pace the reputation can take

1. Stream the listRead in batches, memory stays flat
2. PersonalizePlaceholders filled from list columns
3. Send a batchIn parallel, tagged with campaign
4. Hold the paceAt least a second per batch

A crash never sends twice. A campaign left mid-send is marked aborted when the worker restarts; nothing resends on its own.

The worker streams the list from storage line by line, so a very large file uses the same memory as a small one. It sends one batch in parallel, measures how long that took, and sleeps for the rest of the second. Batch size is the sending rate, and it is read from a configuration table at the start of every campaign, so the team raises it as the domain warms up without a deploy. The same table holds a pause flag: set it, and the worker stops taking new campaigns. On every start the worker looks for campaigns stuck in “dispatching”, which only happens after a crash, and marks them aborted.

Scheduling without a server that waits

A scheduled rule calls the API every few minutes. It reads the index of campaigns in the “scheduled” state and launches preparation for any that are due within the next window or already past. A campaign launched for “now” skips the wait and starts at once. A failed call is retried, and anything missed is simply picked up on the next run, because the state is the source of truth, not the timer.

Preparing a list

The preparation job downloads the list, copies it to its own location and registers it as a table whose columns come from the file’s header. That is what lets a report later join each recipient’s events to any column in the list, such as a customer number. Only then is the campaign put on the queue. If preparation fails, the campaign is marked failed and never queued.

From events to numbers

Every few hours a scheduled job runs one SQL query over a recent window of days and counts, per campaign, sends, deliveries, bounces, complaints, delivery delays, opens, clicks and unsubscribes, both total and unique, and writes them onto the campaign record. The query names the day partitions it needs, so it scans only the days a campaign could have events. Per-recipient reports and top links are built on demand the same way, written to a file and handed back as an expiring download link.

Decisions

Question	Options	We chose	Why
Where sending runs	Functions; a container worker fed by a queue	A container worker, one at a time	A large campaign outlasts a function; one worker keeps pace exact
How fast to send	As fast as possible; fixed in code; a configuration table	Configuration table, read per campaign	Raise it during warm-up without a deploy
How files arrive	Through the API; expiring direct upload links	Direct upload links	The API never handles large files
Where metrics come from	A webhook per event into a database; event archive and SQL	Event archive, queried on a schedule	No per-event write path to scale; pay per query
After a crash mid-send	Resume where it stopped; mark it and let a person decide	Mark it aborted	Never risk the same email twice
Network	Private subnets behind a NAT; public subnets, no inbound	Public subnets, no inbound ports	A NAT bills every hour; tasks only call cloud APIs

Guardrails

• Security. Lists and templates are uploaded through links that expire within the hour, the queue is encrypted at rest, and the containers accept no inbound connections.
• Least privilege. Every component is granted its tables, its queue and the file store explicitly in infrastructure code; nothing is set up by hand in the console.
• Cost. Pay-per-request tables and functions, jobs that run only while there is work, no NAT gateway, logs that expire after a set period, and queries that scan only the days they need.
• Reliability. A failed preparation marks the campaign failed instead of queuing it, scheduler calls are retried, and a crash mid-send ends in “aborted”, never in a silent resend.
• Operations. A pause flag stops sending without a deploy, and the same code deploys one isolated stack per business unit, each in its own account.

Failure modes

What breaks	What happens
The worker crashes mid-campaign	On restart the campaign is marked aborted; nothing resends on its own
Preparation fails	The campaign is marked failed and never queued
The email service rejects an address	The error is logged with the campaign; the batch carries on
A scheduler call fails	It is retried; the campaign stays scheduled for the next run
A metrics query fails	The job records the error; campaigns keep their previous numbers
Something looks wrong mid-send	Set the pause flag; the worker stops taking campaigns

Trade-offs, and when not to use this

One worker sends one campaign at a time, so a second campaign waits in the queue; that is the price of an exact sending rate. Delivery is at most once: a crash aborts the campaign, and resuming from the last recipient sent is not built. A single failed send is logged, not retried. Metrics refresh every few hours, not live. The API is protected by a shared token and is meant for a trusted internal front end; for wider exposure, put proper sign-in in front of it. And this is not an editor or a journeys tool: if your team designs campaigns visually or triggers emails one by one, a hosted marketing platform fits better.

What we would measure

• Sustained sending rate per worker, against the configured rate.
• Time from a campaign’s scheduled time to its first email.
• Bounce and complaint rates per campaign, during and after warm-up.
• Share of sends rejected by the email service.
• Delay between an event and its appearance on the campaign.
• Cost of a metrics refresh, and of a per-recipient report.
• Monthly running cost per business unit, with and without campaigns.