A Circuit Breaker for Amazon SES

The Problem

SES gives you sending quotas and reputation metrics, but no kill switch that fires on its own. The usual failure looks like one of these:

• An IAM access key with SES permissions ends up somewhere it should not, and someone uses it to send spam.
• An application bug or a retry loop sends the same email over and over.
• An old or purchased list gets imported, and bounces and complaints climb fast.

In every case the damage grows each minute nobody reacts. And when someone does react, the first question is always the same: which key, from which IP, with which sender address? Without logs, that is very hard to answer after the fact.

What It Does

• Watches three signals: sends per minute, bounce rate, and complaint rate in each region.
• Stops sending automatically: when any signal crosses its threshold, account-level sending is disabled for that region.
• Logs every email event: send, reject, bounce, complaint, and delivery events go to CloudWatch Logs, kept for a year by default.
• Names the suspects: the alert lists the top senders from the last 30 minutes by IAM user, source IP, and from address.
• Alerts on several channels: Telegram for the team, PagerDuty for whoever is on call, and email as a backup.
• Watches itself: if the responder fails, a separate watchdog alarm says so, because a silent failure here means SES was never stopped.

The whole thing is one CloudFormation template plus a few shell scripts. It is deployed once in every region where SES is used, since sending, metrics, and quotas are all regional in SES.

How the Pieces Connect

Logging
Every identity is attached to a configuration set so its events are captured.

• A ses-guard configuration set publishes email events to the default EventBridge bus.
• An EventBridge rule forwards all aws.ses events to a CloudWatch log group.
• The log group has a Retain deletion policy, so the evidence survives even if the stack is deleted.
• A setup script attaches every SES identity to the configuration set. If an identity already has its own configuration set, the script leaves it alone and only adds the EventBridge destination to it.

Detection
Plain CloudWatch alarms on the metrics SES already publishes.

• Send spike: the sum of sends over one minute is above a per-region limit, 500 by default.
• Bounce rate: the reputation bounce rate reaches 4%.
• Complaint rate: the reputation complaint rate reaches 0.08%.
• Missing data counts as not breaching, so a quiet region never raises an alarm by accident.

Response
All alarms publish to one SNS topic, which invokes a small Python Lambda.

• On an alarm, it calls PutAccountSendingAttributes with sending disabled. This is the actual stop.
• It runs a CloudWatch Logs Insights query over the last 30 minutes of send events to find the top IAM users, IPs, and sender addresses.
• It posts the result to Telegram and opens a PagerDuty incident with the same details.
• The Telegram message includes the exact commands to disable a suspicious access key and to resume sending, so nobody has to look them up mid-incident.
• When an alarm returns to normal, it sends a short notice that also says SES stays stopped until someone resumes it.

Watchdog
A guard that can fail silently is worse than no guard, because people believe they are protected.

• A CloudWatch alarm watches the Lambda’s error metric and notifies a separate SNS topic.
• If the stop call fails, the Lambda raises an error on purpose. That fires the watchdog and makes SNS retry the invocation.
• The watchdog topic goes to email and PagerDuty directly, without passing through the Lambda, so it still works when the Lambda is what broke.

Incident Flow

1. 01
   A key leaks, or a job starts looping
2. 02
   Sends per minute or bounce/complaint rate crosses a threshold
3. 03
   CloudWatch alarm publishes to the SNS alert topic
4. 04
   Lambda disables SES sending in that region
5. 05
   Lambda queries the logs for the top senders in the last 30 minutes
6. 06
   Telegram, PagerDuty, and email get the trigger, action taken, and suspects
7. 07
   A human investigates, disables the key if needed, then resumes

The automation ends at the stop. Resuming is always a person’s decision, made after someone has read the logs.

The Control Script

Day-to-day work goes through one script, ses-guard.sh. Each command takes a region or all.

Command	What it does
status	Shows whether auto stop is on, whether SES is sending, alarm states, and unconfirmed email subscriptions.
top	Lists the top senders by IAM user, IP, and from address for the last N minutes.
logs	Tails raw email events in a compact, readable format.
stop	Emergency stop. Asks you to type the region name to confirm.
resume	Turns sending back on and re-arms the alarms.
deploy	Deploys or updates the stack in every configured region, in live or test mode.
setup	Connects new SES identities to logging.

Secrets such as the Telegram token and the PagerDuty key live in a local env file that git ignores, and they reach CloudFormation as NoEcho parameters. The per-region send limits live in the same file, so each region can have a threshold that matches its normal volume.

Testing the Guard

A safety system that has never fired is only an assumption. The script has three test commands, each with a different level of impact.

Alert test
Switches auto stop off for a moment, forces the send spike alarm into ALARM, waits for the responder, and then switches auto stop back on. It confirms that Telegram, PagerDuty, and email all arrive without touching real sending. A trap restores auto stop even if the script is interrupted halfway.

Stop test
Fires the alarm for real and checks that sending stopped. It asks you to type the region name first, then offers to resume right away.

Watchdog test
Forces the Lambda error alarm into ALARM and back, to confirm the “responder failed” path reaches a person.

There is also a test mode for the whole stack. Deploying with AutoStop=false keeps every alert but never stops SES, which is useful for the first few days while tuning thresholds against real traffic.

Design Decisions

Stop first, ask later
Pausing legitimate email for a few minutes is cheap. Letting a compromised key run for an hour is expensive. So the guard leans toward stopping.

Stop the region, not the key
Disabling sending for the whole region is blunt, but it works however the emails are being sent, whether through SMTP credentials, an API key, or a role. Narrowing it down is a person’s job, with the logs in hand.

Re-arm alarms on resume
CloudWatch alarm actions only run when the state changes. If an alarm stayed in ALARM after an incident, the next breach would trigger nothing, and SES would be unguarded without anyone knowing. So resume sets every guard alarm back to OK. The reset uses a marked reason, and the Lambda ignores those transitions so it doesn’t send a misleading “back to normal” message.

Keep the evidence
The log group is retained when the stack is deleted or replaced, because the logs are the only way to answer “who sent this” afterward.

Keep it small
One template, an inline Lambda, and plain shell scripts. There is no build step and nothing to install beyond the AWS CLI and jq. Something that runs rarely should be easy to read when it finally does.

Limitations

• Reputation metrics lag: SES computes bounce and complaint rates over time, so those alarms react more slowly than the send spike alarm.
• A fixed limit is blunt: a legitimate campaign above the per-minute limit gets stopped too. The limit has to be set with real traffic in mind and raised before planned large sends.
• Only connected identities are logged: a new identity added without running setup still triggers the alarms, but its events won’t appear in the top senders query.
• Per region: a region without the stack is not guarded at all.
• Not a replacement for good IAM: the guard limits the damage from a leaked key but cannot prevent the leak. Scoped permissions and key rotation still matter.

Lessons

Build the response before the incident
The most useful parts of the alert are the list of suspects and the exact commands to run next. Deciding those in advance is what makes the response fast. It is the same idea as rollback in A Practical Deployment Pipeline.

Guard the guard
Ask one question of every safety system: what happens if it fails? Without the watchdog, the answer here would be “nothing, silently.”

Make testing a command
When testing is one command with a safe default, it actually gets done. Running the alert test after each deploy checks every channel end to end, including email subscriptions nobody confirmed.

SES Guard is a few alarms, one function, and some scripts. Before it, a problem like this was discovered by angry recipients or an AWS review. Now it is stopped within minutes and explained in a single message.