Duplicate Email Incident, Fixed at the Root
Today 8 outbound emails were re-sent up to 21 times each to seven different recipients before the deploy's worker restart drained the backlog. Two compounding bugs:
- The email worker caught every error and unconditionally released the job for retry. When its long-lived Postgres connection went stale, every reservation re-threw "no connection to the server" and the same job came back 60 seconds later, forever. One job was reserved 87 times during the incident.
- The send path ran SMTP delivery then updated the row to
sent. When the DB blip landed between those two steps, the email was already delivered but the status update failed -- so the job retried, and each successful retry delivered another duplicate.
The fix is a proper state machine: an atomic status transition gates pending → sending → sent, so a retry on an already-delivered email matches zero rows and becomes a no-op. If the post-SMTP status write fails, the row parks in sending and never re-sends. The email worker also gained the main worker's connection-error handling -- reconnect, bury the job for the reconciliation sweep, and back off -- instead of looping.
You get exactly one email regardless of where the failure lands. New tests cover every state-collision path.
Casino Offer Buckets: Stranded Labels Merged
Follow-up to Saturday's casino fidelity work. Case-folding merged most duplicate offer-type buckets, but some stragglers remained: "OCEANVIEW" doesn't case-fold-match "Ocean View" (the space survives), and soft-removed rows kept pre-fix flag suffixes their active replacements had shed.
Bucket labels are now rebuilt from the normalized room type plus current flags instead of trusting the stored description string, so historical and active rows for the same category finally share one bucket. Buckets containing only soft-removed sailings sort to the bottom -- visible, but out of the way.