The Bucket That Never Emptied (#194)
The design intent was always that a per-tube token bucket divides worker time between cruise lines and Beanstalkd priority is just a tie-breaker. In practice every tube refilled 7,200 tokens an hour against 1-60 per job, so no bucket ever ran dry and every worker watched every tube. Tokens counted HTTP requests rather than time, and priority was uneven (seven lines at HIGH, Royal Caribbean, NCL, and Celebrity at NORMAL). On 9/7 NCL and Celebrity sat stale for three hours while the price-only worker skipped 36 ready Celebrity jobs for newer MSC ones.
The bucket is now denominated in worker-seconds. Each tube gets an allotment (0.35 means "35% of one worker, continuously") and a capacity sized so an hourly batch is metered rather than burst. Three operations, all per job: a worker only watches a tube whose balance covers the expected job duration; it charges the expected duration on reserve, unconditionally, which closes the race where several workers pass the gate together; and when the job ends it settles the difference between actual and expected and nudges a moving average of actuals. Slow scrapers self-correct without anyone maintaining duration tables by hand. Priority is flat across every price line.
First live hour: Celebrity's 36-job batch drained in about ten minutes, twice in a row. Two things needed tuning on the day. Short-job lines now bank a full hourly batch instead of two and a half jobs, since a freed worker that found Celebrity's gate closed would grab a 30-minute Silversea job and be gone. And a sub-second no-op job (a sync bouncing off a stale lock after the deploy) no longer drags the expected-duration average down, which had briefly tripled account-sync concurrency.
The account-sync wave also stopped dumping 87 jobs 30 seconds apart at midnight, 6am, noon, and 6pm; it's spread across most of each six-hour window.
Bigger Box
With everything properly metered, total demand sat right at the six-worker supply, and account sync was draining its wave in 6.8 hours against a 6-hour cadence. Adding a seventh worker on the old 2 vCPU / 4GB droplet meant lowering the memory ceiling to keep seven under what six used. That's a treadmill, so the droplet was resized to 4 vCPU / 8GB the same day. Ten workers now: two dedicated price lanes, one packages-only, seven generalists, with about 10% of worker time left as slack for the internal jobs that don't have a bucket.
Princess Went 25 Hours Without a Price (#195)
Around 3am on 9/10 Princess's pricing endpoint started returning 400 to any request without a booking agency in the body. Every run completed "successfully" in five seconds with zero sailings. The zero-inserts health check (added in July after NCL died silently for six weeks) caught it within the day. The request now sends the agency that matches our app ID; spot-checked fares were identical to the day before. A failed pricing fetch also can't suppress the same-hour retry anymore, since "cached" now requires the per-guest pricing file and not just the product list.