Rate Limiting, Worker Scaling & Live Queue Stats

Rate Limiting

• New token-bucket rate limiter controls how fast we hit external APIs
• Each cruise line gets its own rate limit configuration -- some APIs are faster than others
• When a worker hits the limit, the entire tube pauses briefly instead of burning cycles on retry loops
• Keeps us from overwhelming cruise line servers, which means fewer blocks and more reliable data

Worker Scaling

• Doubled background workers from 4 to 8
• With rate limiting in place, more workers means parallel processing across different cruise lines without risk

Admin Queue Dashboard

• Live-updating tube statistics on the admin queue page
• Token availability display shows rate limit headroom per cruise line
• Tubes sorted by type: external scraping first, internal jobs last

Scraper Improvements

• Silversea scraper: better rate limiting, distinguishes between "price unavailable" and "sold out"
• MSC partitioned scraping now auto-discovers configuration from the scraper registry