Slow, But Not Down (#193)
Saturday evening the homepage was taking over a second to render, with the site returning 200 the whole time, so no health check fired. After five days of uptime five of the six workers had crept from ~60MB to 410-525MB each, about 2.3GB on a 3.9GB box. Swap was at 2.5GB and the disk was busy paging. Restarting the workers put available RAM from 852MB back to 2.5GB.
Workers now exit cleanly between jobs once they pass a jittered lifetime (4 hours, ±15% so six workers started together don't all recycle in the same minute) or once memory passes a ceiling set below the point where the OS starts swapping them. systemd brings each one back in five seconds. Never mid-job.
The scheduler's "replace missing workers" logic needed to learn about this too: it used to restart instances 1 through N whenever any were missing, which would have cascade-restarted healthy workers (including the reserved price lane) several times a day. It now checks each unit individually and only starts the ones that are actually down.
Backups Without the Staging File
The nightly database dump passed 11GB and takes about 90 minutes. Staging it locally meant yesterday's file and tonight's in-progress one sat on the 78GB disk at once, which is exactly what tripped the 80% disk alert at 3:46am on 9/6. The dump now pipes straight from pg_dump through gzip into a multipart upload to DO Spaces in 32MB parts, with no local file at all.
The first real 12GB run died 66 minutes in when one part got a transient 4xx that the SDK's retry logic doesn't cover, throwing away the whole stream. The upload is now driven by hand: each part is buffered, retried up to six times with backoff, and the byte count is verified against the stored object before the run is declared good. Any failure aborts the upload so orphaned parts don't bill.
Casino Departure Ports Finally Resolve
Port alias matching was case-sensitive, and the casino sync stores departure ports in ALL CAPS ("LOS ANGELES, CALIFORNIA"), so the casino offers page kept falling back to the raw string instead of the canonical port. All alias lookups are now case-insensitive, with matching indexes.
While there: the ports tooltip on the homepage and casino page waited 800ms before even starting its ~240ms fetch. It now fetches on hover, caches per sailing, and shows at 250ms with the itinerary filling in.
RC Was Spending 24 of Every 60 Minutes on Cache Maintenance
Each Royal Caribbean month job ended by rebuilding the latest_prices cache for all ~21K sailings, not just the ones it scraped, about 40 seconds a pop. Thirty-six jobs an hour meant the reserved price lane was spending 24 minutes of every hour on cache rebuilds for 7-second scrapes. The scheduler already runs that identical rebuild every 15 minutes for every other line; RC now relies on it too.