Akamai Bot Manager vs. the Scraper
Around June 12, both MSC and Regent went stale. The cause wasn't IP reputation -- the block persisted from a clean residential egress. Both sites sit behind Akamai Bot Manager, which fingerprints the TLS/JA3 handshake itself and 403s anything that looks like plain PHP/curl, no matter where it connects from.
The fix: an opt-in impersonate mode on the HTTP client that routes requests through the curl-impersonate binary, presenting a genuine Chrome TLS fingerprint while preserving the native return shape and raw-response logging. MSC and Regent now scrape through it; both lines are current again.
Alongside it, the scraping proxy gained authentication support (PROXY_AUTH), so we can route through password-protected proxies in both HTTP clients.
Letting Long Scrapes Finish
Two follow-on fixes, because un-blocking the scrapes exposed lifecycle bugs around long scrapes:
- Orphan cleanup was killing live runs. Every worker startup marked ALL in-flight scraper runs as failed -- so with 10 workers cycling, an unrelated restart would "orphan" an MSC or Regent scrape another worker was still actively processing, and those lines could never log a completed run. Cleanup now only reaps runs older than an hour, comfortably past the longest legitimate scrape.
- Regent's job TTR was too short. The full Regent scrape (~683 detail pages) runs slower through curl-impersonate -- about 18 minutes observed -- and the 10-minute reservation window meant Beanstalkd kept releasing and eventually burying the job mid-run. Bumped to 30 minutes.