Job Description
Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5M H200 GPU cluster that regularly lights up tens of thousands of machines.
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling
Desired Experience
You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
You have experience with some high performance language (C++, Rust, etc.)
You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
You're comfortable optimizing a system to an exceptional degree
You care about the problem of finding high quality knowledge and re...