Web Crawling Engineer
Location: Remote / Unspecified · Category: Engineering & Development · Seniority: Mid-Level · Posted: September 7, 2026 · Verified: Oct 10, 2026
Role Overview & Description
Track team expansion and market signals for Novabeyond
Polaris7 tracks open roles, strategic market movements, and M&A developments from verified primary sources. Receive structured Watchlist briefings on Novabeyond and benchmark against direct competitors.
Novabeyond
novabeyond.comProgrammatic advertising platform and managed global media buying service.
Novabeyond
Explore business models, operating units, competitors, and recent signals behind hiring companies.
Similar Positions in DACH
Related roles across advertising, media, and data
Analyze anti-bot systems and design workarounds, build a greenfield agentic environment for testing bypass approaches, and work independently on deep technical problems. Collaborate with senior team members.
This role focuses on building and scaling the infrastructure that crawls the entire web. You will design and operate massive-scale distributed systems to handle web crawling, processing, and data ingestion. The work involves optimizing performance and reliability of the core crawling pipeline that powers Exa's AI search engine.
Analyze anti-bot and bot-mitigation systems (DataDome, Cloudflare Turnstile, Kasada, PerimeterX, Akamai) and design workarounds. Build a greenfield agentic environment for testing anti-bot behavior and bypass approaches. Work independently on deep technical problems.
This role involves working directly with high-value customers to solve complex web extraction and integration challenges. The engineer will dig into customer stacks, troubleshoot issues, and ship solutions to ensure Firecrawl works reliably in real-world scenarios. Strong problem-solving skills and deep technical expertise in web technologies are essential.
More Positions at Novabeyond
3 more open roles
Design and implement algorithms for advertising optimization and targeting. Requires 3-5 years of experience and an undergraduate degree.
Lead development of programmatic advertising algorithms for overseas markets. Requires 3-5 years of experience and an undergraduate degree.
Manage and optimize advertising campaigns for overseas markets. Requires 1-3 years of experience and an undergraduate degree.
Start with a company or market. Track what changes.
Create your Polaris7 workspace to explore extended company profiles, organize relevant players into Watchlists, and receive focused briefings when material changes occur.
Companies Hiring
Explore business models, operating units, competitors, and recent signals behind hiring companies.
