Senior Python Scraping Engineer

AdthenaLondon, United Kingdom7 months ago

As a Senior Python Scraping Engineer, you will design, build, and operate large-scale, resilient web scraping systems, with a strong focus on Google and Google-like search environments. This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale. You will work on complex scraping problems where naïve approaches do not work, including dynamic rendering, anti-bot defences, frequent markup changes, and high-volume data collection. We are leaders in adopting coding with AI tools. The role is ideal for someone who believes in agentic coding and would like to improve their skills.

Responsibilities

Design and maintain large-scale Python scraping systems that operate reliably against Google and other highly defended websites
Build scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid approaches
Continuously adapt to changes in page structure, request flows, and anti-automation mechanisms
Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability
Implement proxy, fingerprinting, and request-routing strategies to maximise stability and throughput
Monitor scrape health, detect failures early, and rapidly debug production issues
Optimise performance, cost, and latency across scraping infrastructure
Work closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted
Document scraping logic and operational runbooks for long-term maintainability

Who We Look For

Required Experience:

Strong professional experience with Python, including production-grade scraping systems
Deep hands-on experience scraping Google or similarly protected platforms
Experience with tools such as Playwright, Selenium, Puppeteer, or equivalent browser automation frameworks
Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques
Experience handling rate limiting, CAPTCHA, IP rotation, and bot detection systems
Comfortable working with asynchronous and concurrent scraping architectures
Experience running scrapers at scale in cloud environments
Strong debugging skills and ability to reason about complex failure modes
Experience with headless browser fingerprinting and anti-bot evasion techniques

Nice To Have:

Familiarity with containerisation and orchestration, such as Docker and Kubernetes
Experience with distributed task systems and job queues
Background in data quality monitoring and anomaly detection
Experience working with search, advertising, or competitive intelligence data

What We Value

Pragmatic problem-solving in hostile and fast-changing technical environments
High ownership of systems that run continuously and at scale
A strong bias toward reliability, correctness, and observability
Ability to explain complex technical behaviour clearly to non-specialists
Curiosity about how large platforms evolve their defences and infrastructure

View all remote jobs at Adthena →

FAQ

Is the Senior Python Scraping Engineer position at Adthena remote?: Yes. This role was posted on Remote Frontend Jobs, a job board that lists exclusively fully remote positions.
When was this Senior Python Scraping Engineer job posted?: This listing was posted on January 15, 2026 and is still open for applications.
How do I apply for the Senior Python Scraping Engineer position at Adthena?: Use the apply button on this page — it takes you directly to the employer’s application page.