The Role
Build and maintain high-scale web scraping and data ingestion infrastructure for social media sources. Design anti-bot evasion systems, browser automation, proxy and session rotation, distributed task queues, workflow orchestration, data lake storage, monitoring, and recovery systems. Process millions of records daily with high reliability while supporting downstream AI modeling and real-time data workflows.
Summary Generated by Built In
This is a remote position.
We are looking for a Senior Data Infrastructure Engineer specializing in web scraping, anti-bot evasion, and large-scale data ingestion. In this role, you will build and maintain the core ingestion engine powering our social media data lab. You will overcome complex platform defenses to deliver millions of profile, post, and video records daily with near-zero downtime.
Responsibilities
- Anti-Bot Evasion Architecture: Build and manage stealth scraping clusters using residential proxy networks, TLS fingerprinting, headful/headless browser farms (Playwright, Puppeteer), and session rotation.
- High-Throughput Pipelines: Build fault-tolerant, scalable web-scraping pipelines that extract data from Instagram, TikTok, YouTube, X, and web sources.
- Pipeline Orchestration: Design distributed queues and workflow engines (Temporal, Ray, Apache Kafka, Celery) to manage millions of asynchronous scraping tasks daily.
- Storage & Data Lake Management: Architect structured and unstructured storage environments (Parquet, Apache Iceberg, Snowflake, S3) for downstream AI modeling.
- Monitoring & Evasion Recovery: Implement automated alerts for platform UI/API changes, blocking patterns, and proxy failures.
Requirements
- 4+ years of experience in high-volume web scraping, data engineering, or reverse engineering.
- Deep experience defeating advanced anti-bot providers (Cloudflare, Akamai, PerimeterX) via TLS impersonation, browser automation, and proxy management.
- Mastery of Python or Go, with deep knowledge of Playwright, Puppeteer, Scrapy, or Selenium.
- Experience with distributed systems and task queues (Temporal, Ray, Kafka, Redis).
- Strong SQL skills and experience with modern analytical data lakes / data warehouses.
Preferred Qualifications
- Direct experience extracting short-form video content and user metadata from TikTok, Instagram, and YouTube.
- Experience integrating scraping outputs directly into vector databases and real-time AI processing queues.
Skills Required
- 4+ years of experience in high-volume web scraping, data engineering, or reverse engineering
- Experience defeating advanced anti-bot providers such as Cloudflare, Akamai, and PerimeterX
- Experience with TLS impersonation, browser automation, and proxy management
- Mastery of Python or Go
- Deep knowledge of Playwright, Puppeteer, Scrapy, or Selenium
- Experience with distributed systems and task queues such as Temporal, Ray, Kafka, and Redis
- Strong SQL skills
- Experience with modern analytical data lakes or data warehouses
- Experience extracting short-form video content and user metadata from TikTok, Instagram, and YouTube
- Experience integrating scraping outputs into vector databases and real-time AI processing queues
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
AWISEE is a global digital marketing agency founded in 2018, specializing in SEO, Digital PR, and KOL and influencer marketing. It helps brands enter or expand in new markets through short-term visibility strategies and long-term growth solutions. Operating across Europe, North America, Asia, and the Middle East, the agency serves technology, SaaS, fintech, e-commerce, crypto, iGaming, and travel businesses worldwide.
