Data Engineering Lead

Role: Data Engineering Lead

Location: Remote (USA)

MediaRadar equips marketing, sales and analytics leaders with the intelligence they need to stay ahead. Our platform delivers always-on, AI-enabled Creative, Competitive, Commercial and Market Intelligence—spanning ad strategy, media spend, creative assets and brand messaging across 30+ media channels and five million brands.

With deep insights into more than 35 million ad and campaign assets and $280 billion in media spend, MediaRadar provides a single, interoperable source of truth that plugs seamlessly into enterprise analytics and AI systems. The result: faster, cleaner and more actionable intelligence that drives competitive advantage.

The Data Engineering Lead is a high-velocity, hands-on "player-coach" responsible for technical stewardship, designing scalable systems, and integrating complex Machine Learning models into robust ETL pipelines. You will lead a lean team through a cultural shift toward cross-trained agility while spending 70-80% of your time in the code. Success is defined by achieving total record processing, maintaining strict cloud cost-efficiency, and shrinking data delivery windows.

• Coding & Technical Stewardship (70-80% Hands-on): Architect and implement complex, end-to-end data pipelines using Azure Databricks and PySpark. Design, build, and maintain a scalable data architecture using the Medallion Architecture (Bronze/Silver/Gold layers).
• Performance & Cost Optimization: Optimize Apache Spark jobs, tune Databricks units, and define cluster policies to minimize compute costs. Proactively audit and refactor pipelines every 3-6 months to maintain effectiveness and reduce cloud costs. Implement caching strategies (e.g., broadcast joins) and manage performance impact.
• System Integrity & SLAs: Develop a proactive monitoring and alerts framework to ensure 99.9% reliability and mitigate system issues before they impact end-users. Build an end-to-end Data Validation Framework (e.g., Great Expectations) to enforce data accuracy and consistency. Minimize job failure rates and ensure data is available in the Gold layer within the required 24-hour turnaround time.
• Database Architecture: Architect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing complex analytical queries.
• Team Leadership & Agility: Lead a lean team toward cross-trained agility, moving away from "siloed specialists". Manage sprint cycles, conduct code reviews, and guide the team on best engineering practices (including CI/CD).
• Strategy & Scalability: Anticipate future data needs and design High-Velocity Architecture that is highly scalable and manageable to handle sudden volume increases (e.g., double the data from new sources like paid social/CTV). A critical function is translating business-level requirements into clear, technical user stories for developers.
• ML Integration: Collaborate with ML teams to integrate automated model orchestration into robust ETL pipelines.
• Collaborate with the offshore team lead to facilitate seamless knowledge transfer and operational continuity across time zones. Establish clear communication protocols, standardized documentation, and robust feedback loops to ensure alignment on project goals. Act as the primary bridge between teams to mitigate bottlenecks and maintain high-quality delivery standards.

• Core: Python, PostgreSQL + pgvector.
• Big Data: Azure Databricks, PySpark, Delta Lake
• DevOps: Docker, Git, Azure DevOps, CI/CD

• 10+ years of experience in Data or Software Engineering with deep codebase involvement.
• 3+ years as a Technical Lead managing agile teams.
• Proven ability to lead lean, high-impact teams while maintaining high individual output.
• Experience with cross-training advocacy and scaling data processing through automation.

Desired Qualifications

• Workflow Orchestration: Experience with Apache Airflow.
• Containerization: Familiarity with Azure Kubernetes Service (AKS).