Be an Early Applicant
Accelerate the ad technology evolution.
The Role
Designs and operates large-scale batch and streaming data pipelines processing billions of events daily. Builds canonical datasets, Airflow workflows, streaming transformations, and data models supporting reporting, billing, and machine learning. Optimizes performance, cost, scalability, reliability, and data correctness across Spark, Kafka, Flink, Hadoop, Ceph, and Kubernetes environments. Establishes data processing standards, influences architecture, leads platform initiatives, and mentors globally distributed engineering teams.
Summary Generated by Built In
At Index Exchange, we’re reinventing how digital advertising works—at scale. As a global advertising supply-side platform, we empower the world’s leading media owners and marketers to thrive in a programmatic, privacy-first ecosystem.
We’re a proud industry pioneer with over 20 years of experience accelerating the ad technology evolution. Our proprietary tech is trusted by some of the world’s largest brands and media owners and plays a crucial role in keeping the internet open, accessible, and largely free.
We process more than 550 billion real-time auctions every day (in comparison, Google processes 8.5 billion searches per day) with ultra-low latency. Our platform is vertically integrated from servers to networks and runs primarily on our own metal and cloud infrastructure. This end-to-end infrastructure is designed to provide both stability and agility, enabling us to adapt quickly as the market evolves.
At the core of it all is our engineering-first culture. Our engineers tackle internet-scale problems across tight-knit, global teams. From moving petabytes of data and optimizing with AI to making real-time infrastructure decisions, Indexers have the agency and influence to shape the future of advertising. We move fast, build thoughtfully, and stay grounded in our core values.
About the role:
We are hiring a Staff Data Engineer to build and evolve the data processing and pipeline layer that powers reporting, billing systems, and real-time data streams at Index Exchange.
This role focuses on designing and operating large-scale batch and streaming data pipelines, enabling reliable, scalable, and efficient data transformation across the platform.
You will work on systems that transform raw, high-volume event data into clean, queryable, and production-grade datasets.
You will work on high-scale data systems that:
- Process billions of events per day across distributed pipelines
- Power core data platform supporting consumers as (reporting, billing, MLO teams)
- Operate across batch (Spark) and streaming (Kafka / Flink) architectures
- Require careful balancing of:
- data correctness
- processing efficiency
- latency vs cost trade-offs
You will solve problems such as:
- Designing pipelines that scale without exploding compute costs
- Managing data correctness at scale (deduplication, late data, joins)
- Building systems that support both:
- historical backfills
- near real-time updates
- Evolving pipelines from centralized processing (Hadoop) toward more distributed and efficient patterns
- Streaming pipelines and Streaming DWs.
What we’re looking for:
- Strong experience in data engineering at scale
- Deep expertise in:
- Distributed Data Processing frameworks
- SQL and data modeling
- Leading Projects end to end
- Experience with:
- Airflow or workflow orchestration
- Kafka or streaming systems
- Managing kubernetes
- Strong understanding of:
- distributed data processing
- data modeling (large-scale datasets)
- performance optimization
- Ability to:
- own pipelines end-to-end
- debug complex data issues
- work in high-scale, evolving environments
Staff-Level Expectations
- Define data processing standards and patterns across teams
- Lead large-scale pipeline and platform initiatives
- Influence data architecture and modeling decisions
- Drive improvements across:
- reliability
- cost efficiency
- scalability
Here’s what you’ll be doing:
Data Pipelines (Batch + Streaming)
- Design and operate pipelines using:
- Spark, Flink etc.
- Kafka / Flink (streaming)
- Transform raw event data into:
- cleaned datasets (silver layer)
- business-ready datasets (gold / reporting tables)
Core Data Models & Datasets
- Build and maintain canonical datasets (aggregated datasets, reporting tables)
- Define data contracts and ensure consistency across pipelines
- Support evolving use cases:
- reporting
- billing
- ML / experimentation
Workflow Orchestration
- Build and maintain Airflow DAGs for:
- pipeline scheduling
- dependency management
- backfills
- Improve reliability and observability of workflows
Data Processing Optimization
- Optimize pipelines for:
- performance (runtime, throughput)
- cost (compute efficiency)
- scalability (data growth)
- Improve:
- partitioning strategies
- data layout
- job execution patterns
Streaming & Near Real-Time Pipelines
- Build pipelines that support:
- incremental updates
- streaming transformations
- aggregation at scale
- Contribute to evolving patterns such as:
- edge aggregation
- streaming → batch convergence
- real-time data availability
Platform & System Design Responsibilities
- Define and evolve data processing patterns:
- batch vs streaming
- aggregation strategies
- incremental vs full recompute
- Work across:
- Spark (core processing)
- Kafka (transport)
- Flink (streaming compute)
- storage systems (Hadoop / Ceph)
- Kubernetes
- Contribute to:
- data platform architecture decisions
- pipeline standardization
- reusable data processing frameworks
- Influence trade-offs:
- latency vs cost
- correctness vs performance
- compute vs storage
You will work closely with:
- Data Products
- ML and experimentation teams
- Infrastructure and on-prem systems
Here's What You Need:
- Bachelor/ Master’s Degree in Computer Science or Engineering related fields
- 8+ years of experience as a Software Engineer in enterprise grade, large scale distributed software product development
- 3+ years of work experience designing and building high performance data pipelines and applications using Hadoop/Ceph, Spark/Flink, Hive, Presto/Trino, Kafka, StarRocks/Vertica, Airflow, Kubernetes or other similar technologies
- Deep understanding of design principles of large scale distributed systems and familiar with mainstream big data related technologies and distributed frameworks
- Strong leadership, mentorship, and communication skills, with experience collaborating in a globally distributed, culturally diverse team
- Knowledge of data modeling, data warehousing, streaming data processing, and business intelligence reporting tools
- Experience working in Agile methodologies with continuous integration and delivery as CI/CD
- Experience working with containerization, and virtualization tools such as Kubernetes and Docker
Why You’ll Love Working Here:
- Comprehensive health, dental, and vision plans for you and your dependents
- Paid time off, health days, and personal obligation days plus flexible work schedules
- Competitive retirement matching plans
- Equity packages
- Generous parental leave available to birthing, non-birthing, and adoptive parents
- Annual well-being allowance plus fitness discounts and group wellness activities
- Commuter benefits and discounts, where available
- Employee assistance program
- Mental health first aid program that provides an in-the-moment point of contact and reassurance
- One day of volunteer time off per year and a donation-matching program
- Bi-weekly town halls and regular community-led team events
- Multiple resources and programming to support continuous learning
- A workplace that supports a diverse, equitable, and inclusive environment – learn more here
Equal employment opportunity
At Index Exchange, we believe that successful products are built by teams just as diverse as the audience who uses them. As such, we are committed to equal employment opportunities. We celebrate diversity of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or expression, or veteran status. Additionally, we realize that diversity is deeper than any status or classification—diversity is the human experience. For those who show grit, passion, and humility—Index will welcome you.
Accessibility for applicants with disabilities
Index Exchange welcomes and encourages individuals with disabilities to apply to work with us.
If you require an accommodation, please share the details of your request and any information how we can assist you with the hiring recruiter when they contact you. Index Exchange will make reasonable efforts to ensure accommodation requests are met throughout the recruitment process.
Index Everywhere, Index Anywhere
Our corporate headquarters are in Toronto, with major offices in New York, Montreal, Kitchener, London, San Francisco, and many other global cities. As a major global advertising exchange, we are committed to operating as a tightly knit global team and embracing and empowering talent wherever our colleagues may be.
AI Disclaimer: Automated tools including artificial intelligence may be used to assist in screening and assessing applicants for this role. All hiring decisions involve human review.
Vacancy Status: This posting is for an existing vacancy.
#LI-ONSIGHT
#LI-HF1
Skills Required
- Bachelor's or master's degree in computer science, engineering, or a related field
- 8+ years of experience as a software engineer developing enterprise-grade, large-scale distributed software products
- 3+ years designing and building high-performance data pipelines and applications
- Experience with Hadoop, Ceph, Spark, Flink, Hive, Presto or Trino, Kafka, StarRocks or Vertica, Airflow, Kubernetes, or similar technologies
- Deep understanding of large-scale distributed-system design principles and distributed data-processing frameworks
- Strong leadership, mentorship, and communication skills
- Experience collaborating with globally distributed and culturally diverse teams
- Knowledge of data modeling, data warehousing, streaming data processing, and business intelligence reporting tools
- Experience working in Agile methodologies with continuous integration and delivery
- Experience with containerization and virtualization tools such as Kubernetes and Docker
Index Exchange Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Index Exchange and has not been reviewed or approved by Index Exchange.
-
Affordable Benefits — Medical, dental, and vision coverage is described as fully paid for employees and often for dependents in the U.S., reducing out-of-pocket costs. While specifics vary by region, public materials consistently emphasize low-premium coverage.
-
Parental & Family Support — Parental leave is positioned as generous and inclusive, with birthing parents able to take extended fully paid leave and structured return‑to‑work support. Coverage also extends to non‑birthing and adoptive parents, with fertility benefits highlighted.
-
Leave & Time Off Breadth — Time off combines substantial PTO with additional health days and personal obligation days, expanding rest and recovery options. Volunteer time off and the ability to work from another location for part of the year further broaden flexibility around time away.
Index Exchange Insights
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Index Exchange is a global advertising supply-side platform enabling media owners to maximize the value of their content on any screen. As a trusted partner and ally, we connect leading experience makers with the world’s largest brands to ensure a quality experience for consumers. We’re a proud industry pioneer with over 20 years of experience accelerating the ad technology evolution. With our radically transparent business practices and dedication to total market efficiency, we’re committed to upholding the integrity of the programmatic ecosystem at large. Our global teams are dedicated to driving industry standards and building technology that delivers scale, efficiency, and long-term value for our customers and partners. In our workplace, we pride ourselves on fostering an environment of openness where everyone feels connected and supported - a place where you can be your best inside or outside of work.
Why Work With Us
When you join Index Exchange, you’ll be working at an engineering-first company. Our foundation is our technology platform and all of the Indexers who come together to improve, extend, and reinvent it. We invest heavily in building an organization that’s committed to DEI&B, our core values, and our vision of total market efficiency.
Gallery
