As a Principal Data Engineer (senior), you will be a technical leader responsible for architecting, building, and operating foundational systems of the enterprise data platform. This role is critical to delivering scalable data processing pipelines, self-service analytics, governance, and advanced analytics capabilities that enable the business to move with speed, agility, and urgency.
You will partner closely with Data Architects and business stakeholders to translate architecture into working systems, modernize platform capabilities, and mentor other engineers. The role provides the opportunity to lead large-scale distributed data systems using Spark and Databricks on Azure, drive platform-wide standards, and influence engineering strategy across the organization.
Responsibilities- Collaborate with Data Architects and business partners to design and evolve enterprise data architecture and platform capabilities.
- Translate architectural strategies into technical execution and influence system design and delivery across teams.
- Serve as a domain expert in data engineering standards, best practices, and patterns across the Data Platform organization.
- Design, code, and optimize complex distributed data processing systems using Spark and Databricks.
- Develop canonical data models, semantic structures, and reusable datasets for reporting and machine learning.
- Drive platform modernization initiatives such as Delta Lake and metadata-driven designs.
- Create reusable frameworks and platform capabilities to accelerate analytics, machine learning, and governed self-service access.
- Provide technical leadership and mentorship to Staff, Senior, and mid-level data engineers to enable high-quality delivery.
- Lead root-cause analysis for major data incidents and implement long-term improvements in data quality, lineage, and observability.
- Participate in architecture reviews, technical design sessions, and platform governance forums.
- Bachelor’s or Master’s degree in Computer Science, Information Systems, or equivalent experience.
- 10+ years of experience in data engineering or a related technical field.
- Expert-level proficiency in SQL, Python, and Spark for large-scale data processing.
- Extensive experience designing and building cloud-native data pipelines, data models, and distributed data systems (Delta Lake, Spark, Unity Catalog, Jobs, Workflows).
- Proven experience with Azure and building data solutions on Azure.
- Strong experience designing, tuning, and operating distributed data processing systems at scale.
- Deep knowledge of data engineering best practices including version control, CI/CD, automated testing, DevOps/DataOps, and observability.
- Proven ability to lead cross-functional technical initiatives and influence architectural direction.
- Strong problem-solving, debugging, and analytical skills in complex, multi-system environments.
- Ability to thrive in agile, dynamic, and collaborative engineering teams.
- Experience with Databricks Unity Catalog, Delta Live Tables, or Databricks Workflows.
- Skilled in advanced data modeling (dimensional, data vault, semantic layers).
- DataOps experience including pipeline observability, monitoring, and automated quality.
- Experience with metadata management and governance platforms (Unity Catalog, Purview, Collibra, Alation).
- Experience with streaming frameworks (Kafka, Event Hubs, Kinesis) and Spark Structured Streaming.
- Experience contributing to architecture review boards, technical councils, or data governance forums.
Top Skills
What We Do
Taller is the enterprise accelerator for digital transformation, expertly orchestrating hybrid teams of senior specialists and AI agents under trusted oversight — the "humans in the loop" delivering unparalleled speed, scale, and strategic impact.
Subscribe to our monthly newsletter covering the latest breakthroughs in enterprise AI: https://hubs.ly/Q03tqbNy0
