We are looking for a Principal Data Engineer to lead the architecture, development, and operation of scalable, reliable data platforms. This role requires deep expertise in ETL, data pipelines, data modeling, and distributed systems, with hands-on proficiency in Apache Spark, Python, and SQL.
As a technical leader, you will shape architecture across team boundaries, anticipate customer needs, and make decisions that balance immediate delivery with long-term scalability, reliability, security, and cost. You will own complex, ambiguous problems from design through production and guide others toward high engineering standards.
You will bring an AI-native approach to engineering—using and improving AI-assisted development practices to accelerate delivery while taking accountability for the quality, correctness, and responsible use of AI-generated work.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
Lead data-platform architecture: Design and evolve ETL workflows, data pipelines, and data models using Spark, Python, and SQL.
Drive hands-on, end-to-end implementation: Build data solutions using Spark (or similar distributed systems), open table formats such as Delta Lake, and Parquet files; develop scalable, secure APIs and data access controls, including RBAC; provision resources through Infrastructure as Code; and deliver production-grade deployments through CI/CD with observability and operational runbooks.
Think beyond immediate requirements: Own complex system-design decisions, evaluate alternatives, and account for cross-team dependencies, scalability, resiliency, disaster recovery, and cost.
Make evidence-based decisions: Proactively gather facts, validate assumptions, test hypotheses, and use telemetry and investigation to resolve complex problems rather than relying on unverified assumptions.
Advance AI-native engineering: Apply AI across design, coding, testing, and troubleshooting; validate AI-generated outputs and help the team adopt effective, responsible practices.
Own production outcomes: Establish monitoring and operational readiness, drive incident resolution and root-cause analysis, and implement preventive improvements. Automate safe deployments, targeting zero-touch deployment where possible.
Raise engineering quality: Lead design and code reviews, strengthen testing, and embed security, privacy, and compliance into solutions throughout their lifecycle.
Collaborate and mentor: Communicate trade-offs clearly, align partner teams on dependencies and ownership, mentor engineers, and share knowledge to improve team capabilities.
Qualifications
Required Qualifications:
- Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
- Master's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR Bachelor's Degree in Computer Science or related technical field AND 15+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
- Experience building data platforms for experimentation, metrics, and scorecards, including understanding success metrics, guardrail metrics, and data quality.
- Experience in Databricks like platforms.
Software Engineering IC6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
Skills Required
- Bachelor's degree in Computer Science or a related technical field and 8+ years of technical engineering experience with coding in languages including C, C++, C#, Java, JavaScript, or Python; equivalent experience accepted.
- Master's degree in Computer Science or a related technical field and 12+ years of technical engineering experience with coding in languages including C, C++, C#, Java, JavaScript, or Python; equivalent experience accepted.
- Bachelor's degree in Computer Science or a related technical field and 15+ years of technical engineering experience with coding in languages including C, C++, C#, Java, JavaScript, or Python; equivalent experience accepted.
- Experience building data platforms for experimentation, metrics, and scorecards, including success metrics, guardrail metrics, and data quality.
- Experience with Databricks-like platforms.
Microsoft Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Microsoft and has not been reviewed or approved by Microsoft.
-
Fair & Transparent Compensation — Pay is presented as broadly competitive overall, with clear role/level/location variation and an emphasis on using posted ranges and band information for apples-to-apples comparisons.
-
Retirement Support — Retirement benefits are described as a standout, highlighted by a strong 401(k) match structure and immediate vesting, plus additional plan features for tax-advantaged saving.
-
Parental & Family Support — Family-oriented benefits are portrayed as a meaningful strength, with substantial paid parental leave and added supports like back-up care and adoption/surrogacy assistance.
Microsoft Insights
What We Do
At Microsoft, our mission is to empower every person and every organization on the planet to achieve more. Our mission is grounded in both the world in which we live and the future we strive to create. Today, we live in a mobile-first, cloud-first world, and the transformation we are driving across our businesses is designed to enable Microsoft and our customers to thrive in this world.
