Job Requisition ID #
Position Overview
We are seeking a highly experienced Principal Engineer to lead the design, development, and evolution of our Batch Processing platform, which powers Autodesk’s Analytics Data Platform (ADP). This role requires deep technical expertise in distributed data systems, large-scale pipeline orchestration, and hands-on leadership in shaping next-generation data platform capabilities. You will partner closely with Engineering Managers, Architects, Product teams, and Partner Engineering to modernize our data lakehouse architecture, deliver highly reliable data pipelines, and establish technical excellence across ingestion, processing, and governance.
Responsibilities
Technical Leadership
Define and drive the technical strategy for Batch Processing within ADP
Design and scale PySpark based distributed processing frameworks integrated with Airflow for orchestration
Champion Apache Iceberg-based data lakehouse adoption (versioning, branching, WAP, indexing)
Introduce performance optimization patterns for data pipelines (e.g., partition pruning, caching, resource tuning)
Establish patterns for metadata-driven, intent-based data processing aligned with AI-assisted pipelines
Architecture & Delivery
Partner with architects to design multi-tenant, secure, and compliant (SOC2/GDPR/CCPA) batch processing services
Define reference architectures and reusable frameworks for cross-domain processing workloads
Lead technical reviews, solution designs, and architectural forums across ADP
Guide the evolution of Airflow APIs and SDKs to enable scalable pipeline self-service
Mentorship & Collaboration
Mentor senior engineers, staff engineers, and tech leads within the Batch Processing and adjacent teams
Partner with the Batch Ingestion and Data Security teams to deliver unified ingestion + processing flows
Innovation & Impact
Drive modernization initiatives to migrate away from legacy tools
Pioneer AI-augmented data engineering practices (e.g., pipeline optimization agents, anomaly detection)
Ensure scalability, cost-efficiency, and reliability for thousands of production pipelines across Autodesk
Influence company-wide data engineering strategy by contributing thought leadership and whitepapers
Minimum Qualifications
8+ years of experience in software/data engineering, with at least 3 years in big data platform leadership roles
Expert in distributed data processing (Spark, PySpark, Ray, Flink)
Deep experience with workflow orchestration (Airflow, Dagster, Prefect)
Strong hands-on expertise in Lakehouse technologies (Iceberg, Delta, Hudi) and cloud platforms (AWS/Azure/GCP)
Proven track record in architecting secure, multi-tenant, and compliant data platforms
Skilled in SQL/NoSQL databases, Snowflake, Glue, and modern metadata/catalog tools
Strong problem-solving, communication, and cross-geo collaboration skills
Experience mentoring engineers, building strong technical culture, and influencing at scale
Preferred Qualifications
Exposure to AI/ML-driven data engineering (optimizers, anomaly detection, auto-scaling)
Experience with data governance, lineage, and observability tools (Atlan, Databand, Collibra, etc.)
Familiarity with streaming + batch hybrid architectures (Kappa/Lambda)
#LI-KS2
Learn More
About Autodesk
Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.
We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.
When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!
Salary transparency
Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.Diversity & Belonging
We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/diversity-and-belonging
Are you an existing contractor or consultant with Autodesk?
Please search for open jobs and apply internally (not on this external site).
Top Skills
What We Do
Autodesk makes software for people who make things. If you’ve ever driven a high-performance car, admired a towering skyscraper, used a smartphone, or watched a great film, chances are you’ve experienced what millions of Autodesk customers are doing with our software. Autodesk gives you the power to make anything.
Over 100 million people use Autodesk software like AutoCAD, Revit, Maya, 3ds Max, Fusion 360, SketchBook, and more to unlock their creativity and solve important design, business and environmental challenges. Our software runs on both personal computers and mobile devices and taps the infinite computing power of the cloud to help teams around the world collaborate, design, simulate and fabricate their ideas in 3D.
We provide exceptional compensation/benefit packages and we’d love for you to join us. We’re proud to be an equal opportunity employer and we consider all qualified applicants without regard to race, gender, disability, veteran status or other protected category. To see our culture in action, check out #AutodeskLife.
We are headquartered in the San Francisco Bay Area and have more than 10,000 employees worldwide.
Why Work With Us
Our work is impactful. Our people are innovative. And our culture is inclusive. As our software shapes new solutions to the world’s biggest challenges, you shape your career path. With us, you lead the way in achieving sustainability, resilient communities, and an equitable workforce. Discover #AutodeskLife.
Gallery
