Principal Engineer for AI Software Resiliency

Posted 13 Days Ago
Be an Early Applicant
Santa Clara, CA
Expert/Leader
Artificial Intelligence • Hardware • Robotics • Software • Metaverse
The Role
As a Principal Engineer for AI Software Resiliency at NVIDIA, you will lead the engineering of resiliency features for AI supercomputers, focusing on reducing downtime, improving software quality, and integrating these features into frameworks like PyTorch and JAX/XLA. Your role involves collaboration across teams and with customers to ensure effective deployment and performance of AI resilience solutions.
Summary Generated by Built In

We are now looking for a Principal Engineer for AI Resiliency. At NVIDIA, we are pushing the boundaries of what’s possible in AI. We are seeking a Principal Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a lead focused on AI Software Resiliency, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

What You'll Be Doing:

  • AI Resiliency Expertise: Serve as a trusted authority on AI software resiliency, guiding the architecture, modeling, and scoping of resiliency features to support frontier model training at scale.

  • Feature Development: Lead the execution and development of software resiliency features, including fast checkpoint-recovery, automatic error detection, error isolation, SDC detection and mitigation, and straggler/hang detection.

  • Software Engineering Leadership: Drive engineering excellence by contributing to large software codebases, ensuring high code quality, rigorous testing, and solving complex challenges. Lead others by example, fostering a culture of collaboration, innovation, and continuous improvement.

  • Cross-Team Collaboration: Work closely with multiple teams and stakeholders across NVIDIA to align on mission requirements, provide regular updates, and ensure the successful integration of resiliency features into AI frameworks like PyTorch and JAX/XLA.

  • Customer Engagement: Collaborate directly with major customers to embed AI resilience features into their AI frameworks, ensuring seamless integration and optimal performance.

  • Product Delivery: Partner effectively with TPMs, PMs, and QA teams to ensure the timely and successful launch of resiliency features.

What We Need to See:

  • Master’s or Ph.D. in Computer Science, Electrical Engineering, Computer Engineering, or a related field from a reputed institution, or equivalent experience.

  • A minimum of 10 years of experience in systems architecture or related fields, with a deep understanding of distributed systems and large-scale AI infrastructure.

  • At least 10 years of hands-on experience in software development for distributed systems and 5 years in developing AI frameworks such as PyTorch or JAX/XLA.

  • Proven track record of working effectively across multiple engineering fields and communicate complex technical concepts to a diverse set of collaborators.

Ways to Stand Out From the Crowd:

  • AI Supercomputing Expertise: Experience with large-scale AI supercomputing applications, including in-depth knowledge of AI workload training and inference requirements.

  • System Architecture Passion: A strong passion for developing AI-specific system architectures, including CPUs, GPUs, memory, storage, and networking.

  • Lifecycle Experience: Hands-on involvement in the design, development, and deployment of large-scale AI supercomputers.

  • HPC Best Practices: Practical experience in adopting and implementing high-performance computing (HPC) software development in large-scale environments.

As NVIDIA continues to expand into the data center business, our team plays a central role in optimizing our growing data center deployments and pioneering a data-driven approach to hardware design and system software development. You will collaborate with a broad spectrum of teams at NVIDIA, from deep learning research and CUDA Kernel development to Silicon Architecture. Intelligent machines powered by Artificial Intelligence computers that can learn, reason and interact with people are no longer science fiction. GPU Deep Learning has provided the foundation for machines to learn, perceive, reason and solve problems. NVIDIA's GPUs run AI algorithms, simulating human intelligence, and act as the brains of computers, robots and self-driving cars that can perceive and understand the world. Increasingly known as “the AI computing company”, NVIDIA wants you.

NVIDIA is recognized as one of the world’s most desirable technology employers, home to some of the most forward-thinking and hardworking people in the world. If you’re creative, autonomous, and driven by innovation, we want to hear from you!

The base salary range is 272,000 USD - 419,750 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Top Skills

C++
The Company
HQ: Santa Clara, CA
21,960 Employees
On-site Workplace
Year Founded: 1993

What We Do

NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern AI — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as “the AI computing company.”

Similar Jobs

Afterpay Logo Afterpay

Senior Data Platform Engineer

Fintech • Payments • Software • Financial Services
Hybrid
8 Locations
900 Employees
126K-223K Annually

Voltage Park Logo Voltage Park

Platform Engineer

Artificial Intelligence • Cloud • Hardware • Machine Learning • Other • Software • Infrastructure as a Service (IaaS)
San Francisco, CA, USA
51 Employees
120K-180K Annually

Crusoe Energy Systems Logo Crusoe Energy Systems

Electrical Engineer - Facilities

Cloud • Greentech • Other • Energy
Hybrid
San Francisco, CA, USA
450 Employees
170K-200K Annually

Crusoe Energy Systems Logo Crusoe Energy Systems

Electrical Engineer - Capital Projects

Cloud • Greentech • Other • Energy
Hybrid
San Francisco, CA, USA
450 Employees
170K-200K Annually

Similar Companies Hiring

Jobba Trade Technologies, Inc. Thumbnail
Software • Professional Services • Productivity • Information Technology • Cloud
Chicago, IL
45 Employees
RunPod Thumbnail
Software • Infrastructure as a Service (IaaS) • Cloud • Artificial Intelligence
Charlotte, North Carolina
53 Employees
Hedra Thumbnail
Software • News + Entertainment • Marketing Tech • Generative AI • Enterprise Web • Digital Media • Consumer Web
San Francisco, CA
14 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account