Senior Platform and EngOps Engineer - Cluster Operations

NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions from artificial intelligence to autonomous cars. NVIDIA is looking for phenomenal people like you to help us accelerate the next wave of artificial intelligence.

Join our team of innovative engineers who develop and maintain software facilitating GPU communication, driving groundbreaking solutions in High Performance Computing and Deep Learning. We're looking for highly motivated EngOps and Platform Engineers to boost execution efficiency while managing and maintaining large GPU clusters interconnected via NVLink and InfiniBand.

What you will be doing:

• Develop automated tools to efficiently deploy, provision, and maintain extensive GPU clusters interconnected via NVLink and InfiniBand

• Implement modern DevOps tools to automate software updates, perform maintenance tasks, and monitor cluster availability, ensuring seamless operations.

• Take ownership of daily cluster failures and issues, troubleshooting them promptly to maintain optimal cluster availability and performance.

• Manage the rollout and rollback of cluster software and firmware updates, ensuring smooth transitions and minimal disruptions.

• Collaborate effectively with dynamic Engineering and Product Teams across multiple time zones to align cluster operations with evolving project requirements.

What we need to see:

• BS or MS in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.

• 7+ years of hands-on experience in deploying and administrating clusters, servers, switches, and related infrastructure.

• Automation expert with hands on skills in Ansible, Python and Shell Scripting.

• Deep understanding of operating systems, computer networks, and high-performance applications.

• Proven ability to work effectively with developers and test engineers across different teams and time zones.

• Proficient with Linux fundamentals.

Ways to stand out from the crowd:

• Familiarity with resource scheduling managers, preferably Slurm.

• Direct experience with industry standard alerting tools and emergency response practices.

• Hands-on experience with GPU-focused hardware and software, such as DGX systems and Compute Clusters.

• Proficiency in crafting and implementing a robust metrics collection and alerting infrastructure.

• Proficiency in designing large scale networking technologies and the associated challenges.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.Applications for this job will be accepted at least until July 29, 2025.NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.