Senior Software Engineer, AI Compute Scheduling

Posted 8 Days Ago
Be an Early Applicant
4 Locations
In-Office
184K-357K Annually
Senior level
Artificial Intelligence • Computer Vision • Hardware • Robotics • Metaverse
The Role
Designs and operates scheduling infrastructure for large-scale GPU fleets across Slurm and Kubernetes. Responsibilities include topology-aware placement, fair sharing, quotas, preemption, reservations, backfill, workflow orchestration, reliability, and performance measurement. The role requires production systems ownership, on-call support, open-source contributions, and collaboration with platform, storage, networking, and AI research teams.
Summary Generated by Built In

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Research Superclusters (MARS), builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you'll help design solutions that power some of the world's most advanced computing workloads.


The MARS Scheduling team decides how NVIDIA's AI research compute gets used. That fleet spans many of the largest GPU clusters in the world, across multiple clouds and partners, running everything from frontier-model pretraining to reinforcement learning and agentic workloads. We run Slurm and Kubernetes at large scale. Our scheduling is coordinated and topology-sensitive on rack-scale systems. We manage fair sharing and quotas for hundreds of research teams. We support preemption and reservations. Elastic jobs recover from failure without losing progress. You will own real parts of the scheduling stack, and the opportunity to ship code into our open-source and internal platform code, and see the result in GPU-hours returned to researchers within days. The researchers training NVIDIA's own models are your users, and they will tell you quickly what works.

What you'll be doing:

  • Design and build scheduling for large GPU fleets on both Slurm and Kubernetes. That covers queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations and backfill that reaches the end of very deep queues.
  • Build the policy layer: fair share, quotas and allocations across many research teams, driven from a central allocation system and enforced the same way on every scheduler.
  • Run Slurm as a cloud-native service on Kubernetes, and extend Kubernetes batch scheduling (Kueue and its ecosystem) for training, reinforcement learning, sandboxes and inference.
  • Build services around the scheduler: reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs that let other platforms submit and track work.
  • Measure and raise goodput: GPU occupancy, waste, fairness and time to start for large jobs. Use the data to choose what to build next.
  • Own reliability for the components you build: on-call, incident analysis and fixes that remove whole classes of failure, so problems are found before researchers hit them.
  • Work upstream and across teams: contribute to Slurm, Kubernetes and related open-source projects, and partner with platform, storage, networking and research teams.

What we need to see:

  • BS or MS in Computer Science, Computer Engineering, Electrical Engineering or a related field, or equivalent experience.
  • 6+ years building production systems software.
  • Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn or a scheduler you built. You should be able to explain its internals, not only its configuration.
  • Strong programming in Go, C or C++, plus Python for tooling and analysis.
  • Hands-on Kubernetes experience: controllers or operators, CRDs, scheduling and resource management.
  • Solid Linux systems knowledge: containers (Docker, Podman, Enroot or Apptainer), cgroups, networking basics and debugging under load.
  • A track record of owning a service in production and making it measurably better.
  • Clear written and spoken communication with engineers and researchers.

Ways to stand out from the crowd:

  • Experience scheduling large-scale AI training: thousands of GPUs, NVLink and InfiniBand or RoCE topology, multi-node jobs that must start together.
  • Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI or similar open-source schedulers.
  • Experience running Slurm on Kubernetes (Slinky or similar), or building Kubernetes operators for batch or HPC workloads.
  • Familiarity with RL, agentic or inference workloads and what they need from a scheduler: heterogeneous jobs, sandboxes, elastic scale and fast restarts.
  • Experience with workflow orchestrators such as Flyte, Airflow, Ray or Kubeflow at scale.

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are:
184,000 USD – 287,500 USD for Level 4; and 224,000 USD – 356,500 USD for Level 5.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until November 14, 2026.

This posting is for an existing vacancy. 

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Skills Required

  • BS or MS in Computer Science, Computer Engineering, Electrical Engineering, a related field, or equivalent experience
  • 6+ years building production systems software
  • Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn, or an equivalent scheduler
  • Strong programming skills in Go, C, or C++, plus Python for tooling and analysis
  • Hands-on Kubernetes experience, including controllers or operators, CRDs, scheduling, and resource management
  • Solid Linux systems knowledge, including containers, cgroups, networking basics, and debugging under load
  • Track record of owning a production service and measurably improving it
  • Clear written and spoken communication with engineers and researchers
  • Experience scheduling large-scale AI training across thousands of GPUs with NVLink and InfiniBand or RoCE topology
  • Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI, or similar open-source schedulers
  • Experience running Slurm on Kubernetes or building Kubernetes operators for batch or HPC workloads
  • Familiarity with reinforcement learning, agentic, or inference workloads
  • Experience with workflow orchestrators such as Flyte, Airflow, Ray, or Kubeflow at scale

NVIDIA Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about NVIDIA and has not been reviewed or approved by NVIDIA.

  • Equity Value & Accessibility — Feedback suggests equity grants, stock options/RSUs, and a discounted ESPP materially boost total compensation and can deliver significant wealth-building when the stock performs well.
  • Healthcare Strength — Feedback suggests medical, dental, and vision plans with in-network preventive care at 100%, telehealth, and concierge-style support are comprehensive and valued.
  • Parental & Family Support — Feedback suggests up to 22 weeks fully paid parental leave, fertility coverage, and adoption/surrogacy reimbursement provide strong family-focused benefits.

NVIDIA Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Santa Clara, CA
21,960 Employees
Year Founded: 1993

What We Do

NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern AI — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as “the AI computing company.”

Similar Jobs

Hilton Logo Hilton

Manager, Quality Assurance

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office or Remote
2 Locations
500000 Employees
40K-100K Annually

Micron Technology Logo Micron Technology

Architect

Artificial Intelligence • Hardware • Information Technology • Machine Learning
In-Office
Richardson, TX, USA
45000 Employees

Micron Technology Logo Micron Technology

Principal Signal Integrity Engineer, Pathfinding

Artificial Intelligence • Hardware • Information Technology • Machine Learning
In-Office
4 Locations
45000 Employees
119K-320K Annually

Micron Technology Logo Micron Technology

Principal AI Systems Performance Engineer

Artificial Intelligence • Hardware • Information Technology • Machine Learning
In-Office
Austin, TX, USA
45000 Employees

Similar Companies Hiring

Unusual Machines, Inc. Thumbnail
Hardware • Robotics
Orlando, FL
300 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account