NVIDIA is looking for a Day 2 Site Reliability Engineer to join the EMEA NVIDIA Cloud Partner (NCP) Day 1 / Day 2 Operations team. EMEA NCP fleets are scaling towards a million deployed GPUs, and this role exists to keep those fleets running at maximum goodput once they are in production — owning fault detection and remediation, preventive maintenance, and proactive firmware and field-notice rollout across partner estates.
We are looking for a hands-on reliability engineer who thinks in SLOs, error budgets and blast radius rather than tickets. You will measure and drive MTBI and job goodput on multi-thousand-GPU clusters, automate the detection-to-remediation loop, and work directly with NCP operations teams, NVEX and TAMs to review incidents and escalations and turn each one into a permanent fix.
NCP estates are open-source-first, home-grown and heterogeneous — upstream Kubernetes, KubeVirt, Slurm, Prometheus/Grafana, Cumulus/SONiC and a long tail of ISV and in-house tooling — so this role is deliberately tool-agnostic: you will meet each partner on the stack they actually run rather than on a single proprietary product, and you will keep GPU capacity generating revenue instead of burning cost in reactive support.
What You’ll Be Doing:
Own Day 2 production stability and reliability at fleet scale across EMEA NCP partners: define, measure and improve MTBI and job goodput on multi-thousand-GPU clusters, and hold the error budget for the platforms you support.
Build and operate the fault detection to remediation loop—GPU, node, fabric and storage fault signatures, automated triage, node drain, cordon, reset and return-to-service workflows—so failures are caught and cleared before they cost training jobs.
Run preventive maintenance and proactive firmware and field-notice rollout campaigns across partner fleets (e.g. CPLD and switch firmware levels), including risk assessment, staged rollout, maintenance-window planning and verification at scale.
Own observability and reliability tooling for GPU fleets: Prometheus/Grafana/Loki, DCGM and XID diagnostics, node-level health agents such as GPUd and NVSentinel, and fleet-wide reliability intelligence—extending or integrating them into the partner’s own home-grown stack.
Lead incident response and post-incident review in collaboration with NVEX, TAM and partner operations teams: drive root-cause analysis on escalations, track corrective actions to closure, and convert every recurring incident into automation, a health check or a runbook.
Own Kubernetes node lifecycle and reliability engineering on heterogeneous open stacks—upstream Kubernetes, KubeVirt, Slurm and GPU-aware schedulers—covering node health policy, upgrade and reboot orchestration, and safe workload draining on GPU-accelerated nodes.
What We Need to See:
BS/MS/PhD in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, or related fields, or equivalent experience.
8+ years in SRE, production engineering or large-scale infrastructure operations roles.
Reliability Engineering Practice: Demonstrated ownership of production reliability at scale—SLO and error-budget definition, on-call and incident command, blameless post-incident review, and measurable improvement of availability or mean time between interruptions.
GPU & HPC Fleet Operations: Hands-on experience operating HPC/AI clusters and NVIDIA GPU-accelerated infrastructure—driver and CUDA toolkit lifecycle, firmware management, workload profiling, and troubleshooting across CPUs, GPUs and high-speed interconnects.
Kubernetes at Scale: Extensive expertise in running Kubernetes in GPU-accelerated and HPC environments—node lifecycle, upgrades and draining, scheduler behaviour, batch schedulers such as Slurm, and mixed bare-metal/virtualised (e.g. KubeVirt) multi-tenant estates.
Observability & Automation: Proficiency in Python and Bash, Infrastructure-as-Code and configuration management (e.g. Ansible, Terraform), GitOps-based fleet lifecycle and upgrade management, and observability stacks (Prometheus, Grafana, Loki) for monitoring, alerting and building fault-tolerant systems.
Linux, Storage & Networking Fundamentals: Deep knowledge of Linux (RedHat, Ubuntu), OS-level tuning and security, data centre networking fundamentals, and parallel/enterprise storage (Lustre, GPFS, ZFS, XFS) as they affect cluster stability.
Ways to Stand Out from the Crowd:
Experience defining and running proactive firmware, microcode or field-notice rollout campaigns across a large production fleet.
Familiarity with GPU health and fleet telemetry tooling—DCGM and XID diagnostics, GPUd, NVSentinel, node-level health agents and fleet-wide reliability intelligence.
Experience with the NVIDIA GPU and Network Operators for automated GPU and network resource lifecycle management in Kubernetes, and with NVIDIA Base Command Manager (BCM) for provisioning and monitoring GPU clusters at scale.
Practical background in AI-native scheduling and inference frameworks on Kubernetes (e.g. KAI, Grove, Dynamo, NVIDIA Cloud Functions) and their failure modes under load.
Expertise in RDMA-based fabrics (InfiniBand or RoCE) in HPC or AI environments. Exposure to Cumulus Linux, SONiC or Spectrum-X fabrics, DPU/DOCA infrastructure services, and NVLink/NVSwitch partition operations (NMX-C / NMX-M) on NVL72-class systems is a strong plus.
Skills Required
- Bachelor’s, master’s, or doctoral degree in Computer Science, Electrical or Computer Engineering, Physics, Mathematics, a related field, or equivalent experience
- 8+ years of experience in SRE, production engineering, or large-scale infrastructure operations
- Experience owning production reliability at scale, including SLOs, error budgets, on-call, incident command, blameless post-incident reviews, and measurable reliability improvements
- Hands-on experience operating HPC or AI clusters and NVIDIA GPU-accelerated infrastructure
- Experience with NVIDIA GPU and CUDA toolkit lifecycle, firmware management, workload profiling, and troubleshooting CPUs, GPUs, and high-speed interconnects
- Extensive experience running Kubernetes in GPU-accelerated or HPC environments, including node lifecycle, upgrades, draining, scheduler behavior, Slurm, and mixed bare-metal or virtualized estates
- Proficiency in Python and Bash
- Experience with Infrastructure as Code and configuration management, such as Ansible or Terraform
- Experience with GitOps-based fleet lifecycle and upgrade management
- Experience with Prometheus, Grafana, Loki, monitoring, alerting, and fault-tolerant systems
- Deep knowledge of Linux, including Red Hat and Ubuntu, OS tuning and security, data center networking, and enterprise or parallel storage
- Experience with proactive firmware, microcode, or field-notice rollout campaigns across large production fleets
- Familiarity with DCGM, XID diagnostics, GPUd, NVSentinel, node health agents, and fleet reliability intelligence
- Experience with NVIDIA GPU and Network Operators or NVIDIA Base Command Manager
- Experience with AI-native scheduling and inference frameworks on Kubernetes, such as KAI, Grove, Dynamo, or NVIDIA Cloud Functions
- Expertise with RDMA fabrics, including InfiniBand or RoCE, and familiarity with Cumulus Linux, SONiC, Spectrum-X, DPU/DOCA, NVLink, or NVSwitch
NVIDIA Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about NVIDIA and has not been reviewed or approved by NVIDIA.
-
Equity Value & Accessibility — Equity awards and a discounted ESPP are highlighted as core parts of total compensation, enabling employees to share in the company’s success. Stock-based compensation and the two-year lookback ESPP are consistently described as especially valuable.
-
Healthcare Strength — Health coverage is portrayed as robust, with comprehensive medical, dental, and vision options alongside mental health support and on-site care resources. Employer HSA contributions and wellness perks reinforce the depth of the offering.
-
Retirement Support — Retirement programs are depicted as strong, featuring a meaningful 401(k) match with Roth options and support for Mega Backdoor Roth contributions. These elements position long-term savings as a notable advantage of the total rewards package.
NVIDIA Insights
What We Do
NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern AI — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as “the AI computing company.”
