Senior Engineer - AI and HPC Observability

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Superclusters (MARS) builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you’ll help design solutions that power some of the world’s most advanced computing workloads.

Observability is at the heart of this transformation. We are looking for a Senior AI & HPC Observability Engineer to design and build the next-generation observability platform for large-scale AI workloads, GPU clusters, and high-performance computing environments. This role blends deep technical engineering with large-scale data systems and developing scalable telemetry pipelines, AI-driven insights, and intelligent monitoring across NVIDIA’s world-class GPU infrastructure.

What You Will Be Doing:

• Design and implement full-stack observability systems covering metrics, logs, traces, and events for GPU-powered AI and HPC workloads.

• Build large-scale telemetry data pipelines leveraging OpenTelemetry, Kafka, Prometheus, and other distributed systems to ingest, process, and analyze massive data streams.

• Develop analytics and anomaly detection frameworks to enable real-time visibility, performance optimization, and predictive insights across multi-tenant environments.

• Architect and tune high-throughput data stores (e.g., TSDBs, columnar databases, OLAP systems) for large-scale observability data.

• Drive self-service analytics capabilities through APIs, dashboards, and recommendation engines that empower developers and operators with actionable insights.

• Collaborate with AI platform, GPU, and cloud infrastructure teams to optimize observability for model training, inference workloads, and HPC performance.

• Leverage machine learning and statistical techniques for correlation, anomaly detection, and intelligent alerting.

• Contribute to performance tuning, scalability, and reliability of observability services across on-prem, and cloud environments.

What We Need To See:

• BS or equivalent experience in Computer Science, Computer Engineering, or a related technical field.

• 8+ years of experience in large-scale observability, data engineering, or performance monitoring systems.

• Proven expertise in building and scaling observability stacks (metrics, logs, traces, events) using OpenTelemetry, Prometheus, Grafana, or Thanos.

• Deep understanding of data collection, transformation, and storage at scale, experience with streaming frameworks (Kafka, Flink, Spark) preferred.

• Hands-on experience with Python, Go, and/or Java for backend development and automation.

• Strong knowledge of API design, data modeling, SQL/NoSQL, and data pipeline architecture.

• Experience working with PromQL, time-series databases, and large-scale monitoring systems.

• Familiarity with AI/ML pipelines, GPU-based workloads, and HPC environments.

• Experience with anomaly detection, log analytics, and recommendation systems using ML or statistical techniques.

• Excellent problem-solving, debugging, and performance-tuning skills in distributed systems.

Ways To Stand Out from The Crowd:

• Proven experience designing and scaling full-stack observability platforms for large-scale AI, GPU, or HPC environments.

• Hands-on expertise with OpenTelemetry, Prometheus, Kafka, and distributed data pipelines handling high-volume telemetry streams.

• Strong background in data engineering, performance tuning, and time-series data modeling for real-time analytics.

• Demonstrated use of machine learning or statistical techniques for anomaly detection, correlation, or intelligent alerting.

• Deep understanding of API design, self-service observability, and building platforms that empower internal developers and operators.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.Applications for this job will be accepted at least until October 24, 2025.NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.