Software Engineer, Collective Communication and Network Performance

Posted Yesterday
Be an Early Applicant
2 Locations
In-Office
173K-338K Annually
Senior level
Artificial Intelligence • Computer Vision • Hardware • Robotics • Metaverse
The Role
Develop production software that detects and diagnoses collective communication and network performance issues across large-scale AI and HPC GPU systems. Build telemetry analysis infrastructure, correlate NCCL, RDMA, NIC, switch, and topology data, reproduce failures, maintain regression tests, and expand detection across communication paths. Collaborate across teams to define requirements, validate diagnoses, and deliver reliable software.
Summary Generated by Built In

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It's a unique legacy of innovation that's fueled by great technology—and amazing people. Today, we're tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what's never been done before takes vision, innovation, and the world's best talent. As an NVIDIAN, you'll be immersed in an inclusive, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.


NVIDIA is looking for a Software Engineer, Collective Communication and Network Performance to join our outstanding team in Santa Clara, CA. We develop software tools that help technical users understand complex computing systems. We are building software that automatically detects and diagnoses collective communication and network performance problems in AI workloads across thousands of GPUs. This is an outstanding opportunity to turn deep knowledge of NCCL and the network fabric into production software with direct impact on the largest AI systems in the world.


What you'll be doing:

  • Develop detection algorithms that identify communication performance problems from always-on telemetry. Distinguish the causes: a late rank, a degraded GPU, fabric congestion, a silent NIC fault, a slower path chosen over NVLink, or a hang.
  • Build the analysis infrastructure these algorithms run on. Ingest per-collective records from many ranks, correlate them with network counters and topology, and produce diagnoses with supporting evidence.
  • Define the data the detectors need from NCCL profiler and tuner plugin interfaces, RDMA transport state, and NIC and switch counters. Collaborate with the owning teams to deliver it at production-acceptable overhead.
  • Reproduce communication problems on internal clusters and verify that automated diagnoses match the known cause. Maintain these cases as regression tests that gate every change.
  • Extend coverage beyond NCCL to point-to-point and other communication paths, guided by evidence from real incidents.

What we need to see:

  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
  • 8+ years of relevant software engineering experience, including hands-on work on a collective communication library, an RDMA transport, or network performance for GPU or HPC workloads.
  • Working knowledge of how NCCL or a comparable library operates: algorithms, protocols, channels, transports and topology detection, and how each affects latency and bandwidth.
  • Working knowledge of RDMA networking in practice: InfiniBand or RoCE, NIC and switch counters, congestion control, and the tools used to measure them.
  • Demonstrated experience diagnosing a multi-node communication problem from measurements, proving the cause, and turning that diagnosis into software used by others.
  • Proven history of landing changes in code owned by other teams, from defining requirements to delivery.
  • Strong debugging skills across component boundaries, with changes validated through code review, tests and independent measurements, including when using automated development tools.
  • Proficiency in C++ and Python, and clear communication of evidence, uncertainty and next steps to engineers and users.

Ways to stand out from the crowd:

  • Contributions to NCCL, UCX, libfabric, NVSHMEM or a comparable communication library, or to their profiling and tuner plugins.
  • Experience with point-to-point and expert-parallel communication paths such as NIXL, DeepEP or NVSHMEM, or with fabrics such as EFA.
  • Experience building detection or root-cause analysis that ran in production, with insight into its precision and recall.
  • Experience operating or debugging training or inference jobs above a thousand GPUs, including hang localization.

Join NVIDIA and help build the software that keeps the world's largest AI systems running at full speed!


#FleetGPUProfiling

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary range for this position is 172,500 USD – 338,000 USD.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until November 21, 2026.

This posting is for an existing vacancy. 

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Skills Required

  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience
  • 8+ years of relevant software engineering experience
  • Hands-on experience with a collective communication library, RDMA transport, or network performance for GPU or HPC workloads
  • Working knowledge of NCCL or a comparable communication library, including algorithms, protocols, channels, transports, and topology detection
  • Working knowledge of RDMA networking, InfiniBand or RoCE, NIC and switch counters, congestion control, and network measurement tools
  • Experience diagnosing multi-node communication problems from measurements and implementing software-based diagnoses
  • Experience delivering code changes owned by other teams, from requirements definition through delivery
  • Strong debugging skills across component boundaries, with validation through code review, testing, and independent measurements
  • Proficiency in C++ and Python
  • Clear communication of evidence, uncertainty, and next steps
  • Contributions to NCCL, UCX, libfabric, NVSHMEM, comparable communication libraries, or profiling and tuner plugins
  • Experience with point-to-point or expert-parallel communication paths such as NIXL, DeepEP, or NVSHMEM, or fabrics such as EFA
  • Experience building production detection or root-cause analysis systems, including precision and recall evaluation
  • Experience operating or debugging training or inference jobs involving more than 1,000 GPUs

NVIDIA Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about NVIDIA and has not been reviewed or approved by NVIDIA.

  • Equity Value & Accessibility — Feedback suggests equity grants, stock options/RSUs, and a discounted ESPP materially boost total compensation and can deliver significant wealth-building when the stock performs well.
  • Healthcare Strength — Feedback suggests medical, dental, and vision plans with in-network preventive care at 100%, telehealth, and concierge-style support are comprehensive and valued.
  • Parental & Family Support — Feedback suggests up to 22 weeks fully paid parental leave, fertility coverage, and adoption/surrogacy reimbursement provide strong family-focused benefits.

NVIDIA Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Santa Clara, CA
21,960 Employees
Year Founded: 1993

What We Do

NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern AI — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as “the AI computing company.”

Similar Jobs

Hilton Logo Hilton

Banquet Manager- Hilton Austin

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office
Austin, TX, USA
500000 Employees

Hilton Logo Hilton

Beverage Server- Hilton Americas Houston

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office
Houston, TX, USA
500000 Employees

Cloudflare Logo Cloudflare

Account Director

Cloud • Information Technology • Security • Software • Cybersecurity
Hybrid
4 Locations
4400 Employees
313K-449K Annually

Cloudflare Logo Cloudflare

Senior Systems Engineer

Cloud • Information Technology • Security • Software • Cybersecurity
Hybrid
4 Locations
4400 Employees
185K-254K Annually

Similar Companies Hiring

Unusual Machines, Inc. Thumbnail
Hardware • Robotics
Orlando, FL
300 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
Vega Thumbnail
Artificial Intelligence • Automotive • Insurance • Transportation
US
65 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account