Senior Software Engineer, RL Post-Training Frameworks

Posted 7 Days Ago
Hiring Remotely in Santa Clara, CA, USA
In-Office or Remote
184K-357K Annually
Senior level
Artificial Intelligence • Computer Vision • Hardware • Robotics • Metaverse
The Role
Design and build RL post-training infrastructure for AI, optimizing training-inference-rollout on distributed systems while ensuring fault tolerance and scalability.
Summary Generated by Built In

Reinforcement learning post-training is driving some of the most significant capability gains in AI today. It is the process that teaches a model to reason through hard problems, follow complex instructions, and act as an autonomous agent. It is also one of the hardest infrastructure challenges in the field. RL requires inference, rollout generation, and training running in a continuous loop. The rollout step is what makes it hard: the model must interact with environments, tools, and other models to produce the signal that drives learning. Coordinating actor, critic, and reward models across heterogeneous hardware at scale pushes the limits of what distributed systems can do.

NVIDIA is building an RL Frameworks engineering team to develop the open-source tools and infrastructure that AI researchers and post-training teams depend on. The team spans the full software stack, from collaborating closely with the researchers and labs pushing the frontier, to contributing to RL frameworks like VeRL, Miles, and TorchTitan, to improving the distributed runtimes they depend on, including Ray and Monarch. Whether your strength is working with researchers to understand and address their need optimizing deep learning frameworks, or building distributed infrastructure, we want to hear from you. Come join us to build the systems that enable the next generation of AI.

What you will be doing:

You will architect and build RL post-training infrastructure that scales efficiently from experimentation on a single GPU to production across thousands of nodes. This means tuning RL training-inference-rollout loops on GPUs, CPUs, and LPUs for performance where it matters, contributing to and improving the performance and usability of open-source RL frameworks, and partnering with the teams who own them. The role also spans fault tolerance, elastic scaling, and fast restarts so long-running distributed training jobs survive failures, stragglers, and resource contention.

Beyond GPU-accelerated training, this work includes partnering with teams building CPU-driven rollout workloads, including tool-use, code execution, and agentic environments, supplying the systems and framework engineering needed to run them efficiently alongside GPU- or LPU-accelerated generation and GPU-accelerated training. It also means advocating for researcher and partner needs with NVIDIA's networking, math library, and compiler teams so the capabilities RL workloads require get prioritized and delivered, and working with hardware teams to take advantage of next-generation hardware capabilities in post-training workloads.

What we need to see:

  • MS or PhD in Computer Science, Computer Engineering, or a related field (or equivalent experience)

  • 5+ years of professional experience in distributed systems, high-performance computing, deep learning infrastructure, or ML systems engineering

  • Strong proficiency in Python and C/C++

  • Demonstrated experience building or contributing to large-scale distributed systems or runtime frameworks in production at a frontier AI lab, hyperscaler, or major technology company

  • Strong verbal and written communication skills and the ability to collaborate across organizational and geographic boundaries

Depth in one or more of the following technical areas:

  • Reinforcement learning for LLM post-training (RLHF, PPO, GRPO, DPO, reward modeling), including how algorithms map to distributed execution and the systems challenges they create (heterogeneous placement, rollouts, environment execution, resharding between training and generation)

  • PyTorch internals, including distributed training primitives (FSDP, tensor parallelism, pipeline parallelism) and their composition

  • Kubernetes runtime internals (container lifecycle, pod scheduling, resource quotas, GPU allocation)

  • End-to-end distributed systems design (service boundaries, data flows, consistency models, failure modes, recovery approaches)

Experience in any of the following areas is a plus:

  • Deep expertise in networking (NCCL, NVLink, InfiniBand), advanced multi-dimensional parallelisms (Megatron-LM, FSDP2, TP/DP/PP, MoE), or memory optimizations (quantization-aware training, mixed precision)

  • Experience integrating high-performance inference engines (vLLM, SGLang, TensorRT-LLM) into RL training loops for GPU-accelerated rollout

  • Strong background in actor- and task-based distributed programming (Ray, Monarch, or comparable systems)

  • Familiarity with multi-turn training, multi-agent co-evolution, or VLM post-training

Ways to stand out from the crowd:

  • Open-source contributions to RL post-training or distributed training projects (e.g., VeRL, Miles, TorchTitan, OpenRLHF, NeMo-Aligner, DeepSpeed-Chat), including significant work on framework internals where applicable

  • Kubernetes work beyond routine operations (custom operators, GPU device plugins, or scheduling contributions)

  • Direct experience operating frontier-scale training (RL post-training at thousands of GPUs and/or large-scale LLM or multimodal pre-training)

  • Hands-on experience with production distributed failures at scale (stragglers, resource contention, hardware faults)

Widely considered to be one of the technology world’s most desirable employers, NVIDIA offers highly competitive salaries and a comprehensive benefits package. As you plan your future, see what we can offer to you and your family www.nvidiabenefits.com/

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until April 27, 2026.

This posting is for an existing vacancy. 

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Santa Clara, CA
21,960 Employees
Year Founded: 1993

What We Do

NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern AI — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as “the AI computing company.”

Similar Jobs

Huntress Logo Huntress

Principal Threat Intelligence Analyst

Information Technology • Cybersecurity
Easy Apply
Remote
United States of America
630 Employees
200K-225K Annually

Affirm Logo Affirm

Compliance Exam Lead

Big Data • Fintech • Mobile • Payments • Financial Services
Easy Apply
Remote
United States
2200 Employees
115K-170K Annually

SoFi Logo SoFi

Member Service Representative

Fintech • Mobile • Software • Financial Services
Easy Apply
Remote or Hybrid
United States
4500 Employees
16-20 Hourly

Pfizer Logo Pfizer

Senior Oncology Account Specialist TIVDAK Orange County CA

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Remote or Hybrid
3 Locations
121990 Employees
109K-251K Annually

Similar Companies Hiring

Idler Thumbnail
Artificial Intelligence
San Francisco, California
6 Employees
Fairly Even Thumbnail
Hardware • Other • Robotics • Sales • Software • Hospitality
New York, NY
30 Employees
Bellagent Thumbnail
Artificial Intelligence • Machine Learning • Business Intelligence • Generative AI
Chicago, IL
20 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account