Essential Skills
- HPC & AI Infrastructure:
- Expert-level knowledge of NVIDIA Base Command Manager (BCM) or Metal-as-a-Service (MaaS) provisioning tools.
- Deep understanding of Slurm configuration (cgroups, plugin development, accounting).
- Proficiency with NVIDIA DGX/HGX hardware architecture and the associated software stack (Drivers, CUDA, DCGM).
- Linux & Automation (DevOps for Hardware):
- Mastery of Red Hat Enterprise Linux (RHEL) / Ubuntu internals (Systemd, Kernel Tuning, Hugepages).
- Advanced proficiency in Ansible (writing custom modules/roles) and Python (automating admin tasks).
- Experience with Git workflows (Branching, PRs, CI/CD).
- Containerisation:
- Hands-on experience with Docker, Singularity/Apptainer, and Kubernetes (specifically NVIDIA GPU Operator and NVIDIA Network Operator).
Desirable Experience
- Network Awareness: Ability to troubleshoot basic InfiniBand/RoCEv2 issues (ibstat, perf query) to distinguish between a "Node Issue" and a "Network Issue."
- Storage Integration: Experience mounting high-performance parallel file systems (VAST/Lustre/WEKA/GPFS) and tuning client-side performance.
- Certifications:
- NVIDIA Certified Associate - AI in the Data Center.
- Red Hat Certified Engineer (RHCE).
- CKA (Certified Kubernetes Administrator).
Success Metrics (KPIs)
- Deployment Velocity: Reduction in "Time-to-Hello-World" (time from power-on to running the first successful GPU job) for new clusters.
- Code Quality: >95% of Pull Requests pass automated linting and require fewer than 2 review cycles before merge.
- Stability: Zero "Configuration Drift" incidents in production (e.g., manual changes breaking the cluster) due to strict IaC enforcement.
#LI-RB1
Preferred QualificationsAbout the Role
The Senior HPC Engineer is the "Foreman" of the “AI Factory”. While the Domain Architect defines the architectural vision, you are responsible for the hands-on build and deployment. You act as the Technical Squad Lead for our offshore engineering teams, bridging the gap between the onshore architectural vision and the hands-on execution.
As a System Integrator, we thrive on velocity and precision. You will not just "maintain" clusters; you will lead the automated deployment of NVIDIA SuperPOD and BasePOD infrastructure for global enterprise clients. You are the "Lieutenant" to the Domain Architect, translating High-Level Designs (HLDs) into executable Ansible playbooks and ensuring your squad of HPC Engineers delivers defect-free infrastructure.
In this role, you are 100% Delivery-Focused, split between Technical Leadership (40%) and Hands-on Engineering (60%). You are the escalation point for complex kernel panics, the guardian of our Infrastructure-as-Code (IaC) repository, and the mentor who unblocks junior engineers when a Slurm job fails to schedule.
Key Responsibilities
- Hands-on Engineering & Automation (60%)
- Cluster Provisioning Factory:
- Lead the deployment of NVIDIA Base Command Manager (BCM) (formerly Bright Cluster Manager) to provision bare-metal DGX/HGX nodes at scale.
- Develop and maintain the Ansible / Terraform library used to configure OS settings, user authentication (LDAP/AD), and storage mounts across hundreds of nodes.
- Execute HPL (High-Performance Linpack) and NCCL-tests to validate cluster performance, tuning BIOS and OS parameters to hit "Gold Standard" benchmarks.
- Scheduler & Workload Orchestration:
- Configure complex Slurm Workload Manager policies, including Fair Share, Preemption, and GPU Partitioning (MIG).
- Integrate Kubernetes-based orchestrators (e.g., NVIDIA Base Command, Run:AI, or Red Hat OpenShift) with the underlying HPC hardware.
- Deep-Dive Troubleshooting:
- Debug "Silent Data Corruption" and "Xid Errors" on GPUs, analysing nvidia-smi logs and kernel message buffers (dmesg).
- Diagnose fabric-related performance drops (e.g., Identifying a specific flapping link causing global slowdowns) in collaboration with the Network Squad.
- Squad Leadership & Quality Assurance (40%)
- Technical Direction (The "Foreman"):
- Translate the Low-Level Design (LLD) provided by the Domain Architect into granular Jira tasks for your squad of HPC Engineers.
- Conduct daily stand-ups to unblock engineers, clarifying requirements and making technical decisions on the fly (e.g., "Use Ansible roles, not shell scripts for this").
- Code Quality & Governance:
- Act as the Primary Gatekeeper for the code repository. Perform mandatory Code Reviews on all Pull Requests (PRs) to ensure idempotency and error handling.
- Enforce "Config-as-Code" discipline, ensuring no manual changes are made to production clusters without a committed playbook.
- Mentorship:
- Guide mid-level and junior engineers on best practices for Linux Systems Administration and HPC environments.
Skills Required
- Expert knowledge of NVIDIA Base Command Manager or Metal-as-a-Service provisioning tools
- Deep understanding of Slurm configuration, including cgroups, plugins, and accounting
- Proficiency with NVIDIA DGX or HGX hardware and drivers, CUDA, and DCGM
- Mastery of Red Hat Enterprise Linux or Ubuntu internals, including systemd, kernel tuning, and hugepages
- Advanced Ansible proficiency, including custom modules and roles
- Python proficiency for administrative automation
- Experience with Git workflows, branching, pull requests, and CI/CD
- Hands-on experience with Docker, Singularity or Apptainer, and Kubernetes
- Experience with NVIDIA GPU Operator and NVIDIA Network Operator
- Ability to troubleshoot InfiniBand or RoCEv2 issues
- Experience mounting and tuning VAST, Lustre, WEKA, or GPFS parallel file systems
- NVIDIA Certified Associate certification in AI in the Data Center
- Red Hat Certified Engineer certification
- Certified Kubernetes Administrator certification
- Experience deploying NVIDIA SuperPOD and BasePOD infrastructure
- Experience configuring Slurm Fair Share, preemption, GPU partitioning, and MIG
- Experience translating low-level designs into Ansible playbooks and leading engineering squads
World Wide Technology Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about World Wide Technology and has not been reviewed or approved by World Wide Technology.
-
Healthcare Strength — Health insurance is considered excellent and affordable, with bundled medical, dental, and vision coverage that reduces out-of-pocket costs. Wellness programs and an Employee Assistance Program further enhance access to care.
-
Leave & Time Off Breadth — Generous paid time off grows with tenure, alongside paid sick days and holidays. Parental and military leave are also provided.
-
Retirement Support — Financial benefits include profit sharing and a 401(k) with company matching, complemented by life and disability insurance and tuition reimbursement. These elements strengthen total rewards beyond base salary.
World Wide Technology Insights
What We Do
World Wide Technology is a systems integrator, provides information technology and supply chain solutions. Fueled by creativity and ideation, World Wide Technology strives to accelerate our growth and nurture future innovation. From our world class culture, to our generous benefits, to developing cutting edge technology solutions, WWT constantly works towards its mission of creating a profitable growth company that is a great place to work. We encourage our employees to embrace collaboration, get creative and think outside the box when it comes to delivering some of the most advanced technology solutions for our customers. At a glance, WWT was founded in 1990 in St. Louis, Missouri. We employ over 9,000 individuals and closed nearly $17 Billion in revenue. We have an inclusive culture and believe our core values are the key to company and employee success. WWT is proud to announce that it has been named on the FORTUNE "100 Best Places to Work For®" list for the 12th consecutive year!
Why Work With Us
Our extensive partnership with best in class technology companies, coupled with our strong culture allow for world class delivery of transformative business solutions driven by IT.
.jpeg)
