ASRC Federal is searching for a Staff HPC Engineer to support ASRC Federal Technology Solution out of Manhattan, KS. ASRC Federal Technology Solution proudly supports the USDA Animal and Plant Health Inspection Service (APHIS) Marketing and Regulatory Programs – Information Technology Division (MRP-ITD). The USDA APHIS mission requires rapid, secure, and reliable access to large-scale scientific data for genomics, diagnostics, modeling, and analytics to support decision-making, research, and emergency outbreak responses.
The Scientific Computing (SciComp) HPC system and the Marketplace for Data Analytics (MDA) form the computational, data analytics, and data-sharing backbone for APHIS’s laboratory network as well as other MRP Program areas. These systems facilitate high-speed data transfer, AI modeling, data visualization, and reproducible scientific outcomes through the integration of on-premises HPC clusters, cloud-based analytics platforms, and secure data transfer via Internet2 (I2). Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement and assimilation of the industry’s best practices.
Summary: The successful candidate will be an active supporting member of the ASRC Federal team. The HPC Systems Engineer/Administrator is responsible for the administration, maintenance, optimization, and lifecycle management of a high-performance computing (HPC) environment supporting scientific research. This position ensures the availability, security, and performance of compute, storage, networking, and authentication infrastructure while providing technical support to USDA APHIS researchers and collaborating with on-site facilities, networking, and cybersecurity teams.
Primary Roles, Duties and ResponsibilitiesHPC Infrastructure Administration
- Administer and maintain a Linux-based HPC environment consisting of compute, login, management, and GPU nodes.
- Manage Slurm workload manager configuration, partitions, queues, scheduling policies, and resource allocation.
- Perform operating system installation, provisioning, upgrades, and lifecycle management.
- Develop and maintain automation using Ansible, shell scripting, and other administrative tools (NVIDIA Base Command Manager).
- Monitor overall cluster health, capacity, utilization, and performance.
- Administer and optimize a multi-petabyte Lustre parallel file system (HPE E1000).
- Configure and maintain storage quotas, project directories, tiering policies, and data placement strategies.
- Troubleshoot Lustre performance, metadata operations, changelog processing, and client connectivity.
- Monitor storage utilization, capacity growth, and system performance using enterprise monitoring platforms (Starfish Storage).
- Administer and troubleshoot NVIDIA InfiniBand fabrics, including Subnet Manager, routing, and multi-rail configurations.
- Maintain NVIDIA Base Command Manager and Cumulus Linux infrastructure supporting the HPC environment.
- Optimize MPI, UCX, NCCL, and RDMA communications for distributed scientific workloads.
- Support Internet2 connectivity and high-performance data transfers (Globus Data Transfer Nodes).
- Support NVIDIA GPU compute nodes and associated software stacks.
- Install, configure, and maintain CUDA, NCCL, GPU drivers, and related scientific software.
- Benchmark and validate GPU performance following infrastructure changes.
- Install, configure, and maintain scientific software packages and development environments.
- Build and install software from source using Make, CMake, Ninja, and other build systems.
- Configure and troubleshoot software compilation using GCC, G++, and other compiler toolchains.
- Resolve library dependencies, compiler compatibility issues, and software linkage problems.
- Develop and maintain environment modules supporting multiple software versions.
- Support Conda, R, Apptainer, compilers, MPI implementations, CUDA toolkits, and GPU-enabled applications (Apptainer and Singularity containers).
- Assist researchers with software deployment, optimization, and troubleshooting.
- Administer authentication and authorization services including Active Directory integration, LDAP, SSSD, Kerberos, and single sign-on technologies.
- Coordinate operating system patching and security updates.
- Support vulnerability remediation and infrastructure hardening activities.
- Maintain least-privilege access controls and security best practices.
- Evaluate system performance using industry-standard benchmarks.
- Analyze CPU, GPU, memory, storage, and network performance bottlenecks.
- Optimize NUMA configuration, processor affinity, MPI communication, and storage performance for scientific workloads (bioinformatics).
- Validate infrastructure performance following hardware or software changes.
- Develop and maintain standard operating procedures, technical documentation, and knowledge base articles.
- Participate in change management processes and maintain accurate operational documentation.
- Coordinate maintenance activities and production system changes.
- Support disaster recovery planning, backup strategies, and infrastructure lifecycle planning.
- Provide advanced technical support to researchers and scientific staff.
- Collaborate with networking, storage, cybersecurity, and vendor support teams to resolve complex technical issues.
- Mentor junior technical staff through knowledge transfer, collaborative troubleshooting, and hands-on training.
- Participate in procurement planning, hardware evaluations, and technology modernization initiatives.
Typical Daily Activities
- Monitor cluster health and respond to system alerts.
- Troubleshoot compute, storage, network, and authentication issues.
- Deploy operating system, firmware, kernel, and software updates.
- Optimize HPC performance for scientific workloads.
- Install and validate scientific software requested by researchers.
- Develop automation to improve operational efficiency.
- Maintain system documentation and operational procedures.
- Coordinate infrastructure maintenance and change activities.
- Assist users with job scheduling, software execution, and performance optimization.
- Work with vendors to diagnose and resolve hardware and software issues.
Requirements:
- Bachelor’s degree in computer science or related field
- Strong computer science background with in-depth systems-level knowledge in operating systems and networking
- 5+ years demonstrated competency in provisioning and configuration management of medium-sized Linux environments using tools such as RHEL Satellite, Ansible, Base Commander
- Ability to work with system configuration management tools (e.g., Puppet, Ansible) and revision control software such as CVS, SVN, Git
- Superior scripting skills and excellent attention to detail; proficiency in at least Python, Perl, or Bash
- Demonstrated troubleshooting skills in a complex environment beyond systems, from networks to other dependent systems
- Excellent customer service skills and attention to detail supporting hundreds of scientific users
- Excellent communication and people skills; excellent time management and organizational skills
Preferred Qualifications:
- Experience with Base Commander or other orchestration platforms
- Previous experience in support of HPC environments and complex scientific computing centers
- Familiarity/proficiency with OpenMP and Message Passing Interface (MPI) programming
- Experience with Lustre and InfiniBand
- Experience with Slurm
#USDA,#HPC,#LINUX,#RHEL,#ANSIBLE,#BaseCommander,#GPU,#Slurm
Skills Required
- Bachelor's degree in computer science or a related field
- Strong computer science background with in-depth systems-level knowledge of operating systems and networking
- 5+ years of experience provisioning and managing medium-sized Linux environments using tools such as RHEL Satellite, Ansible, or Base Commander
- Experience with configuration management tools such as Puppet or Ansible
- Experience with revision control software such as CVS, SVN, or Git
- Proficiency in at least one of Python, Perl, or Bash
- Demonstrated troubleshooting skills across systems, networks, and dependent infrastructure
- Excellent customer service skills supporting scientific users
- Excellent communication, people, time management, and organizational skills
- Experience with Base Commander or other orchestration platforms
- Previous experience supporting HPC environments and complex scientific computing centers
- Familiarity or proficiency with OpenMP and MPI programming
- Experience with Lustre and InfiniBand
- Experience with Slurm
ASRC Federal Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about ASRC Federal and has not been reviewed or approved by ASRC Federal.
-
Healthcare Strength — Health coverage includes three Cigna medical plans (two PPOs and one HDHP) with dental, vision, HSA/FSA, and regional options such as Kaiser and HMSA. Feedback suggests these choices are competitive and comprehensive for a government contractor.
-
Retirement Support — The 401(k) plan features an employer match that is immediately vested. Feedback suggests this structure provides strong, dependable retirement value.
-
Leave & Time Off Breadth — Paid time off is complemented by nine core holidays, two floating holidays, and paid parental, military, jury, and bereavement leave. Feedback suggests this breadth supports work-life needs across varied situations.
ASRC Federal Insights
What We Do
ASRC Federal’s family of companies help federal civilian, defense and intelligence agencies achieve successful mission outcomes and elevated performance. We are dedicated to building an enduring enterprise focused on our people, customers and shareholders.
