xAI

OT Network Engineer (Physical Infrastructure)

Posted One Month Ago

2 Locations

In-Office

Mid level

Information Technology

The Role

Design, deploy, operate, and troubleshoot highly available, low-latency networks for AI supercomputer campuses. Responsibilities include data-center and campus networking, hardware deployment, network automation, monitoring and telemetry, change management, cluster support, incident response, documentation, fiber infrastructure, segmentation, and cybersecurity compliance. The role provides on-call operational support, collaborates with infrastructure and site teams, supports campus expansions, and requires up to 20% travel, physical site work, and occasional extended hours.

Summary Generated by Built In

SpaceXAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All employees are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

ABOUT THE ROLE:

SpaceXAI is looking for an exceptional network engineer with experience in mission-critical or large-scale production environments to support our supercomputer infrastructure. As a member of the Physical Infrastructure Networking team, you will provide operational and designs services for physical infrastructure networks. The ideal candidate thrives in intense, high flux environments, brings a strong sense of urgency balanced with operational excellence, communicates effectively, and demonstrates high levels of technical acumen.
RESPONSIBILITIES:

Design and implement highly available networks, carefully balancing various routing and redundancy technologies depending on unique requirements between production and launch.
Design and maintain mission data center and site networks in accordance with company network standards. Collaborating with adjacent infrastructure and enterprise teams.
Evaluate, procure, and deploy network hardware including switches, firewalls, multiplexing and SDWAN appliances.
Contribute to ever maturing network automation tooling; implementing configuration analysis, linting, and scalability into the deployment framework.
Plan and coordinate network outage windows with stakeholders to perform software updates, hardware refreshes, and general network maintenance (sometimes on weekends and evenings).
Troubleshoot and resolve network-related issues, publishing root cause analysis (RCA) documentation and hosting retrospective data reviews.
Provide direct networking support during launch and test campaigns and be on-call or provide real-time support as networking responsible engineer during operations.
Proactively tailors network monitoring to provide ever more valuable operational telemetry with emphasis on issue detection prior to operational impact.
Continuously create and update network documentation, including architecture overviews, design drawings, and operational procedures.
Collaborate with cross-functional teams to proactively identify and resolve potential technical issues with network designs, especially in regards to systemic, cascading failure modes, and superficial redundancy.
Perform job walks with customers, vendors, and contractors to gather network requirements and create implementation plans.
Ensure networks are configured and maintained in compliance with industry and cybersecurity standards (e.g., ITAR, ISO, NIST), with particular attention to controls network segmentation and security.
Design and validate resilient, scalable multi-site OT network topologies optimized for critical facilities.
Audit, and enforce zero-trust firewalls to decouple facility infrastructure from the corporate DMZ.
Author comprehensive system/network architecture drawings, IP/VLAN schemes, and data flow diagrams.
Support on-site system deployment, physical network cutovers, and new site commissioning.

BASIC QUALIFICATIONS:

Bachelor’s degree in computer science, computer engineering, or other STEM discipline and 3+ years of professional network engineering experience;
- OR 5+ years of professional network engineering experience in lieu of a degree.
2+ years of hands-on experience in Operational Technology (OT) networking, critical facility infrastructure, industrial automation, or controls engineering.
Extensive hands-on experience designing, deploying, supporting, and troubleshooting Layer 2 and Layer 3 networks in latency sensitive and/or industrial environments.
Functional experience with multiple network vendors either in production or lab environments.
Experience with GitOps and Infrastructure as Code Frameworks, both as user and contributor.

PREFERRED SKILLS AND EXPERIENCE:

Strong understanding of the OSI model and network standards.
Hands-on experience with Cisco, Arista, and Juniper data center class switches.
Experience with Wave Division Multiplexing (WDM) and fiber optics infrastructure.
Extensive knowledge of common enterprise and media protocols.
Experience with switch port security features, network segmentation, QoS, multicast, and redundancy protocols.
Familiarity with network monitoring tools and layer 1 testing tools.
Proficiency in scripting (Bash/PowerShell/Python) and automation frameworks (Terraform, Ansible, etc.).
Linux and Windows system administration experience professionally or from labs.
Industry standard certifications such as CCNA or CCNP.
Experience supporting real-time systems, industrial control networks, or operational technology (OT) environments in aerospace, defense, energy, oil & gas, or similar high-reliability industries.
Excellent communication skills with the ability to communicate with internal/external customers, vendors, and management in both formal and informal situations.

ADDITIONAL REQUIREMENTS:

Ability to work in tight quarters; physical dexterity is necessary to perform job functions.
Availability for extended hours and/or weekends as the schedule varies depending on site operational needs; flexibility is required.
Ability to provide 24x7 on-call support in emergency situations and participate in an after-hours on-call rotation.
Willingness to travel (up to 20%, including flying to other sites worldwide) and potential time at sea supporting marine systems.
Ability to lift 30 lbs.
Ability to work at heights.
Ability to drive (active valid driver’s license).

SpaceXAI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.

Skills Required

Bachelor’s degree in computer science, computer engineering, or another STEM discipline and 3+ years of professional network engineering experience
Alternatively, 5+ years of professional network engineering experience in lieu of a degree
Hands-on experience designing, deploying, supporting, and troubleshooting Layer 2 and Layer 3 networks in latency-sensitive, industrial, or data-center environments
Experience with multiple network vendors in production or lab environments
Experience using and contributing to GitOps and Infrastructure as Code frameworks
Understanding of the OSI model and network standards
Hands-on experience with Cisco, Arista, Juniper, or NVIDIA Spectrum-X data-center switches
Experience with RoCEv2 Ethernet AI/HPC fabrics; InfiniBand experience is a plus
Knowledge of AI training and inference traffic patterns, collectives, congestion, ECMP, adaptive routing, or NCCL
Experience with WDM, large-scale fiber plants, OTDR, and acceptance testing
Experience with port security, network segmentation, QoS, multicast, and redundancy protocols
Familiarity with network monitoring and Layer 1 test tools, including operational telemetry and dashboards
Proficiency in Bash, PowerShell, or Python and automation frameworks such as Terraform or Ansible
Professional or laboratory Linux and Windows system administration experience
Industry-standard certification such as CCNA or CCNP
Experience supporting real-time systems, industrial control or OT networks, or high-reliability environments
Excellent communication skills with customers, vendors, management, and internal teams
Ability to pass applicable background checks for site access
Ability to work in tight quarters, at heights, and with required physical dexterity
Ability to lift 30 pounds
Active valid driver’s license and ability to drive
Availability for extended hours, weekends, 24x7 emergency support, and an after-hours on-call rotation
Willingness to travel up to 20% between supercomputer campuses and related sites