We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Skills Required
- Experience orchestrating distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline)
- Experience optimizing networking (InfiniBand/RDMA) and memory management for large-scale training
- Experience automating checkpointing and failure recovery for long-running training jobs
- Experience managing SLURM or Kubernetes-based GPU clusters
- Strong systems engineering background with C++, CUDA, and Python
What We Do
Hyphen Connect is a Web3 and AI talent agency and crypto-integrated software solutions provider that connects blockchain, DeFi, NFT, and AI companies with specialized technical and go-to-market talent globally and remotely. They deliver headhunting, data-driven research, and recruitment services across infrastructure, exchanges, gaming, and DeFi projects, plus industry analysis and hiring insights to help clients build engineering and product teams.
