About the Role
XPerf makes AI compute reliable, economically efficient, and fully utilized. We validate and monitor heterogeneous GPU infrastructure to help enterprises and neo-cloud providers maximize their compute ROI.
We build two products: ClusterReady (pre-deployment validation during CI) and ClusterBeacon (real-time production observability). We work across NVIDIA, AMD, and Intel GPUs and are purpose-built for multi- vendor, air-gapped environments where observability is rare and failures are expensive.
As a Software Engineer, you will build end-to-end features that span cloud backend, node agents, and GPU workload containers. You will debug real GPU infrastructure failures, implement observability systems, and work directly with production deployments at scale.
What You'll Work On
- Own features end-to-end: problem → approach → ship → iterate.
- Implement backend APIs shaped for frontend: time-series aggregation, streaming data, real-time dashboards.
- Build AI agent workflows that automate cluster validation and diagnostics.
- Debug real failure modes in production: network issues, hardware faults, driver incompatibilities, thermal problems.
- Profile LLM inference on real GPUs, measure bottlenecks, identify optimization opportunities.
- Implement tracing, metrics, structured logging. Help determine what operators actually need to see.
- Write comprehensive tests (pytest, integration, CI/CD). Own the quality of code you ship.
What We're Looking For
You have shipped multi-component systems in production. You can reason about data models, service boundaries, and failure modes. You understand how backend services, distributed agents, and GPU workloads fit together.
Required
- Python, C++, CUDA, Bash — comfortable working across languages.
- PyTorch (including torch.compile) and either TensorRT or ONNX Runtime.
- Transformer models, LLM inference optimization, quantization techniques.
- Nsight Systems, Nsight Compute, PyTorch Profiler, or DCGM — can read traces and identify bottlenecks.
- One distributed training tool (NCCL, FSDP, DeepSpeed) and one inference engine (vLLM, SGLang, TensorRT-LLM).
- Python backend: FastAPI, async I/O, REST/gRPC. Pytest testing (async + integration).
- OpenAPI/contract-first API design. Prometheus metrics and OpenTelemetry instrumentation.
- SSE or WebSocket in production — understand streaming patterns, reconnect handling.
- Docker, Kubernetes basics, Linux systems work on NVIDIA GPUs.• Basic understanding of NVLink, InfiniBand/RoCE, node-to-node communication.
- Have debugged GPU issues and/or shipped features that mattered.
Nice to Have
- Advanced GPU optimization: Triton, CUTLASS, FlashAttention, cuDNN, XLA/TVM/MLIR.
- Multi-node scaling: Megatron-LM, NeMo, DeepSpeed, Triton Inference Server, TGI.
- Multi-vendor experience: ROCm, RCCL, AMD GPU optimization.
- Shipped an AI agent or tool used by others. MCP, LangGraph, Anthropic/OpenAI SDKs.
- Kubernetes operators, Helm, Argo CD, Slurm, batch schedulers.
- Kafka/Redis Streams, Grafana, high-volume telemetry pipelines.
- MLPerf LoadGen, benchmarking tooling.
- NVIDIA DALI, parallel filesystems, NVMe tuning, I/O optimization.
Why This Matters
Enterprises operate GPU fleets for many years. XPerf helps them extend cluster lifetimes and maximize ROI by validating hardware reliability over time. Neo-cloud providers need to deploy and validate new infrastructure fast. XPerf accelerates time-to-market with rapid multi-vendor qualification. Your work directly impacts $50M+ infrastructure investments.
Equity & benefits: Meaningful stock options. Solid health/dental/vision, flexible time off, professional development support.
Direct candidates only, no recruiters.
Skills Required
- Python, C++, CUDA, and Bash experience
- PyTorch, including torch.compile, and TensorRT or ONNX Runtime
- Transformer models, LLM inference optimization, and quantization techniques
- Experience with Nsight Systems, Nsight Compute, PyTorch Profiler, or DCGM
- Experience with one distributed training tool: NCCL, FSDP, or DeepSpeed
- Experience with one inference engine: vLLM, SGLang, or TensorRT-LLM
- Python backend development with FastAPI, async I/O, REST or gRPC
- Pytest testing, including asynchronous and integration testing
- OpenAPI and contract-first API design
- Prometheus metrics and OpenTelemetry instrumentation
- Production experience with SSE or WebSocket streaming patterns and reconnect handling
- Docker, Kubernetes basics, and Linux systems work on NVIDIA GPUs
- Basic understanding of NVLink, InfiniBand or RoCE, and node-to-node communication
- Experience debugging GPU issues or shipping impactful production features
- Advanced GPU optimization with Triton, CUTLASS, FlashAttention, cuDNN, XLA, TVM, or MLIR
- Multi-node scaling experience with Megatron-LM, NeMo, DeepSpeed, Triton Inference Server, or TGI
- Multi-vendor GPU experience with ROCm, RCCL, or AMD GPU optimization
- Experience shipping an AI agent or tool used by others; MCP, LangGraph, or Anthropic/OpenAI SDKs
- Kubernetes operators, Helm, Argo CD, Slurm, or batch scheduler experience
- Kafka, Redis Streams, Grafana, or high-volume telemetry pipeline experience
- MLPerf LoadGen or benchmarking tooling experience
- NVIDIA DALI, parallel filesystems, NVMe tuning, or I/O optimization experience
