AI Engineer - CUDA/GPU Kernel

• CUDA
• CUTLASS
• C/C++ and Python binding tools

The role is based in the Bay Area [San Francisco and Palo Alto]. Candidates are expected to be located near the Bay Area or open to relocation.

• Developing and improving low-level CUDA kernel optimizations for state-of-the-art inference and training software stack.
• Profiling, debugging, and optimizing single and multi-GPU operations using tools such as Nsight.
• Understanding GPU memory hierarchy and computation capabilities.
• Implementing the latest methods from the deep learning literature in low-level CUDA kernels.
• Innovating new ideas that bring us closer to the limits of a GPU.

• Building high-performance GeMM CUDA kernels using Tensor cores or CUDA cores from scratch or by utilizing CuTe/CUTLASS.
• Implementing features for attention kernel by extending existing kernels or writing them from scratch.
• Comfortable with writing both forward and backward kernels and ensuring its correctness while considering floating point errors.
• Optimizing for both memory-bound and compute-bound operations.
• Reasoning about register pressure, shared-memory usage and GPU utilization through tools such as Nsight and removing bottlenecks.
• Being familiar with the latest and the most effective techniques in optimizing inference and training workloads.
• Using pybind to integrate custom-written kernels into a framework, specially JAX/XLA.

After submitting your application, the team reviews your CV and statement of exceptional work. If your application passes this stage, you will be invited to a 15-minute interview (“phone interview”) during which a member of our team will ask some basic questions. If you clear the initial phone interview, you will enter the main process, which consists of four technical interviews:

1. Coding assessment in a language of your choice.
2. Systems hands-on: Demonstrate practical skills in a live problem-solving session.
3. Project deep-dive: Present your past exceptional work to a small audience.
4. Meet and greet with the wider team.

Our goal is to finish the main process within one week. All interviews will be conducted via Google Meet.

$180,000 - $440,000 USD

Base salary is just one part of our total rewards package at xAI, which also includes equity, comprehensive medical, vision, and dental coverage, access to a 401(k) retirement plan, short & long-term disability insurance, life insurance, and various other discounts and perks.