Senior Engineer I, Inference Service

We’re seeking an experienced Software Engineer to drive the design, optimization and scaling of our inference systems. In this role, you will build systems for optimized inference serving of well known open source / open weights models, develop novel techniques for serving custom models and scale the platform to handle millions of users across the globe. 

As a Senior Software Engineer at DigitalOcean, you will join a dynamic team dedicated to revolutionizing cloud computing and AI. 

• Design and implement distributed inference platform for serving large language models
• Optimize runtime and infrastructure layers of the inference stack for best model performance
• Building native cross platform inference support across NVIDIA and AMD GPUs for a variety of model architectures
• Contribute to open source inference engines to make them perform better on DigitalOcean cloud.
• Building tooling and observability to monitor system health, and build auto tuning capabilities
• Building benchmarking frameworks to test model serving performance to guide system and infrastructure tuning efforts
• Mentor engineers on inference systems, GPU infrastructure and distributed inference best practices

Required Qualifications

• Bachelor’s or Master’s in Computer Science, Electrical Engineering, or related field
• Have experience building distributed systems with Kubernetes, gRPC, golang, python
• Have experience with GPU programming with CUDA, ROCm
• Have experience with L3-L7 network protocols, block storage, object storage
• Have experience with building multi-tenant systems - identity management, tenant isolation etc
• Proven track record defining and achieving performance KPIs (latency, throughput, cost)

Preferred Qualifications

• Experience with one or more inference engines: vLLM, SGLang, Modular etc
• Familiarity with one or more distributed inference serving frameworks - llm-d, NVIDIA dynamo, Ray Serve etc
• Knowledge of distributed inference optimization techniques - tensor/data parallelism, KV cache optimizations, smart routing etc
• Familiarity with common LLM architectures and inference optimization techniques (e.g. continuous batching, quantization etc)
•      Experience with GPU Interconnect technologies like NVlink, XGMI, RoCE
•    Open source contributions to inference libraries, frameworks, model kernels etc

• We innovate with purpose. You’ll be a part of a cutting-edge technology company with an upward trajectory, who are proud to simplify cloud and AI so builders can spend more time creating software that changes the world. As a member of the team, you will be a Shark who thinks big, bold, and scrappy, like an owner with a bias for action and a powerful sense of responsibility for customers, products, employees, and decisions.
• We prioritize career development. At DO, you’ll do the best work of your career. You will work with some of the smartest and most interesting people in the industry. We are a high-performance organization that will always challenge you to think big. Our organizational development team will provide you with resources to ensure you keep growing. We provide employees with reimbursement for relevant conferences, training, and education. All employees have access to LinkedIn Learning's 10,000+ courses to support their continued growth and development.
• We care about your well-being. Regardless of your location, we will provide you with a competitive array of benefits to support you from our Employee Assistance Program to Local Employee Meetups to flexible time off policy, to name a few. While the philosophy around our benefits is the same worldwide, specific benefits may vary based on local regulations and preferences.
• We reward our employees. The salary range for this position is between $114,200 - $142,700 based on market data, relevant years of experience, and skills. You may qualify for a bonus in addition to base salary; bonus amounts are determined based on company and individual performance. We also provide equity compensation to eligible employees, including equity grants upon hire and the option to participate in our Employee Stock Purchase Program. 
• We value diversity and inclusion. We are an equal-opportunity employer, and recognize that diversity of thought and background builds stronger teams and products to serve our customers. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, ancestry, national origin, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service.

*This is a remote role

#LI-Remote