Software Engineer (Inference Engine) is responsible for developing and optimizing a high-performance inference engine for Large Language Models (LLMs) and multimodal LLMs running on FuriosaAI NPUs.
In this role, you will proactively research and apply the state-of-the-art inference optimization techniques to our inference engine. You will work in close collaboration with the compiler and hardware teams to enhance the engine's performance to its full potential.
ResponsibilitiesDesign and implement FuriosaAI’s next-generation inference engine for large and multimodal language models—comparable in capability to frameworks such as vLLM and SGLang—optimized for throughput, latency, and memory efficiency.
Design and implement advanced inference optimizations—such as speculative decoding, KV-cache management, tensor/model parallelism, memory-efficient execution, and scheduling—in our production inference engine.
Design and develop capabilities for distributed and scalable inference, including prefill–decode (PD) and encode–prefill–decode (EPD) disaggregation, disaggregated speculative decoding, and hierarchical and external KV-cache storage such as HiCache and Mooncake.
Collaborate closely with the Compiler team to co-design and optimize execution for FuriosaAI NPUs, improving system-level throughput, latency, and memory utilization.
Proactively research, evaluate, and integrate state-of-the-art inference optimization techniques and key features of LLM serving frameworks into our production inference engine.
BS degree in Computer Science, Engineering, or a related field, with at least 3 years of relevant industry experience, or equivalent practical experience
Proficiency in Rust or C++ programming skill
Knowledge and passion of deep learning, LLM, and/or generative AI models
Excellent problem-solving and data analysis skills.
Strong communication and collaboration skills.
Experience in building inference serving systems for large models, encompassing batching, scheduling, caching, and load balancing.
A deep understanding of performance optimization systems.
Proficiency in C++/CUDA or Triton kernel development
Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM.
Skills Required
- BS degree in Computer Science, Engineering, or related field, with at least 3 years of relevant industry experience, or equivalent practical experience
- Proficiency in Rust or C++ programming skill
- Knowledge and passion of deep learning, LLM, and/or generative AI models
- Excellent problem-solving and data analysis skills
- Strong communication and collaboration skills
- Experience in building inference serving systems for large models, including batching, scheduling, caching, and load balancing
- A deep understanding of performance optimization systems
- Proficiency in C++/CUDA or Triton kernel development
- Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM
What We Do
FuriosaAI designs and develops data center accelerators for the most advanced AI models and applications. Our mission is to make AI computing sustainable so everyone on Earth has access to powerful AI. Our Background Three misfit engineers with each from HW, SW and algorithm fields who had previously worked for AMD, Qualcomm and Samsung got together and founded FuriosaAI in 2017 to build the world’s best AI chips. The company has raised more than $100 million, with investments from DSC Investment, Korea Development Bank, and Naver, the largest internet provider in Korea. We have partnered on our first two products with a wide range of industry leaders including TSMC, ASUS, SK Hynix, GUC, and Samsung. FuriosaAI now has over 140 employees across Seoul, Silicon Valley, and Europe. Our Approach We are building full stack solutions to offer the most optimal combination of programmability, efficiency, and ease of use. We achieve this through a “first principles” approach to engineering: We start with the core problem, which is how to accelerate.
