Sr. Inference Engineer (LLM Serving)

Posted 2 Days Ago
Be an Early Applicant
Prague, CZE
Hybrid
Senior level
Software • Analytics
The Role
Own the production serving layer for open-weight LLMs and task models on company GPUs. Deploy and optimize models across regions, improving latency, throughput, caching, batching, quantization, parallelism, and cost. Evaluate models on analytics workloads, expand inference routing and semantic caching, and adapt models through LoRA and fine-tuning. Collaborate across engineering and product while operating Kubernetes-based infrastructure in AWS.
Summary Generated by Built In

Our platform turns governed business data into answers, reports and agents that companies put in front of their own customers. To make that AI trustworthy, private and affordable, we run it on our own inference: open-weight models on our own GPUs, so customer data never has to leave our boundary.

The role

You will own the serving layer of GoodData Native Inference. That means the models, the engine and the GPUs behind every AI answer our customers get. You will join the team that builds GoodData AI and become the person the whole company turns to for anything about running models in production.

What you will do
  • Deploy, operate and upgrade open-weight LLMs and smaller task models (guardrails, PII detection, classifiers, judges) on our GPUs, in production and across regions.

  • Make inference fast and cheap: tune batching, KV and prefix caching, quantization and parallelism, and track time to first token, tokens per second per GPU, p95 latency and cost per answer.

  • Choose the models: put new open-weight models through our evaluation harness on real analytics workloads, and recommend what goes to production.

  • Grow our inference gateway: routing by intent and complexity across models, a semantic cache, and validation that escalates weak answers.

  • Adapt models to our use cases with LoRA adapters and fine-tuning, such as generating queries for our semantic layer, root-cause analysis and narrative reports.

What you bring
  • 5+ years of software engineering, including 2+ years running ML or LLM models in production.

  • Hands-on production experience with at least one modern LLM serving engine: vLLM, SGLang or TensorRT-LLM.

  • A working understanding of inference bottlenecks (KV cache, attention, memory bandwidth, batching, parallelism), and the habit of reasoning about latency, throughput and cost in numbers.

  • Experience with Docker, Kubernetes and GPU instances in a public cloud.

  • Discipline in benchmarks and evaluation: you do not ship a model you have not measured.

  • Clear communication in English, with engineers and with product people.

Nice to have
  • Speculative decoding, multi-LoRA serving, or PEFT fine-tuning.

  • Contributions to vLLM, SGLang, PyTorch or similar open-source projects.

  • CUDA or Triton kernels, and the NVIDIA GPU stack (drivers, NCCL).

  • LLM routing, guardrail models, embeddings and rerankers.

  • Go or Rust; Terraform and Helm.

  • Observability for LLM systems (OpenTelemetry, Langfuse, Prometheus, Grafana).

  • Experience with analytics, semantic layers or text-to-SQL.

Our stack

Python, Kubernetes on AWS, NVIDIA GPUs, SIE (an open-source inference engine on SGLang), vLLM, PyTorch, gRPC, Langfuse, and our own evaluation harness.

About GoodData.AI

At GoodData.AI, we’re building the future of agentic AI for data intelligence.
Our full-stack data intelligence platform empowers companies to turn raw data into real business impact — from agentic AI apps to embedded analytics that scale securely and seamlessly. With GoodData.AI, organizations don’t just analyze data, they monetize it.
But what really sets GoodData.AI apart is our mission: to put intelligence wherever decisions are made. That’s why we’ve built a platform that’s AI-native, composable, and built for the way modern enterprises work.

 

#LI-Hybrid

Skills Required

  • 5+ years of software engineering experience
  • 2+ years running machine learning or LLM models in production
  • Hands-on production experience with at least one modern LLM serving engine: vLLM, SGLang, or TensorRT-LLM
  • Understanding of inference bottlenecks, including KV cache, attention, memory bandwidth, batching, and parallelism
  • Experience with Docker, Kubernetes, and GPU instances in a public cloud
  • Experience with benchmarks and model evaluation before production deployment
  • Clear communication in English with engineers and product stakeholders
  • Experience with speculative decoding, multi-LoRA serving, or PEFT fine-tuning
  • Contributions to vLLM, SGLang, PyTorch, or similar open-source projects
  • Experience with CUDA or Triton kernels and the NVIDIA GPU stack, including drivers and NCCL
  • Experience with LLM routing, guardrail models, embeddings, or rerankers
  • Experience with Go or Rust, Terraform, or Helm
  • Experience with LLM observability using OpenTelemetry, Langfuse, Prometheus, or Grafana
  • Experience with analytics, semantic layers, or text-to-SQL
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: San Francisco, CA
341 Employees
Year Founded: 2007

What We Do

At GoodData, we help companies turn data into insights. Our leading composable data and analytics platform gives our customers the flexibility to build and scale any of their data use cases — from self-service and embeddable analytics, to machine learning and IoT — all while maintaining the performance, cost-efficiency, and easy change management of a central and integrated solution.

Similar Jobs

ServiceNow Logo ServiceNow

Account Executive

Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Remote or Hybrid
Prague, CZE
29000 Employees

Expedia Group Logo Expedia Group

Associate Account Management (PGA) – Spanish Speaking

AdTech • eCommerce • Information Technology • Software • Travel • Generative AI
Hybrid
Praha, Hlavní město Praha, CZE
16000 Employees
731K-1M Annually

Adyen Logo Adyen

Implementation Engineer

Fintech • Payments • Financial Services
Easy Apply
Hybrid
Prague, CZE
4771 Employees

Pfizer Logo Pfizer

Director, Oncology Scientific Communications

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
In-Office or Remote
34 Locations
121990 Employees
177K-294K Annually

Similar Companies Hiring

Ford Energy Thumbnail
Automotive • Software • Energy • Utilities • Manufacturing • Renewable Energy
US
Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account