Senior AI Application Engineer

At Klaviyo, the ARIA (Applications, Resources, Intelligence, and Automations) team is building the AI systems that transform how our employees work. We are looking for a Senior IT App Engineer to design, build, and scale production-grade LLM integrations and AI-powered automation tools from the ground up. This role sits at the intersection of applied AI research and full-stack software engineering. You will own everything from LLM integration architecture and RAG system design to CI/CD pipeline management and production observability. If you want to build AI that real people use at scale every day, this is the role for you.

How you’ll make a difference:

• Design and build LLM-powered applications: Architect and develop production-grade systems leveraging large language models (GPT-4, Claude, Mistral, Llama) for internal automation, knowledge retrieval, and intelligent workflow assistance.
• Build and maintain AI integration pipelines: Design robust, scalable API integrations between AI models and internal systems such as HRIS, ticketing, CRM, and knowledge bases, ensuring reliability, low latency, and high availability.
• Develop full-stack AI features end-to-end: Own the complete development of AI features across the stack from backend model orchestration and API layers to frontend interfaces that make AI accessible to non-technical employees.
• Architect and manage CI/CD pipelines for AI systems: Build and maintain automated deployment pipelines for AI models and services, including model evaluation frameworks, A/B testing infrastructure, and safe rollout strategies.
• Implement RAG and retrieval systems: Design and build retrieval-augmented generation systems grounded in Klaviyo internal knowledge, ensuring accuracy, relevance, and up-to-date responses at scale.
• Evaluate and fine-tune AI models: Lead benchmarking, evaluation, and fine-tuning of foundation models for Klaviyo use cases, including prompt engineering and parameter-efficient fine-tuning techniques.
• Establish AI engineering best practices: Define coding standards, testing frameworks, and architectural patterns for AI development at Klaviyo, enabling the team to ship reliable AI features consistently.
• Build AI observability and monitoring tooling: Instrument AI systems with latency monitoring, output quality evaluation, hallucination detection, cost dashboards, and production alerting.
• Collaborate with Product and Data Science: Work closely with PMs, data scientists, and ML engineers to translate requirements into technical implementations, maintaining a tight feedback loop between experimentation and production.
• Drive security and compliance in AI systems: Ensure AI integrations follow data privacy best practices, implement guardrails for sensitive data handling, and maintain compliance with Klaviyo security and governance standards.

Who you are: 

• 3+ years of software engineering experience, with at least 2 years focused on building AI, ML, or LLM-powered systems in production.
• Deep expertise with large language models such as OpenAI, Anthropic, Mistral, or Llama, and experience integrating them into production applications via API.
• Strong full-stack engineering skills across Python (FastAPI, Flask, or Django) and TypeScript or JavaScript, with experience in modern frontend frameworks.
• Proven experience designing and building CI/CD pipelines for software or ML systems using tools such as GitHub Actions, Jenkins, or CircleCI.
• Experience with RAG systems, vector databases (Pinecone, Weaviate, pgvector), and embedding models.
• Proficiency with cloud infrastructure (AWS, GCP, or Azure), containerization (Docker, Kubernetes), and infrastructure-as-code tools such as Terraform or Pulumi.
• Experience with prompt engineering, model evaluation frameworks, and LLM observability tools such as LangSmith or Weights and Biases.