Senior AI Engineer

This role is for one of Weekday’s clients

Min Experience: 10+ years

Location: Bengaluru and Palo Alto

Department: Engineering Employment
JobType: full-time

We are looking for a highly experienced Senior AI Engineer to design, develop, and operate the backend and AI infrastructure required to run intelligent AI agents securely, efficiently, and reliably in production. This role combines hands-on AI engineering with strong backend development, platform engineering, and production operations.

You will work across LLM-powered applications, AI agents, model routing, tool integrations, inference services, multi-agent workflows, and evaluation frameworks. You will be responsible for transforming early-stage prototypes into scalable, observable, secure, and production-ready services.

The ideal candidate has strong proficiency in Python, practical experience building AI applications and agentic systems, and a solid understanding of the Model Context Protocol (MCP). Experience with Rust-based services, cloud infrastructure, and AI platform engineering will be valuable.

Key Responsibilities

• Design, build, and deploy production-grade AI applications and agent infrastructure using Python.
• Develop agent workflows involving tool calling, memory, retrieval-augmented generation (RAG), planning, and multi-agent collaboration.
• Integrate hosted and self-hosted language models, inference endpoints, and enterprise systems.
• Build MCP servers, clients, tools, and gateway integrations to enable secure and governed access to external tools and data.
• Implement structured outputs, retries, fallbacks, error recovery, and human approval workflows.
• Improve AI system performance through evaluation, experimentation, tracing, and continuous feedback.

• Develop scalable APIs, services, asynchronous workers, and backend components for AI-powered systems.
• Design reusable abstractions for models, agents, tools, workflows, policies, and execution environments.
• Build event-driven services for long-running AI workloads.
• Work with PostgreSQL, Redis, message queues, object storage, and vector databases.
• Contribute to Rust-based platform services and architectural decisions where required.
• Write maintainable, well-tested code with clear interfaces and strong operational visibility.

• Build repeatable workflows for packaging, deploying, versioning, evaluating, and rolling back AI services.
• Develop real-time and batch inference pipelines and model-serving integrations.
• Monitor model and agent quality, latency, reliability, errors, token usage, and operational costs.
• Manage model, prompt, configuration, and evaluation versions.
• Establish evaluation pipelines and feedback loops to improve AI system performance.
• Transition AI prototypes into secure, scalable, and observable production environments.

• Own services throughout development, deployment, monitoring, and production support.
• Build and maintain CI/CD pipelines, release automation, and infrastructure-as-code.
• Deploy and operate applications using Docker, Kubernetes, and cloud infrastructure.
• Implement health checks, autoscaling, alerting, graceful failure handling, and rollback strategies.
• Apply best practices for authentication, authorization, secrets management, service identity, and tenant isolation.
• Use logs, metrics, and distributed tracing to troubleshoot issues and improve reliability.

Required Qualifications

• 10+ years of professional experience in software engineering, AI engineering, backend development, or platform engineering.
• Strong proficiency in Python, including experience developing production-grade backend and AI applications.
• Hands-on experience building and deploying LLM-powered applications, AI agents, or agentic workflows.
• Practical understanding of MCP (Model Context Protocol), including tool integration, MCP servers, clients, or gateway-based architectures.
• Experience with LLM APIs, tool calling, embeddings, vector search, RAG, and structured generation.
• Strong backend fundamentals covering API design, asynchronous processing, databases, and distributed services.
• Experience deploying and operating production systems using Docker, Kubernetes, cloud platforms, and CI/CD.
• Familiarity with AI evaluation, observability, monitoring, and inference infrastructure.
• Ability to independently take technical capabilities from design through implementation, deployment, and production support.
• Strong problem-solving, communication, documentation, and architectural decision-making skills.

Must-Have Skills

• Python Programming
• Artificial Intelligence (AI) Engineering
• MCP (Model Context Protocol)
• LLM Applications and AI Agents
• Backend Development and API Design
• AI Tool Integration and Agent Workflows
• Production Deployment and Monitoring

Good-to-Have Skills

• Rust Programming
• LangGraph, AutoGen, Agno, or similar agent frameworks
• RAG, embeddings, vector databases, and semantic search
• LLM routing, AI gateways, and model-serving infrastructure
• vLLM, Triton, ONNX Runtime, or Ray Serve
• PostgreSQL, Redis, Kafka, or RabbitMQ
• Docker, Kubernetes, Helm, and Terraform or Pulumi
• OpenTelemetry, Prometheus, and Grafana
• MLflow, Weights & Biases, and AI evaluation pipelines
• AI security, governance, policy enforcement, and token-cost optimization
• Multi-agent orchestration and distributed systems

Preferred Domain Experience

Candidates with direct experience building AI infrastructure, LLM routers, model gateways, MCP gateways, agent runtimes, workflow engines, inference platforms, AI observability tools, evaluation systems, or AI governance products will be strongly preferred.

Experience with GPU-backed inference, multi-tenant SaaS platforms, customer-managed deployments, open-source AI infrastructure, or Rust-based production services will be an additional advantage.