AI Engineer - L3

At Robots & Pencils, we build meaningful, scalable digital products by blending strategy, design, and engineering. We are seeking a Level 3 AI Engineer to support the delivery of production LLM-powered applications for enterprise clients as part of long-term engagement teams.

You will work hands-on alongside senior AI engineers, product partners, and delivery teams to implement and evolve intelligent features using modern LLM tooling. This role focuses on building reliable AI capabilities that move beyond experimentation into real-world production systems.

You will contribute to RAG pipelines, prompt design, backend integration, and evaluation of workflows while helping scale AI solutions that operate reliably in enterprise environments.

What You’ll Do

AI & LLM Application Development

· Contribute to building and optimizing Retrieval Augmented Generation (RAG) pipelines, including document ingestion, chunking strategies, and retrieval workflows

· Develop and refine prompts and guardrails supporting domain-specific LLM applications

· Support implementation of hallucination mitigation and response validation techniques

· Build embeddings-based semantic search and recommendation capabilities.

· Assist in testing AI features with users and iterate based on feedback and performance data

Evaluation, Monitoring & Reliability

· Support implementation of evaluation frameworks measuring LLM quality and relevance

· Assist with monitoring production AI systems and identifying quality or performance issues

· Analyze evaluation data to improve prompts, retrieval strategies, and outputs

· Participate in debugging and continuous improvement of live AI systems

Platform & Backend Integration

· Develop backend services in Python supporting AI workflows and application integration

· Work with vector databases to enable semantic search and retrieval

· Integrate AI features into user-facing products alongside product and engineering teams

· Support scalable and secure deployment of AI capabilities aligned with enterprise standards

Delivery & Collaboration

· Collaborate closely with product managers, designers, and engineers in cross-functional teams

· Participate in agile delivery environments with evolving requirements

· Communicate technical concepts clearly with teammates and stakeholders

· Contribute to delivery outcomes across experimentation, iteration, and production deployment

Required Skills & Experience

· 4+ years of professional software engineering experience

· 1–3 years working with applied AI/ML, data platforms, or LLM-powered applications

· Experience building backend services using Python

· Exposure to LLM applications including prompt engineering or RAG workflows

· Familiarity with embeddings, semantic search, or vector databases

· Experience integrating APIs or cloud services into production applications

· Understanding of software engineering best practices including testing and observability

· Ability to work effectively in ambiguous or fast-moving environments

· Bachelor’s degree in Computer Science, Engineering, Data Science, or equivalent practical experience

Nice to Have

· Experience with Azure AI services or OpenAI APIs

· Exposure to document processing pipelines (PDF extraction, OCR)

· Experience working with cloud platforms (Azure or AWS)

· Familiarity with evaluation tooling such as Langfuse or custom monitoring systems

· Experience contributing to enterprise or B2B software delivery

Tech Stack

· LLMs: Claude (Anthropic), Azure OpenAI

· Backend: Python

· Vector Databases: Weaviate or similar

· Infrastructure: Azure or AWS

· Evaluation & Observability: Langfuse or similar tooling

How You Work

· You are hands-on and collaborative, contributing to delivery outcomes

· You are curious about AI and continuously improving your technical skills

· You balance experimentation with production reliability

· You communicate clearly and work well across disciplines

· You are comfortable learning in evolving technical environments

Why Robots & Pencils

· Work on production AI solutions delivering measurable client impact

· Learn alongside senior engineers across the full AI lifecycle

· Exposure to modern GenAI delivery including RAG, evaluation, and observability

· Collaborative teams with direct exposure to enterprise clients