Job Title: AI Engineer / Generative AI Engineer
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Extension Possible)
Experience: 15+ Years
Job Summary
We are seeking a hands-on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production-grade AI agents.
Key Responsibilities
Develop AI agents using Python and Claude Agent SDK.
Deploy agents using AWS Bedrock AgentCore Runtime.
Design golden datasets and implement LLM-as-a-Judge evaluations.
Integrate automated evaluations into CI/CD pipelines as quality gates.
Implement Langfuse/OpenTelemetry for AI observability and tracing.
Evaluate agent accuracy, task completion, tool execution, and performance.
Support RAG pipelines and AWS data processing workflows.
Required Skills
Python and Agentic AI development.
AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
Golden-dataset design and curation.
LLM-as-a-Judge design and calibration against human labels.
Automated evaluation and CI gate integration.
Langfuse and/or OpenTelemetry.
Good to Have
RAGAS or equivalent evaluation frameworks.
Trajectory evaluation and agentic red-teaming.
Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.
Qualifications
6–8+ years of software engineering, cloud engineering, or AI/ML experience.
Hands-on experience with production AI agents and continuous evaluation.
Strong problem-solving and troubleshooting skills.
All your information will be kept confidential according to EEO guidelines.
Skills Required
- Python and Agentic AI development experience
- AWS Bedrock AgentCore Runtime and Evaluations experience
- Golden dataset design and curation experience
- LLM-as-a-Judge design and calibration against human labels
- Automated evaluation and CI/CD gate integration experience
- Langfuse and/or OpenTelemetry experience
- 6-8+ years of software engineering, cloud engineering, or AI/ML experience
- Hands-on experience with production AI agents and continuous evaluation
- Strong problem-solving and troubleshooting skills
- RAGAS or equivalent evaluation framework experience
- Trajectory evaluation and agentic red-teaming experience
- Bedrock Guardrails, Strands Agents SDK, and GitHub Actions experience
What We Do
Technology Management Solutions (TMS, LLC), headquartered in Parsippany, New Jersey, is a premier IT company specializing in digital product engineering and innovative technology solutions. Since our inception in 2021, we have empowered businesses across industries—Financial Services, Retail, Medical, Entertainment, Manufacturing, and Automotive—to achieve their digital transformation goals. Our mission is to design and deliver cutting-edge solutions that drive growth, streamline operations, and enhance modern digital experiences.At TMS, we partner with our clients throughout the entire product development lifecycle, from ideation to commercialization. Our expertise spans next-generation technologies, including Artificial Intelligence (AI), Machine Learning (ML), Internet of Things (IoT), and custom software development. With tailored engagement models, we accelerate product development, ensuring faster time-to-market and measurable success.
