Be an Early Applicant
Cat Digital delivers digital and technology solutions that power Caterpillar's world-famous yellow iron.
The Role
Leads an AI Evaluation Engineering team responsible for validating generative AI products, including intelligent agents and digital assistants. Provides technical direction, manages team performance and development, owns product quality, and establishes engineering best practices. The role requires expertise in GenAI architecture, evaluation methodologies, observability, automated testing, cloud and edge environments, reliability, safety, RAG assessment, and human-in-the-loop validation.
Summary Generated by Built In
Career Area:
Technology, Digital and Data
Job Description:
Your Work Shapes the World at Caterpillar Inc.
When you join Caterpillar, you're joining a global team who cares not just about the work we do - but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here - we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.
Job Summary:
Join the AI Engineering team of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutions-including intelligent agents, digital assistants, and other innovative capabilities. You will shape the development of cutting-edge products that redefine how customers and dealers interact with technology through AI-driven experiences.
What You Will Do:
What You Will Have:
Considerations For Top Candidates:
Summary Pay Range:
$147,760.00 - $240,110.00
Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location, job-related knowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.
Benefits:
Subject to plan eligibility, terms, and guidelines. This is a summary list of benefits.
* These benefits also apply to part-time employees
Visa Sponsorship is not available for this position.
Posting Dates:
September 1, 2026 - September 20, 2026
Any offer of employment is conditioned upon the successful completion of a drug screen.
Caterpillar is an Equal Opportunity Employer, Including Veterans and Individuals with Disabilities. Qualified applicants of any age are encouraged to apply.
Not ready to apply? Join our Talent Community.
Technology, Digital and Data
Job Description:
Your Work Shapes the World at Caterpillar Inc.
When you join Caterpillar, you're joining a global team who cares not just about the work we do - but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here - we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.
Job Summary:
Join the AI Engineering team of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutions-including intelligent agents, digital assistants, and other innovative capabilities. You will shape the development of cutting-edge products that redefine how customers and dealers interact with technology through AI-driven experiences.
What You Will Do:
- Providing strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.
- Overseeing the performance of both individual team members and the team as a whole, fostering a culture of learning by identifying and addressing training and development needs.
- Taking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meet high standards.
- Establishing and supervising the implementation of engineering best practices to maintain consistency and excellence in development processes.
What You Will Have:
- Products and Services: Knowledge of major products and services and product and service groups; ability to apply knowledge of product and service appropriately to diverse situations.
- Software Development: Knowledge of software development tools and activities; ability to produce software products or systems in line with product requirements.
- Software Development Life Cycle: Knowledge of software development life cycle; ability to use a structured methodology for delivering and managing new or enhanced software products to the marketplace.
- Software Quality Assurance and Testing: Knowledge of software quality assurance and testing; ability to apply appropriate processes, tools, and techniques for assuring a high level of quality in computer software products and systems.
Considerations For Top Candidates:
- Proven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.
- Demonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments, leveraging modern software engineering practices including CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies (e.g., blue/green and canary deployments).
- Deep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; fine-tuning techniques such as LoRA; and emerging standards including MCP and A2A.
- Hands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.
- Deep understanding of AI evaluation methodologies and the challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.
- Expertise in modern AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and the use of AI technologies to enhance testing through automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.
Summary Pay Range:
$147,760.00 - $240,110.00
Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location, job-related knowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.
Benefits:
Subject to plan eligibility, terms, and guidelines. This is a summary list of benefits.
- Medical, dental, and vision benefits*
- Paid time off plan (Vacation, Holidays, Volunteer, etc.)*
- 401(k) savings plans*
- Health Savings Account (HSA)*
- Flexible Spending Accounts (FSAs)*
- Health Lifestyle Programs*
- Employee Assistance Program*
- Voluntary Benefits and Employee Discounts*
- Career Development*
- Incentive bonus*
- Disability benefits
- Life Insurance
- Parental leave
- Adoption benefits
- Tuition Reimbursement
* These benefits also apply to part-time employees
Visa Sponsorship is not available for this position.
Posting Dates:
September 1, 2026 - September 20, 2026
Any offer of employment is conditioned upon the successful completion of a drug screen.
Caterpillar is an Equal Opportunity Employer, Including Veterans and Individuals with Disabilities. Qualified applicants of any age are encouraged to apply.
Not ready to apply? Join our Talent Community.
Skills Required
- Knowledge of major products and services and the ability to apply that knowledge appropriately
- Knowledge of software development tools and activities
- Knowledge of the software development life cycle and structured software delivery methodologies
- Knowledge of software quality assurance and testing processes, tools, and techniques
- Proven experience leading software quality, test automation, validation, and AI evaluation teams
- Experience scaling processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives
- Experience delivering enterprise-scale software and generative AI solutions across hybrid cloud and embedded or edge environments
- Experience with CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies
- Expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models
- Experience with prompt engineering, agentic systems, tool orchestration, RAG architectures, vector databases, embeddings, chunking, and LoRA fine-tuning
- Experience with AI development and evaluation ecosystems such as Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, or Phoenix
- Deep understanding of AI evaluation methodologies for non-deterministic systems, including metric design, RAG quality assessment, reliability, safety evaluation, A/B testing, and human-in-the-loop validation
- Expertise in AI observability and monitoring, including tracing, telemetry, evaluation data collection, root-cause analysis, automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation
What the Team is Saying
Pradeep
Software Engineer
Christina
Platform Services
Chad
Business Intelligence & Visualization
Madison
Recruiter
Rakshan
Digital Product Manager
MacGregor
Lead Software Engineer
Stan
Software Engineering Manager
Caterpillar Compensation & Benefits Highlights
-
Retirement Support — Company materials and labor agreements describe a strong 401(k) program with matching and additional employer contributions for many groups, with recent union terms further enhancing retirement provisions. This breadth and employer funding position retirement as a core strength across regular full‑time roles.
-
Healthcare Strength — The Total Rewards overview lists comprehensive medical, dental, and vision coverage, HSAs/FSAs, and an enhanced EAP with up to 16 counseling or coaching sessions per covered person, alongside wellness tools like Healthcare Bluebook. Such depth and supporting resources indicate robust healthcare support for most eligible employees.
-
Leave & Time Off Breadth — Offerings include PTO (vacation, holidays, volunteer time), parental and adoption leave, and in some locations a year‑end shutdown, with tenure‑based accrual growth; union updates also expanded paid‑absence policies. These features provide multiple pathways for time away and family support, though exact practices can vary by site.
Caterpillar Insights
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Cat Digital is the digital and technology arm of Caterpillar Inc., responsible for bringing digital capabilities to our world-famous yellow iron. With over 1.5M connected assets worldwide, our teams use data, technology, advanced analytics and AI capabilities to help solve our customers' toughest challenges.
Why Work With Us
At Cat Digital, we're accelerating Caterpillar's digital transformation to help customers solve their toughest challenges. Our teams use enterprise-wide data, emerging technologies, and customer insights to deliver solutions that drive real-world impact. Join us as we create innovative experiences that help customers succeed and power the world aro
Gallery
Caterpillar Teams
Caterpillar Offices
Hybrid Workspace
Employees engage in a combination of remote and on-site work.
Typical time on-site:
Flexible
