Software QA Lead – AI & GenAI Platforms

Work with Us. Change the World.

At AECOM, we're delivering a better world. Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thrive. We are the world's trusted infrastructure consulting firm, partnering with clients to solve the world’s most complex challenges and build legacies for future generations.

There has never been a better time to be at AECOM. With accelerating infrastructure investment worldwide, our services are in great demand. We invite you to bring your bold ideas and big dreams and become part of a global team of over 50,000 planners, designers, engineers, scientists, digital innovators, program and construction managers and other professionals delivering projects that create a positive and tangible impact around the world.

We're one global team driven by our common purpose to deliver a better world. Join us.

Build the quality engine behind enterprise AI.

We’re looking for a hands-on QA leader to drive the strategy and execution of quality assurance for enterprise-grade Generative AI assistant applications.

You’ll own how AI systems are tested, validated, and scaled, spanning full-stack applications powered by LLMs and RAG workflows. This role is critical to ensuring reliable AI outputs, strong user experiences, and responsible, ethical AI behavior in production.

This position will offer flexibility for hybrid work schedules to include both in-office presence and telecommute/virtual work, to be based in Chicago, IL.

Key Responsibilities

• Define and execute QA strategy for end-to-end AI applications (frontend, backend, and AI workflows)

• Lead validation of LLM outputs for accuracy, coherence, and reliability across diverse scenarios

• Test and optimize RAG pipelines to ensure high-quality retrieval and grounded responses

• Design user-centric test scenarios (multi-turn conversations, edge cases, failure handling)

• Conduct performance, load, and latency testing across AI and application layers

• Build and scale automated testing frameworks (API, UI, integration) within CI/CD pipelines

• Establish and track quality metrics (accuracy, relevance, latency, user satisfaction)

• Ensure ethical AI standards, including bias detection and responsible output validation

• Partner cross-functionally to embed QA practices across the development lifecycle

Minimum Qualifications

• Bachelor's Degree plus at least 10 years of software development experience, or demonstrated equivalency of experience and/or education

• Proven experience leading QA for complex, full-stack applications

• Strong expertise in testing Generative AI systems (LLMs, RAG workflows)

• Hands-on experience with test automation tools (e.g., Selenium, Appium)

• Experience with API testing, distributed systems, and CI/CD pipelines

• Familiarity with performance testing tools (e.g., JMeter, LoadRunner)

• Strong analytical, problem-solving, and communication skills

Preferred Qualifications

• Experience with AI evaluation techniques (prompt testing, output scoring, bias testing)

• Background in conversational AI, NLP, or chatbot testing

• Familiarity with enterprise data/retrieval systems and knowledge bases

• Experience working in cloud environments (AWS, Azure, or GCP)

Additional Information

• Relocation assistance is not available for this position

• Sponsorship for US work authorization is not available for this position, now or in the future.

About AECOM 

AECOM is proud to offer comprehensive benefits to meet the diverse needs of our employees. Depending on your employment status, AECOM benefits may include medical, dental, vision, life, AD&D, disability benefits, paid time off, leaves of absences, voluntary benefits, perks, flexible work options, well-being resources, employee assistance program, business travel insurance, service recognition awards, retirement savings plan, and employee stock purchase plan. 

AECOM is the global infrastructure leader, committed to delivering a better world. As a trusted professional services firm powered by deep technical abilities, we solve our clients’ complex challenges in water, environment, energy, transportation and buildings. Our teams partner with public- and private-sector clients to create innovative, sustainable and resilient solutions throughout the project lifecycle – from advisory, planning, design and engineering to program and construction management. AECOM is a Fortune 500 firm that had revenue of $16.1 billion in fiscal year 2025. Learn more at aecom.com. 

What makes AECOM a great place to work 

You will be part of a global team that champions your growth and career ambitions. Work on groundbreaking projects - both in your local community and on a global scale - that are transforming our industry and shaping the future. With cutting-edge technology and a network of experts, you’ll have the resources to make a real impact. Our award-winning training and development programs are designed to expand your technical expertise and leadership skills, helping you build the career you’ve always envisioned. Here, you’ll find a welcoming workplace built on respect, collaboration and community—where you have the freedom to grow in a world of opportunity. 

As an Equal Opportunity Employer, we believe in your potential and are here to help you achieve it. All your information will be kept confidential according to EEO guidelines.