AI Tester

Job Summary: 
As an AI Tester, you will be responsible for identifying risks to the business and investigating the quality, accuracy, and fairness of applications built on top of Large Language Models. Working closely with data scientists, engineers, and business stakeholders, you will design and execute test strategies that address functional and non-functional requirements. Your expertise in AI testing will be critical in ensuring the application performs as intended, that the guardrails implemented mitigates risks, and aligns with regulatory and ethical standards. 

Key Responsibilities: 

Test Planning and Strategy: 

• Develop comprehensive test plans and strategies for the LLM based application implementation, covering functional, performance, security, and ethical considerations. 

• Collaborate with stakeholders to understand use cases and key business processes the AI agent will support. 

Application Validation and Functional Testing: 

• Test the accuracy, consistency, and reliability of the LLM’s outputs across various scenarios and use cases. 

• Validate the model’s adherence to prompts, contextual understanding, and user intent. 

• Identify and address potential failure modes in conversational flows. 

Bias and Fairness Testing: 

• Assess the model for biases and unfair treatment in outputs, ensuring compliance with ethical guidelines and company policies. 

• Collaborate with data teams to test fairness across diverse customer demographics. 

Performance and Scalability Testing: 

• Measure the response times, throughput, and scalability of the application under different load conditions. 

• Test integrations with APIs, backend systems, and other enterprise tools. 

Security and Compliance Testing: 

• Identify vulnerabilities and ensure the LLM meets security and data protection standards. 

• Validate compliance with industry regulations such as GDPR, CCPA, or other industry standards & regulations. 

Automation and Tooling: 

• Develop and maintain automated test suites for model retraining and updates. 

• Use AI-specific testing tools and frameworks to improve efficiency and accuracy in validation efforts. 

Defect Reporting and Resolution: 

• Document issues, collaborate with developers to resolve them, and retest to confirm fixes. 

• Maintain clear and detailed test artifacts to support audits and reviews. 

Qualifications: 

Required: 

• Bachelor’s degree in Computer Science, Engineering, or related field. 

• 3+ years of experience in software testing, including at least 1 year of AI/ML testing experience. 

• Familiarity with Large Language Models, NLP (Natural Language Processing), and AI frameworks. 

• Strong understanding of testing methodologies and frameworks for AI systems, including bias testing, performance testing, and security testing. 

• Programming skills in Python, JavaScript, or another language relevant to test automation. 

• Excellent analytical and problem-solving skills with attention to detail. 

• Experience working in agile development environments. 

Preferred: 

• Experience with cloud-based AI services (e.g., AWS, Azure, Google Cloud AI). 

• Knowledge of insurance industry processes and systems. 

• Certification in AI/ML or software testing (e.g., ISTQB, Certified Machine Learning Specialist). 

• Experience with testing tools such as LangTest, DeepChecks, or similar AI-focused platforms. 

Why Join Us? 

• Be a part of transformative AI initiatives. 

• Collaborate with a diverse, innovative team. 

• Access opportunities for professional growth and upskilling in cutting-edge technologies. 

• Competitive salary, benefits, and flexible work arrangements. 

If your experience or qualifications is similar to our ideal of a successful candidate, please consider applying. Experience comes in many ways; skills may be transferred, but passion for your career can't be substituted. At TTC, we understand the importance of diversity and how much value it brings to the table. Diversity brings about creativity and new perspectives, which is why we beckon everyone to apply.