About Boomi and What Makes Us Special
Are you ready to work at a fast-growing company where you can make a difference? Boomi aims to make the world a better place by connecting everyone to everything, anywhere. Our award-winning, intelligent integration and automation platform helps organizations power the future of business. At Boomi, you’ll work with world-class people and industry-leading technology. We hire trailblazers with an entrepreneurial spirit who can solve challenging problems, make a real impact, and want to be part of building something big. If this sounds like a good fit for you, check out boomi.com or visit our Boomi Careers page to learn more.
How You'll Make An Impact: As a Software Engineer in AI Quality, you will play a role in ensuring the quality, reliability, and performance of our cutting-edge Generative AI features. This is an exciting opportunity to work at the intersection of software quality engineering and AI, where you'll help define testing strategies and evaluation frameworks for Agentic applications. You'll grow your skills in both software engineering and emerging AI quality practices while contributing to products that push the boundaries of what's possible with AI.
What You Will Do:
Develop and execute comprehensive test plans for Generative AI features, including prompt testing, output validation, and behavioral testing of Agentic systems.
Design and implement evaluation frameworks to assess AI output quality, consistency, accuracy, and reliability across different use cases and scenarios.
Write automated tests to validate AI feature behavior, performance, and edge cases.
Collaborate closely with other team members and product stakeholders to understand AI feature requirements and identify potential quality risks early in the development cycle.
Perform both manual and automated testing of AI applications, documenting findings and communicating issues clearly to the development team.
Experiment with and leverage AI evaluation tools and frameworks such as AWS Bedrock evaluations, LangSmith, or custom-built testing solutions.
Contribute to the development of quality standards and best practices specifically tailored for Generative AI applications.
Monitor and analyze AI system behavior in development and production environments, identifying patterns and anomalies that may indicate quality issues.
Create clear, comprehensive documentation including test plans, evaluation criteria, test results, and quality reports.
Stay current with emerging trends, tools, and techniques in AI quality assurance and LLM evaluation.
The Experience You Bring
AI & Gen AI Interest: Strong curiosity and foundational knowledge of Generative AI concepts. Eagerness to deepen your understanding of how AI systems work and how to effectively test them.
Software Engineering Foundation: Solid grasp of core computer science concepts including data structures, algorithms, debugging techniques, and API interactions. Understanding of software development lifecycle and quality assurance principles.
Programming Skills: Hands-on coding experience in Python.
Testing & Quality Mindset: Understanding of testing methodologies including functional testing, integration testing, and test case design. Willingness to learn prompt-based evaluation techniques, LLM-specific testing approaches, and AI quality metrics.
Evaluation & Analysis: Strong analytical skills with attention to detail. Ability to assess AI outputs critically, identify edge cases, and think creatively about potential failure modes in AI systems.
Versatility & Adaptability: Comfortable working across different types of tasks—from manual exploratory testing to writing automated test scripts to creating documentation. Willingness to experiment with new tools and approaches in an evolving field.
Problem-Solving: Good troubleshooting abilities and systematic approach to identifying root causes of quality issues.
Communication & Collaboration: Strong written and verbal communication skills. Ability to document findings effectively and collaborate with cross-functional teams using tools like Jira and Confluence.
Learning Orientation: Self-motivated learner who stays curious and proactive about acquiring new skills in AI, testing frameworks, and quality practices.
Education & Experience
Bachelor's degree in Computer Science, Software Engineering, or a related field with upto 2 years of experience in software testing, software development in Gen AI applications.
Bonus points if you have:
Hands-on experience testing Gen AI applications or working with Gen AI & Agentic evaluation tools.
Familiarity with prompt engineering or conversational AI systems.
Experience with test automation frameworks (pytest, unittest, Selenium, etc.).
Understanding of CI/CD pipelines and automated testing integration.
Participation in AI-related projects, hackathons, or coursework focused on machine learning or natural language processing.
#LI-ML-1
Be Bold. Be You. Be Boomi. We take pride in our culture and core values and are committed to being a place where everyone can be their true, authentic self. Our team members are our most valuable resources, and we look for and encourage diversity in backgrounds, thoughts, life experiences, knowledge, and capabilities.
All employment decisions are based on business needs, job requirements, and individual qualifications.
Boomi strives to create an inclusive and accessible environment for candidates and employees. If you need accommodation during the application or interview process, please submit a request to [email protected]. This inbox is strictly for accommodations, please do not send resumes or general inquiries.
What the Team is Saying
Similar Jobs
What We Do
Boomi, the leader in AI-driven automation, enables organizations worldwide to connect everything, automate processes, and accelerate outcomes. The Boomi Enterprise Platform — including Boomi Agentstudio — unifies integration and automation along with data, API, and AI agent management, in a single, comprehensive solution, helping organizations radically simplify the complexity of enterprise software. Trusted by over 25,000 customers, with a user community of 250,000+ users, and supported by a network of 800+ partners, Boomi is driving agentic transformation — helping enterprises of all sizes achieve agility, efficiency, and innovation at scale.
Why Work With Us
Boomi boasts an award-winning work culture with an emphais on being transparent, innovative, accountable, true to our authentic selves, and winning together as One Boomi. As we grow rapidly, invest in talent, and cultivate careers, much opportunity exists for professional growth and participation in our vibrant culture and employee resource groups.
Gallery
Boomi Offices
Hybrid Workspace
Employees engage in a combination of remote and on-site work.
Boomi is committed to leveraging flexible work solutions for team members, whether it’s remote or providing co-working spaces across the country
