Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.
Key Responsibilities for Automation QA (AI Testing Suite):
- Framework Development:
- Design and implement a scalable automation testing framework for AI applications using .NET technologies.
- Build reusable libraries and components that enhance the efficiency of the testing process.
- Test Script Development:
- Write, maintain, and optimize automated test scripts for functional, regression, and integration tests using frameworks like Selenium, SpecFlow, or NUnit.
- Ensure test scripts are designed to handle the unique requirements of AI applications, including validating model outputs and performance.
- AI Model Validation:
- Collaborate with data scientists and engineers to understand the architecture and outputs of AI models.
- Develop specific test cases to evaluate the accuracy, performance, and reliability of AI models, including edge case scenarios.
- Data Management:
- Create and manage diverse test datasets that accurately reflect real-world scenarios, ensuring comprehensive coverage for testing AI models.
- Implement data validation processes to ensure input data integrity and correctness.
- Integration into CI/CD:
- Integrate automated testing processes into CI/CD pipelines using tools like Azure DevOps or Jenkins to facilitate continuous integration and delivery.
- Collaborate with DevOps teams to ensure smooth deployment and execution of automated tests.
- Performance Testing:
- Conduct performance and load testing of AI applications using tools like JMeter or LoadRunner to assess scalability and responsiveness under various conditions.
- Analyze performance metrics and work with development teams to identify and resolve performance bottlenecks.
- Collaboration and Agile Practices:
- Work closely with cross-functional teams, including developers, product managers, and data scientists, to gather requirements and clarify testing objectives.
- Participate in agile ceremonies, contributing to sprint planning, daily stand-ups, and retrospectives.
- Documentation and Reporting:
- Maintain clear and comprehensive documentation of the automation framework, test cases, and testing outcomes.
- Utilize bug tracking tools like Azure DevOps, JIRA to report, track, and prioritize defects, ensuring clear communication with development teams.
- Continuous Improvement:
- Stay updated with the latest trends in automation and AI testing, evaluating new tools and techniques for potential integration into the testing process.
- Proactively suggest improvements to existing testing methodologies and frameworks to enhance efficiency and effectiveness.
Expected Technology Stack:
- AI/ML Tools:
- Familiarity with frameworks like TensorFlow or PyTorch (to understand AI model validation)
- Tools for model evaluation (e.g., Scikit-learn)
- End-to-end development cycle of LLM-based AI applications – (e.g. Promptflow)
- Understanding on Prompt Evaluation Metrics.
- Programming Languages (preferred):
- C# (primary language for .NET development)
- Python (for AI-related testing and data manipulation)
- JAVA (at least beginner level understanding)
- Protractor (Good to Have)
- Automation Frameworks:
- Selenium (for web application testing)
- SpecFlow (for behavior-driven development)
- NUnit (for unit testing)
- Continuous Integration/Continuous Deployment (CI/CD):
- Azure DevOps
- Jenkins
- GitHub Actions
- Performance Testing Tools:
- Apache JMeter
- LoadRunner
- Bug Tracking and Documentation:
- Azure DevOps / JIRA (for documentation, defect logging and tracking)
- Version Control: Git (for source control)
Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.
Candidate Privacy Policy
Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:
- What information we collect during our application and recruitment process and why we collect it;
- How we handle that information; and
- How to access and update that information.
Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.
Top Skills
What We Do
Orion is a leading digital transformation and product development services firm. Headquartered in Edison, NJ, we have a global team of 6,200+ associates, with engineers in 14 major delivery centers across North America, Europe, Asia Pacific and Latin America.
For over 25 years, Orion has been solving complex business problems for our clients. Our transformative business solutions are rooted in digital strategy, experience design, and engineering, empowering our clients to operate with agility at scale.
Our mission is to serve as an agile and trusted partner for business transformation initiatives, providing deep emerging technology, experience design, and domain expertise.
Our business has more than tripled over the last three years.
We have grown aggressively both organically and inorganically, adding new clients, complementary skills, domain expertise, and strengthening our global footprint.
 (1).png)
