In addition to the traditional QA automation tasks, the role also includes test data creation, validation of applications integrated with AI / ML / GenAI workflows.
Responsibilities
Automation Framework and Testing:
Design, develop, and maintain modular, extensible, and reusable test automation frameworks and scripts, while maintaining high stability.
Set up, configure, and maintain the automation environment and data to ensure smooth testing processes.
Develop and execute efficient automated test scripts to validate stability and responsiveness.
Create and maintain a comprehensive test automation library with scenarios to ensure thorough requirements and regression coverage.
Testing and Validation:
Assist in functional, regression, and integration testing for new or modified software programs using both manual and automated approaches.
Collaborate with developers and product teams to identify test requirements and ensure coverage of various workflows.
Perform validation and create test data for various testing scenarios for the applications / APIs / Agents integrated with AI / ML / GenAI workflows and models as necessary.
Implement Risk Gating processes to validate outputs for accuracy, bias, toxicity, correctness, and traceability.
Develop and maintain automation frameworks for continuous validation of applications and AI-driven features.
Identify, report, and track product defects, providing clear and concise documentation of issues and potential improvements.
Collaboration and Process Improvement:
Participate in product design reviews to provide input on requirements, automation feasibility, and testing implications.
Write, review, and verify test plans and procedures for product evaluations and defect identification.
Accurately estimate testing timelines and work toward targeted on-time delivery by proactively removing roadblocks.
Collaborate with cross-functional teams to align product goals and ensure quality standards are met.
Operational Excellence:
Provide feedback and insights to improve the software quality assurance process and methodology.
Stay up to date with the latest automation tools and best practices involving AI/ML to optimize the automation framework.
Qualifications
Required
At least 5 - 7 years of experience in developing and executing automated test cases and test plans scripts.
Proficiency with tools such as Selenium/WebDriver, Playwright or Postman (API automation).
Strong programming skills in one of Python, Java, JavaScript/TypeScript, or C#.
Strong understanding of software quality assurance standards, practices, and methodologies.
Hands-on experience validating end-to-end workflows, edge cases, and non-functional aspects.
Ability to design checks for accuracy, hallucinations, bias, correctness, and traceability of model outputs.
Familiarity with model evaluation metrics (precision/recall, BLEU/ROUGE, toxicity/bias scores) and guardrail patterns.
Strong skills in synthetic test data generation, data masking, and scenario design using SQL or CSVs.
Ability to balance individual and team effort in collaborative processes while meeting set deadlines
Proficiency in API and application / functional validation and experience with API testing tools.
Extensive understanding and continued learnings around QA methodology, tools, and approaches
Strong written and verbal communication skills, with the ability to collaborate effectively across technical and non-technical teams.
Experience testing AI-driven products and LLM-integrated workflows.
Exposure to ML pipelines and data validation (e.g., feature store checks, dataset drift detection).
Knowledge of data visualization tools (e.g., Matplotlib, Seaborn).
Knowledge of cloud platforms (AWS, Azure, or GCP)
Netsmart is proud to be an equal opportunity workplace and is an affirmative action employer, providing equal employment and advancement opportunities to all individuals. We celebrate diversity and are committed to creating an inclusive environment for all associates. All employment decisions at Netsmart, including but not limited to recruiting, hiring, promotion and transfer, are based on performance, qualifications, abilities, education and experience. Netsmart does not discriminate in employment opportunities or practices based on race, color, religion, sex (including pregnancy), sexual orientation, gender identity or expression, national origin, age, physical or mental disability, past or present military service, or any other status protected by the laws or regulations in the locations where we operate.
Netsmart desires to provide a healthy and safe workplace and, as a government contractor, Netsmart is committed to maintaining a drug-free workplace in accordance with applicable federal law. Pursuant to Netsmart policy, all post-offer candidates are required to successfully complete a pre-employment background check, which is provided at Netsmart’s sole expense.
Skills Required
- 5-7 years hands-on experience in developing and executing automated test cases and test plans
- Proficiency with Selenium/WebDriver, Playwright or Postman (API automation)
- Strong programming skills in one of Python, Java, JavaScript/TypeScript, or C#
- Strong understanding of software quality assurance standards, practices, and methodologies
- Hands-on experience validating end-to-end workflows, edge cases, and non-functional aspects
- Ability to design checks for accuracy, hallucinations, bias, correctness, and traceability of model outputs
- Familiarity with model evaluation metrics and guardrail patterns (precision/recall, BLEU/ROUGE, toxicity/bias scores)
- Strong skills in synthetic test data generation, data masking, and scenario design using SQL or CSVs
- Proficiency in API and application/functional validation and experience with API testing tools
- Strong written and verbal communication skills and ability to collaborate across technical and non-technical teams
- Experience testing AI-driven products and LLM-integrated workflows
- Exposure to ML pipelines and data validation (feature store checks, dataset drift detection)
- Knowledge of data visualization tools such as Matplotlib or Seaborn
- Knowledge of cloud platforms (AWS, Azure, or GCP)
NetSmart Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about NetSmart and has not been reviewed or approved by NetSmart.
-
Wellbeing & Lifestyle Benefits — Offerings include a designated mental wellness day, paid volunteer time, telehealth, wellness events, and free 1:1 sessions with counselors and coaches. These wellbeing resources are positioned as stronger than many employers.
-
Healthcare Strength — Medical, dental, and vision coverage are paired with short- and long‑term disability and telehealth. The breadth of core health coverage aligns with a comprehensive package.
-
Leave & Time Off Breadth — PTO, paid holidays, paid volunteer time, and a designated mental wellness day are part of the offering, with some materials referencing flexible or unlimited PTO. This variety provides multiple avenues for time away.
NetSmart Insights
What We Do
Netsmart designs, builds and delivers electronic health records (EHRs), solutions and services that are powerful, intuitive and easy-to-use. Our platform provides accurate, up-to-date information that is easily accessible to care team members in behavioral health, care at home, senior living and social services. We make the complex simple and personalized so our clients can concentrate on what they do best: provide services and treatment that support whole-person care. By leveraging the powerful Netsmart network, care providers can seamlessly and securely integrate information across communities, collaborate on the most effective treatments and improve outcomes for those in their care. Our streamlined systems and personalized workflows put relevant information at the fingertips of users when and where they need it. For 50 years, Netsmart has been committed to providing a common platform to integrate care. SIMPLE. PERSONAL. POWERFUL. Our more than 2,200 associates work hand-in-hand with our 600,000+ users in more than 25,000 organizations across the U.S. to develop and deploy technology that automates and coordinates everything from clinical to financial to administrative.
