Staff AI Engineer - QA

Posted 3 Hours Ago
Hiring Remotely in North Carolina, USA
Remote
120K-160K Annually
Senior level
Healthtech
The Role
Build and own end-to-end AI quality and evaluation infrastructure, including datasets, automated test harnesses, replay systems, severity-weighted metrics, monitoring, and adversarial testing. Establish accuracy and safety standards for healthcare AI systems, collaborate with clinical and product leaders, mentor engineers, and guide systems from prototype through production.
Summary Generated by Built In

Harris Computer is hiring a Staff AI Engineer to join their, expanding team! We’re seeking a seasoned expert to pioneer the quality and reliability of our cutting-edge AI and machine learning solutions for healthcare. As a Staff AI Engineer focused on Quality Assurance, you will be the technical authority responsible for ensuring our AI systems are robust, fair, accurate, and safe.


This is a unique opportunity to define the future of AI quality in a domain where it matters most, mentoring a talented team and establishing the gold standard for testing intelligent systems that impact patient care. This is a full-time, remote position open to candidates in the U.S. or Canada.


What your impact will be:

  • Build the core AI system end-to-end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.
  • Design and build the evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.
  • Build and maintain evaluation datasets with clinician-reviewed correct answers, ensuring they evolve as our systems change.
  • Create error metrics weighted by severity, so mistakes with clinical or financial impact are prioritized accordingly.
  • Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.
  • Write adversarial tests for AI systems, including hidden or malicious instructions in documents and invented content.
  • Work with product and clinical leads to define accuracy thresholds and the criteria for expanding or pulling back an AI capability.
  • The candidate must possess a curiosity and willingness to actively adopt and leverage emerging AI tools to improve workflows, solve problems, and drive efficiency along with being comfortable using a range of AI-enabled tools (such as copilots, chat-based AI, and automation solutions) as part of everyday work.

What we’re looking for:

  • 7 or more years in software or machine learning engineering, with at least 3 years focused on evaluation, testing, or quality for AI systems.
  • A track record of designing and building complete systems from scratch, taking them from prototype to production.
  • A track record of building and running evaluation systems that teams use in production, not just for one-off analyses.
  • At least 2 years of experience evaluating large language model (LLM) systems in a production environment.
  • Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.
  • 1 year + of experience designing evaluation sets for document extraction, classification, or structured output.
  • Deep knowledge of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).
  • Clear written communication, including design documents, and the ability to explain complex trade-offs to clinicians, engineers, and executives.

What would make you stand out:

  • Experience working within the healthcare or another highly regulated industry.
  • Familiarity with AI-specific testing techniques like metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.
  • A history of mentoring engineers and influencing technical direction across multiple teams.

What we offer:

  • Plenty of opportunities to grow your career
  • Comprehensive medical, dental, and vision benefits
  • 3 weeks of vacation plus 5 personal days to recharge
  • Employee stock ownership, RRSP program, 401k matching
  • A chance to give back through community involvement
  • Flexible work arrangements to suit your lifestyle

Salary range: 120,000-160,000 depending on experience and qualifications.


About Harris Computer:

 Harris provides mission critical software solutions for the Public Sector, Healthcare, Utilities and Private Sector verticals throughout North America, Europe, Asia and Australia.
Working for Harris is the perfect opportunity to fulfill your professional goals as well as achieve your personal dreams!
Our employees enjoy a casual work environment that offers comfort while providing superior service to our customers. We offer a comprehensive benefit package as well as other additional “Perks”!

  • We empower our employees to make a difference
  • We have an award-winning culture
  • We offer opportunity to learn
  • We are financially strong and we are owned by the largest software company in Canada (CSI)
  • We have fun!

#LI-remote

Skills Required

  • 7 or more years of experience in software or machine learning engineering
  • At least 3 years focused on evaluation, testing, or quality for AI systems
  • Experience designing and building complete systems from prototype to production
  • Experience building and running production evaluation systems
  • At least 2 years evaluating large language model systems in production
  • Strong statistical skills, including sampling, confidence intervals, and error analysis
  • At least 1 year designing evaluation sets for document extraction, classification, or structured output
  • Deep knowledge of explainable AI techniques and AI-powered tools such as SHAP, LIME, and Claude
  • Clear written communication and ability to explain complex trade-offs to clinicians, engineers, and executives
  • Experience in healthcare or another highly regulated industry
  • Familiarity with metamorphic testing, adversarial testing, or data validation tools such as Great Expectations
  • Experience mentoring engineers and influencing technical direction across multiple teams

Harris healthcare Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Harris healthcare and has not been reviewed or approved by Harris healthcare.

  • Leave & Time Off Breadth — PTO and paid holidays are characterized as generous, with ample time off emphasized. This breadth of time-off options is highlighted alongside favorable impressions of vacation and sick leave.
  • Flexible Benefits — Remote-work flexibility and trust to work from home are emphasized, indicating adaptable arrangements for many roles. Flexibility is positioned as a tangible part of the overall package.
  • Healthcare Strength — Core coverage includes medical, dental, and vision, along with life and disability insurance and HSA/FSA options. Health plans are often characterized as good, contributing to a solid foundational offering.

Harris healthcare Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Niagara Falls, New York
185 Employees
Year Founded: 1993

What We Do

For over 25 years, Harris Healthcare has been rising to the challenge of bringing together the most innovative and sustainable solutions for today’s ever-changing healthcare environment, in order to improve patient care and safety. Each one of our solutions brings organizational efficiencies on its own. Powerful synergies are achieved when multiple solutions are implemented together. The Harris Healthcare portfolio includes the following solutions: ♦ HARRIS Flex - an enterprise-level EHR solution that improves patient safety and clinical workflows. It includes a full complement of applications integrated in one single database, provides solid clinical decision support to your clinicians and helps standardize care while enforcing protocols and best practices at any Healthcare Organization. HARRIS Flex conveys the digital solution’s flexibility and strength. Healthcare organizations are continuously faced with new challenges and situations and require flexible EHR’s that can be rapidly adapted to their evolving clinical practice. Contrary to other EHR solutions which are inflexible and where customizations require costly support from the vendor, HARRIS Flex gives you the freedom to "flex" your EHR as you need it entirely on your own. The enhanced HARRIS Flex solution comes with new functionality including: ♦Flex Telehealth which enables virtual visits directly from within the EHR/EPR, and ♦Flex Clinical Insight which facilitates extraction and analysis of your EHR/EPR data to improve your processes and outcomes. ♦ SynergyCheck – a proactive interface monitoring solution watching over Clinical, Financial and other interfaces 24/7 to ensure data is flowing between systems

Similar Jobs

Enverus Logo Enverus

Consultant

Big Data • Information Technology • Software • Analytics • Energy
In-Office or Remote
2 Locations
1800 Employees
60K-90K Annually

Enverus Logo Enverus

Staff Product Designer

Big Data • Information Technology • Software • Analytics • Energy
In-Office or Remote
2 Locations
1800 Employees
90K-95K Annually

Babylist Logo Babylist

Senior SEO & AEO Manager

eCommerce • Healthtech • Kids + Family • Retail • Social Media
Easy Apply
Remote or Hybrid
United States
300 Employees
144K-192K Annually

Applied Systems Logo Applied Systems

Associate Billing Coordinator

Artificial Intelligence • Cloud • Payments • Software • Business Intelligence • Generative AI • Automation
Remote or Hybrid
United States
3116 Employees
45K-51K Annually

Similar Companies Hiring

OneImaging Thumbnail
Healthtech
Miami, FL
62 Employees
Pomelo Care Thumbnail
Healthtech • Telehealth
New York, NY
550 Employees
Vitalize Thumbnail
Artificial Intelligence • Healthtech • Software
San Francisco, California
50 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account