ML QA Analyst
Boston (Hybrid)
About OnCorps
OnCorps LLC is a trailblazer in the financial services sector, harnessing the power of advanced AI and machine learning technologies to revolutionize the industry. Our innovative solutions empower financial institutions to enhance efficiency, security, and customer satisfaction – all in support of our mission to reshape financial services through technology.
The Opportunity
This role is central to how our AI systems learn, improve, and perform. As an ML QA Analyst, you’ll work under the Data Science umbrella and own key tasks across model evaluation, training, and ongoing maintenance for the pipelines and work products built by our DS team. You’ll validate agent behavior, strengthen prompts, ensure high-quality labeled data, and keep our models running reliably in production.
It’s an ideal fit for someone with a STEM background, strong analytical instincts, and a passion for precision who wants hands-on experience in a cutting-edge AI environment. This role also offers a clear growth path—successful team members often progress into Data Science roles or transition into Product as Business/Product Analysts.
Key Responsibilities
- Labeling: Owning data labeling for PDF / Excel-based ML ingestion engines across the company. This includes both training-based labeling and live-run corrective labeling.
- Agent Evaluation Review: Review evaluation jobs where agents are underperforming, digging into individual logs and traces and providing documentation of possible prompt weaknesses.
- Agent Regression Testing: Interacting with customer facing “chat” agents. Validating the integrity of an agent’s response to both “happy-path” and adversarial prompts pre-deployment & when model updates are in testing.
- Dashboarding: Owning client-specific dashboard deployments. Working with product/CS stakeholders to deploy client-specific visualizations. Testing dashboards for errors / logical gaps.
- Maintenance-Mode Model Upkeep: Initiating and monitoring routine model retraining for models in a steady-state & fixing job failures in production. Working with Data Science to determine appropriate thresholds for escalation.
- AI Table Annotation
Requirements
- STEM BA/BS
- High attention to detail
- Ability to think critically and analytically
- Ability to follow and understand a complex ruleset
- Familiarity with LLMs/Agents is a plus
- Knowledge of SQL is a plus
- Knowledge of Python is a plus
- Comfortable working in fast-paced environments and meeting tight deadlines
- U.S. based with work authorization
Work Environment
- Primarily desk-based role during standard hours, utilizing PC, video, and phone
- Requires flexibility to work in-office and remotely based on business needs
Why You Should Join Us
Innovative Environment: Work with cutting-edge AI and machine learning technologies in a company that is at the forefront of financial services innovation.
Growth Opportunities: As a rapidly growing startup, we offer significant opportunities for professional development and career advancement.
Collaborative Culture: Be part of a collaborative and inclusive team where your ideas are valued, and your contributions make a real impact.
Work-Life Balance: Enjoy the flexibility of a hybrid working environment, allowing you to balance your professional and personal life effectively.
Competitive Compensation: We offer a competitive salary and a comprehensive benefits package.
Apply today and be a part of the future of financial services technology!
If you are passionate about solutions engineering and eager to work in a cutting-edge AI-driven company, we would love to hear from you!
OnCorps LLC is an equal opportunity employer committed to diversity and inclusion. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, pregnancy, age, marital status, veteran status, or disability status. We provide reasonable accommodations for persons with disabilities. Applicants with equal employment opportunity questions should contact OnCorps at [email protected] or (617) 286-4448. Retaliation against applicants raising EEO issues is prohibited.
Top Skills
What We Do
At OnCorps, our mission is to elevate work and performance by applying AI to handle predictable and laborious calculations, decisions, and tasks. We provide advanced AI solutions to the financial services industry. Our system has substantially reduced manual work, errors and risk. We solve problems with cross-disciplinary teams, from top-tier research universities and consultancies with industry, data science, and computer science expertise.
Our pioneering algorithms have shown dramatic results. Our operational AI systems have reduced labor costs by over 90 percent. Our algorithms have identified millions in errors. And our forecasting systems have consistently beaten client forecasts. Our success has led to strong demand, attracting clients with over $9 trillion in assets under management - 20 percent of the global mutual fund industry. Our innovative algorithms have earned us the 2019 NOVA Award by NICSA and the 2019 Fintech Breakthrough Award for best banking infrastructure software.
Our business model has also achieved breakthrough results. We have consistently designed and deployed complex AI solutions in less than 100 days. We do this by leveraging state-of-the-art cloud technologies to design workflows, train algorithms, and integrate with legacy systems. Expert, cross-disciplinary teams deploy these technologies rapidly. In all cases, our clients have paid no external consulting fees nor required internal technical team support. We believe results from AI can only be achieved by providing constant, high-end support. We provide process, algorithm, and systems support delivered by the same team that designed the solution. This simple difference reduces significant inefficiencies and elevates our service far beyond that of any other firm.
We are committed to delivering to our clients the most cutting-edge solutions possible. To accomplish this, we work closely with scientific advisors from Yale, Harvard, and Oxford to formulate and test algorithms.
