Rubric Engineering and Measurement Specialist

About Child Mind Institute

We're dedicated to transforming the lives of children and families struggling with mental health and learning disorders by giving them the help they need. We've become the leading independent nonprofit in children's mental health by providing gold-standard evidence-based care, delivering educational resources to millions of families each year, training educators in underserved communities, and developing tomorrow's breakthrough treatments.

Position Details:

As part of the Center for Data Analytics, Innovation, and Rigor team, you will report to Technical Project Manager, you will lead the implementation and validation of evaluation frameworks for assessing both AI systems in clinical and mental health research contexts, and the symptoms of their users. You will own the distillation of clinical definitions into evaluation rubrics, combining applied clinical science with evaluation operations to ensure that rubrics are clinically grounded and operationally scalable. You will work closely with both the clinical and data science teams, ensuring knowledge is effectively operationalized to the satisfaction of both specializations. Supervising a small team of data science and software professionals, you will establish reliability standards, oversee the design of metrics for use in multi-rater evaluation systems, and ensure scientific rigor through quality assurance, versioning, and comprehensive documentation.

This is a full-time, exempt position. A hybrid schedule will be required for employees based in the New York Tri-State area or California Bay Area, with a minimum of four (4) days per week in the office, on a schedule determined by your supervisor. Remote positions will be considered for candidates located outside of these regions . The in-office requirement and schedule are subject to change based on the needs of the program and the organization.

 Please note: We are not currently able to employ individuals residing in Ohio, North Dakota, Washington, or Wyoming.

You Will:

• Translate clinical symptom evaluations and manualized therapeutic protocols into reliable, consistent, precise, and scorable evaluation criteria.
• Design and implement metadata standards and protocols for multi-rater clinical evaluation systems, enabling crowd-sourced ratings from diverse clinical experts.
• Set reliability standards and adjudication logic for resolving disagreements among clinical annotators, ensuring consistency across large-scale annotation efforts.
• Establish and enforce quality assurance processes, versioning systems, and comprehensive documentation for all evaluation rubrics and measurement frameworks.
• Contribute to scientific publications, white papers, and technical documentation of measurement methodologies and validation studies.
• Collaborate with clinical leadership to ensure evaluation criteria accurately reflects clinical best practices, safety standards, and clinician feedback.
• Supervise and mentor a small team of data analysts and engineers, providing guidance on evaluation methodologies, statistical analysis, and rubric implementation.
• Collaborate with other technical project stakeholders, to ensure that technical definitions and requirements of all curated rubrics and resources are being met.
  

You Have:

• PhD, PsyD, MD, or equivalent experience in measurement science, psychometrics, biostatistics, clinical psychology, epidemiology, or related field
• 8+ years of applied evaluation experience in clinical research, health outcomes research, or psychometric assessment
• 5+ years of experience with patient evaluation, clinical rating scales, or performance assessment frameworks
• 5+ years of experience in at least one scientific programming language (e.g., Python/R, Matlab) and related toolboxes or frameworks (e.g., Tidyverse, Scipy, Sklearn, Polars, Pytorch) is required. Experience can be concurrent with education.
• 5+ years of experience with applied statistics, including reliability analysis, classical test theory, item response theory, or factor analysis.
• Demonstrated leadership of multi-rater or benchmark systems.
  

#LI-hybrid #LI-remote

Our Benefits

Our great compensation package and benefits include medical insurance, 401(k), paid parental leave, dependent care, flexible work schedules, discounted tickets and entertainment perks programs. For more information about our benefits, please visit our employee benefits website.

Pay Range

The salary range for the position is posted. Factors such as candidate's work experience, education/training, job-related skills, internal peer equity, as well as market and business considerations affect the salary offered within this range. In addition, this salary may be subject to a geographic adjustment (according to a specific city and state and depending on the role), if an authorization is granted to work outside of the location listed in this posting.

EEO Disclaimer

Child Mind Institute is committed to fostering an inclusive and equitable workplace where all individuals are treated with respect and dignity. We are proud to be an equal opportunity employer and prohibit discrimination and harassment of any kind.

We provide equal employment opportunities to all employees and applicants for employment, regardless of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), sexual orientation, gender identity, gender expression, age, national origin, ancestry, citizenship status, marital status, military or veteran status, physical or mental disability, genetic information, medical condition, or any other characteristic protected by applicable federal, state, or local laws.

In compliance with California law, we also prohibit discrimination based on reproductive health decision-making, status as a victim of domestic violence, sexual assault, or stalking, or any other category protected by the California Fair Employment and Housing Act (FEHA). In New York, we extend this prohibition to include status as a victim of domestic violence, familial status, or any other characteristic protected by the New York State Human Rights Law (NYSHRL).

Child Mind Institute is dedicated to ensuring accessibility and reasonable accommodations for individuals with disabilities or medical conditions. If you require an accommodation to participate in the application process or perform your job, please contact our HR Department at [email protected]

This policy applies to all aspects of employment, including recruitment, hiring, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, benefits, and training.