The Role
Tests, evaluates, and optimizes enterprise generative AI applications, agents, workflows, prompts, models, and architectures. Builds automated evaluation pipelines, benchmarks, regression tests, monitoring, telemetry, and governance controls. Develops multi-agent workflows using LangGraph and LangChain, evaluates model versions, and supports documentation, traceability, model registries, and experiment tracking. Collaborates with engineering, data science, security, compliance, governance, and business stakeholders.
Summary Generated by Built In
Care Lumen | Remote (US Only)
Employment Type: Contractor (3-6 months, with possibility of extension)
Salary Range: $129 to $150 an hour
Scope
The Senior Generative AI Engineer is an individual contributor responsible for testing, evaluating, optimizing, and monitoring enterprise Generative AI systems. The primary focus is improving the quality, reliability, and performance of existing GenAI solutions through systematic testing, evaluation of new model versions, prompt optimization, and application architecture improvements. The role will also help drive evaluation methodology, monitoring capabilities, and AI governance requirements for selected initiatives. Approximately 25% of the role is expected to support AI governance activities, including model documentation, versioning, registry management, experiment tracking, auditability, and related controls.
The role collaborates with engineering, data science, architecture, security, compliance, governance, and business stakeholders. It does not have direct reports or a formal leadership mandate.
What You’ll Do
- Test and evaluate GenAI applications, AI agents, and LLM-powered workflows for quality, reliability, accuracy, safety, and task effectiveness.
- Develop and execute repeatable GenAI evaluation methodologies, test datasets, benchmark suites, regression tests, and automated evaluation pipelines.
- Use DeepEval and related evaluation approaches to measure GenAI output quality and identify regressions across prompts, workflows, application changes, and model versions.
- Optimize GenAI solution architecture by decomposing complex prompts and workflows into appropriate subtasks, tools, and specialized agents.
- Build and improve multi-agent workflows using orchestration frameworks such as LangGraph and LangChain.
- Optimize prompts, system instructions, context management, agent interactions, and model selection / configurations using evaluation results (including latency and model performance) and production feedback (including user feedback).
- Evaluate new foundation models and model versions to determine quality, behavior, risks, performance, (groundedness, completeness, bias, etc) and suitability for specific use cases.
- Design and implement monitoring capabilities for GenAI systems, including quality metrics, operational telemetry, audit trails, alerts, and performance reporting.
- Help drive AI governance activities for selected initiatives, including model and application documentation, risk assessments, evaluation evidence, model versioning, registry management, experiment tracking, and change traceability.
- Collaborate across technical, governance, security, compliance, and business teams to improve GenAI solutions and support responsible production use.
Competencies
- GenAI testing and quality evaluation
- GenAI evaluation methodology and automated evaluation
- Prompt, context, and application architecture optimization
- Multi-agent architecture and orchestration
- Foundation model and model-version testing
- AI governance and model lifecycle management
- GenAI monitoring, observability, and traceability
- Traditional machine learning testing and validation
- Analytical problem-solving and cross-functional collaboration
What’s Required
- 10+ years of overall professional experience in data science, machine learning, AI engineering, or a related technical field.
- 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems.
- Demonstrated experience testing GenAI applications and establishing systematic methods for assessing LLM output quality.
- Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries.
- Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows.
- Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking.
- Practical experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability.
- Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with a strong understanding of model testing and validation methods.
- Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders.
- Ability to maintain at least four hours of overlap with U.S. Eastern Time (EST/EDT) each workday.
- Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline.
What’s Preferred
- Experience with Databricks Mosaic AI.
- Experience working with PHI / PII data inside highly regulated domain like healthcare.
- Experience testing or optimizing multi-agent GenAI systems in production or production-like environments.
- Experience working with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures.
- Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment.
Compensation & Benefits
The expected total cash compensation for this role is market-driven, with an expected compensation rate of $129 to $150 per hour. Hourly rate is ultimately determined based on a candidate's skills, expertise, and experience.
Offers are contingent upon successful completion of a background screening. Background screenings will be conducted in compliance with applicable laws and will not be initiated without your consent.
Care Lumen is an equal opportunity employer committed to building a diverse and inclusive workplace. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other legally protected status.
We are committed to providing reasonable accommodations to qualified individuals with disabilities in our job application process. If you need assistance or an accommodation due to a disability, please contact us at [email protected].
About Care Lumen
Care Lumen is transforming healthcare by connecting people, care, and data—so that every decision leads to better outcomes. We empower care teams with a unified platform that delivers intelligent, real-time insights, streamlined workflows, and stronger collaboration across clinical, operational, and administrative domains.
Our vision is bold: to reimagine how care is delivered by unifying clinical operations and unlocking insights that improve lives. We believe in the power of great people and purposeful technology—and with all of us aligned around a common goal, our potential grows even stronger. We're building for impact today and evolving through innovation tomorrow.
We’re a remote-first company that values flexibility, autonomy, and results over hours logged. We're guided by values that push us to excel, stay curious, and take smart risks. The best solutions come from teams that listen, learn, and build together. And we're focused on outcomes that matter—not just for our clients, but for the people they serve.
Care Lumen’s LinkedIn
Care Lumen’s website
Ready to Apply?
We are committed to providing reasonable accommodations to qualified individuals with disabilities in our job application process. If you need assistance or an accommodation due to a disability, please contact us at [email protected].
About Care Lumen
Care Lumen is transforming healthcare by connecting people, care, and data—so that every decision leads to better outcomes. We empower care teams with a unified platform that delivers intelligent, real-time insights, streamlined workflows, and stronger collaboration across clinical, operational, and administrative domains.
Our vision is bold: to reimagine how care is delivered by unifying clinical operations and unlocking insights that improve lives. We believe in the power of great people and purposeful technology—and with all of us aligned around a common goal, our potential grows even stronger. We're building for impact today and evolving through innovation tomorrow.
We’re a remote-first company that values flexibility, autonomy, and results over hours logged. We're guided by values that push us to excel, stay curious, and take smart risks. The best solutions come from teams that listen, learn, and build together. And we're focused on outcomes that matter—not just for our clients, but for the people they serve.
Care Lumen’s LinkedIn
Care Lumen’s website
Ready to Apply?
About
Newfire Global Partners is a leading custom software development company that specializes in working on complex problems. With a strong focus on meeting the unique needs of each organization, they connect clients with local leaders and trusted remote teams to deliver exceptional IT services and software development solutions. Their range of expertise spans various domains, including Data & Analytics, AI/ML, Software Engineering, Product Management, and Marketing. By leveraging recent innovations in artificial intelligence, Newfire Global Partners ensures that their clients can transform the new normal into a competitive advantage. Through their advisory services, experienced leaders guide businesses through transformations and de-risk operations throughout the entire lifecycle. They also offer talent augmentation, bringing together highly skilled professionals from multi-disciplinary teams to support software, product, and marketing needs. Newfire Global Partners is committed to providing excellent service and driving client growth. Their team of industry experts works closely with organizations, whether it's driving transformation, building from scratch, or getting things done efficiently. Trusted by VC & PE firms, established businesses, and startups, Newfire Global Partners is proud to be a reliable partner in various industries, including healthcare, enterprise, and fintech. Partner with Newfire Global Partners today and unlock the expertise that accelerates success.
Skills Required
- 10+ years of professional experience in data science, machine learning, AI engineering, or a related technical field
- 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems
- Experience testing GenAI applications and establishing systematic methods for assessing LLM output quality
- Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries
- Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows
- Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking
- Experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability
- Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with strong understanding of model testing and validation
- Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders
- Ability to maintain at least four hours of overlap with U.S. Eastern Time each workday
- Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline
- Experience with Databricks Mosaic AI
- Experience working with PHI or PII data inside a highly regulated domain such as healthcare
- Experience testing or optimizing multi-agent GenAI systems in production or production-like environments
- Experience with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures
- Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment
Similar Jobs
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Newfire Global Partners is a technology and advisory firm founded in 2016. They specialize in software development, staff augmentation, and strategic advisory services, helping organizations navigate digital transformation. Their work spans high-impact industries, with a particular emphasis on digital healthcare, fintech, and cybersecurity. They focus on solving complex challenges through product-led thinking, business strategy, and the practical application of AI to drive measurable growth and operational success.
