LLM Ops Engineer

Job Description

Join Our Team at Litera: Where Legal Technology Meets Excellence

Litera has been at the forefront of legal technology innovation for over 25 years, crafting legal software to amplify impact and maximize efficiency. Developed by the best legal minds in the industry, our comprehensive suite of integrated legal tools is both powerful and user-friendly and simplifies the way modern firms manage core legal workflows, secure collaboration, and organize firm knowledge and experience. Every day, we help more than 2.3 million legal professionals focus on their craft. Litera: Less busy work, more of your life’s work.

About the Role

We are seeking an experienced LLM Engineer to join our newly established LLM Ops Team. In this role, you will be responsible for managing the complex lifecycle of Large Language Models from development to deployment, monitoring, and continuous improvement.

Key Responsibilities

Model Development & Training

• Fine-tune pre-trained models for specific use cases

• Curate and prepare datasets for training

• Manage training infrastructure, resources, and computational environments

• Implement optimization techniques to improve model performance

Deployment & Serving

• Develop and manage APIs for model serving

• Scale infrastructure to handle varying demand loads

• Build and maintain the GenAI middleware/sidecar layer

• Integrate LLMs with existing systems and data sources

Monitoring & Evaluation

• Track performance metrics including latency and throughput

• Monitor quality metrics such as hallucination rates and accuracy

• Optimize costs associated with model inference and training

• Create and maintain dashboards for real-time performance insights

Testing & Quality Assurance

• Create and maintain golden datasets for benchmark testing

• Implement statistical validation methods for model outputs

• Set up similarity matching criteria for response evaluation

• Develop confidence score thresholds for production systems

Feedback Loops & Iteration

• Design and implement user feedback collection systems

• Establish continuous improvement processes

• Create A/B testing frameworks for model and feature evaluation

• Conduct trace analysis to identify areas for performance optimization

Safety & Compliance

• Implement content moderation systems

• Detect and mitigate bias in model outputs

• Ensure regulatory compliance in AI systems

• Develop output validation frameworks

Prompt Management

• Version and store prompts systematically

• Create and maintain prompt templates

• Set up playground environments for prompt testing

• Abstract prompts from application code for better maintainability

Qualifications

• Experience with LLM development, fine-tuning, and deployment

• Strong programming skills, particularly in Python

• Experience with Kubeflow, Apache Airflow, or other LLM Pipeline technology

• Experience with Azure OpenAI, AWS Sagemaker, and/or Vertex AI

• Understanding of machine learning operations and MLOps principles

• Knowledge of infrastructure scaling and optimization

• Experience with AI monitoring tools and dashboard creation

• Familiarity with AI safety, bias detection, and compliance requirements

• Strong problem-solving abilities and analytical thinking

• Familiarity with ISO 27001 and SOC2 Certification

Join Our Team

Be part of building the future of AI operations as we establish robust systems for managing the entire LLM lifecycle. This role offers the opportunity to work at the cutting edge of artificial intelligence implementation and operations.

Why Join Litera?

• The company culture: We emphasize helping each other grow, doing the right thing always, and being part of a journey to amplify impact, creating an exciting and fulfilling work environment

• Commitment to Employees: Our people commitment is based on what employees love most about being part of the team, focusing on tools that matter to the difference-makers in the legal world and amplifying their impact

• Global, Dynamic, and Diverse Team: Our is a global company with ambitious goals and unlimited opportunities, offering a dynamic and diverse work environment where employees can grow, listen, empathize, and problem-solve together

• Comprehensive Benefits Package: Experience peace of mind with our health insurance, retirement savings plans, generous paid time off, and a supportive work-life balance. We invest in your well-being and future, ensuring a rewarding career journey.

• Career Growth and Development: We provide career paths and opportunities for professional development, allowing employees to progress through various technical and leadership roles

California, Colorado, Connecticut, Illinois, Maryland, Nevada, New Jersey, New York, Ohio, Rhode Island, or Washington Residents Only: The salary range for California, Colorado, Connecticut, Illinois, Maryland, Nevada, New Jersey, New York, Ohio, Rhode Island, or Washington residents is $100,000 to $132,000. Pay is based on several factors including but not limited to education, work experience, certifications, etc. In addition to your salary, Litera offers benefits such as a comprehensive benefits package, incentive and recognition programs, and 401k contribution (all benefits are subject to eligibility requirements).