Lead Data Engineer

Job Title:  Lead Data Engineer

Position Summary:

The Lead Data Engineer is both a technical authority and individual contributor for delivery, and continuous improvement of healthcare data products and platform components. With a proven track record of architecting secure, high-throughput pipelines and directing the work of high-performing teams (4-5 engineers), they are responsible for translating business strategy into data engineering execution. They guide the team in the adoption of tools like AWS Glue, Iceberg, SageMaker, and feature flag frameworks while ensuring HIPAA and HITRUST compliance.  They will lead in best practices for ingesting/cleaning/validating client data and orchestrate a streamlined implementation process to onboard new clients and increase the efficiency of existing ones.

Key Duties and Responsibilities:

• Lead the creation of automated data validation frameworks to ensure client data integrity is at the level needed for our solutions
• Follow best practices and coding standards to create custom scripts to clean/map/align client data to the expected inputs for the ETL pipelines of various solutions
• Architect and lead the adoption of serverless data platforms, integrating services like AWS Glue, Lambda, DynamoDB, and Lakehouse Formation.
• Lead GitOps strategy for infrastructure and pipeline deployment, ensuring compliance, rollback safety, and peer-reviewed workflows.
• Establish best practices for feature flag toggling strategies in critical data pipelines.
• Drive internal knowledge sharing and technical writing culture, including mentoring junior staff on documentation standards.
• Thrives in collaborative team environments and takes pride in mentoring junior team members by providing guidance, support, and best practices.
• Adept at designing scalable architectural solutions while effectively communicating complex concepts to both technical and non-technical audiences. Skilled in tailoring messaging and presentation style based on audience needs to ensure clarity, alignment, and impact across stakeholders.
• Owns technical environment and tooling strategy across the data engineering organization. This role ensures standards, documentation, and collaboration practices are in place and consistently applied.
• GitHub (repo standards)
• AWS Console (Glue, Iceberg, EMR, IAM governance)
• Jupyter/Python IDE (advanced notebook-based development)
• Lucidchart (platform-wide architecture)
• Microsoft Teams/Zoom (engineering leadership, stakeholder updates)
• Jira & Confluence (governance documentation, delivery tracking)

Education

• Bachelor’s degree in Computer Science, Engineering, or a related field of study, or equivalent experience

Experience

• 10+ years in data engineering roles.
• Extensive experience working with large-scale data systems and cloud-based architectures (AWS).
• Prior experience with healthcare or regulated datasets strongly preferred.

Technical Skills

• Python, PySpark, Scala (10+ yrs) – Directed large-scale data platform development with an emphasis on modularity and performance.
• Airflow (6+ yrs) – Defined standards for DAG design, observability, and reusability across the team.
• AWS (Glue, EMR, Lambda, Step Functions) (5+ yrs) – Oversaw hybrid orchestration strategies using serverless and EMR-based pipelines.
• Iceberg (4–5 yrs) – Led lakehouse strategy including governance, versioning, and cost optimization.
• Deep expertise in designing and optimizing data lake and lakehouse architectures at scale, including governance, schema evolution, cost efficiency, and access control.
• Leads strategy and implementation of hybrid lakehouse ecosystems leveraging technologies like Iceberg, Delta Lake, and catalog services across multiple environments (e.g., dev, staging, prod).
• Directed team of data engineers of 5–8 engineers, driving architecture decisions, enforcing engineering standards, and championing Agile delivery practices within a high-performance data engineering environment.
• Docker/Kubernetes (6–8 yr) – Oversaw Kubernetes-based orchestration for enterprise data pipelines; established Docker standards across teams and integrated container practices with compliance and DevOps strategies.

This position provides work direction only

Knowledge, Skills, and Abilities :               

• Excellent written, verbal, and interpersonal communication skills.
• Excellent organizational skills and attention to detail.
• Excellent documentation skills.
• Team player.
• Ability to provide clear and accurate information through multiple media.
• Ability to extrapolate, from historical trends, future volumes, and staffing needs.
• Ability to manage open requests and follow up when necessary, without outside direction.
• Ability to manage time effectively with strong attention to detail.
• Ability to read and interpret documents including safety rules, operating and maintenance instructions, procedure manuals and general correspondence.
• Ability to write routine reports and correspondence.
• Ability to deal with problems involving several concrete variables in standardized situations.
• Ability to interact politely, tactfully, and firmly with a wide range of people and personalities.
• Ability to work in an environment with potential interruptions.
• Ability to manage multiple simultaneous tasks with individual timeframes and priorities.
• Ability to motivate and encourage a team of front-line employees.
• Ability to set and manage to expectations for a team.

Compensation and Benefits:

The expected base salary for this position ranges from $125,000 – $145,000 USD with a bonus target of up to 5% of the base salary. At Convey, we believe that the right benefits package can improve the quality of our employees and their family members’ lives. Therefore, we offer a competitive total rewards package, which includes health and welfare benefits, 401(K) savings with employer matching, paid time off, life, AD&D and disability insurances, and various ancillary benefits, such as pet insurance, virtual physical therapy, and employee assistance programs.

Equal Employment Opportunity Statement

Convey Health Solutions is an Equal Opportunity Employer committed to fostering an inclusive and diverse workforce. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, veteran status, or any other characteristic protected by applicable federal, state, or local laws.

Convey Health Solutions also provides reasonable accommodations to qualified individuals with disabilities in accordance with applicable laws. Applicants requiring accommodation during the application or interview process should contact the Human Resources department.