Sr. Data Engineer/Tech Lead

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

As a Senior Data Engineer, you will :

• demonstrate expert skills in ETL/ELT, data integration, ML Ops, and SQL, as well as intermediate to advanced skills in Python, Pyspark, AI/ML, and data visualization.
• demonstrate the ability to review, optimize, document, and mentor data/visualization engineers on data pipelines, mapping, cleansing, and visual design using various tools and platforms.
• possess ability to break down moderately complex problems to implement for increased business impact
• support other team members and helps them to be successful. Actively shares learnings with team members
• drive and enforce the team process improvements, ensuring others are brought along in understanding the benefits and tradeoffs
• actively promote new and innovative ideas across multiple teams and capabilities

Key Responsibilities

Hands-On Development (75%)

• Build, and maintain scalable data platforms and infrastructure on AWS
• Implement end-to-end data pipelines for batch and real-time data processing
• Build robust ETL/ELT workflows to ingest, transform, and load data from diverse sources
• Implement data lake/Lakehouse architectures using AWS S3, Glue, Athena, and Lake Formation
• Design and optimize data warehouse solutions (Redshift, Snowflake) for analytics and reporting
• Establish data quality frameworks and automated monitoring systems
• Write production-quality Python code for data processing, transformation, and automation
• Build scalable data pipelines using Apache Airflow, AWS Step Functions, or similar orchestration tool
• Develop streaming data solutions using Kinesis, Kafka, or AWS MSK
• Optimize SQL queries and database performance for large-scale datasets
• Implement data validation, cleansing, and quality checks
• Build APIs and microservices for data access and integration
• Create monitoring, alerting, and observability solutions for data pipelines
• Debug and resolve data pipeline failures and performance bottlenecks

Technical Leadership & Collaboration (25%)

• Mentor junior and mid-level data engineers through code reviews and technical guidance
• Establish best practices for data engineering, testing, and deployment
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Work with ML engineers to build data pipelines supporting machine learning workflows
• Partner with platform/infrastructure teams on cloud architecture and cost optimization
• Lead technical design discussions and architectural reviews
• Document data architectures, pipelines, and processes
• Evangelize data engineering best practices across the organization

Required Qualifications

Technical Expertise

• 10+ years of professional experience in data engineering or related roles
• Expert-level proficiency in Python for data engineering:
  • Data processing libraries: Pandas, PySpark, Dask, Polars
  • API development: FastAPI, Flask
  • Testing: Pytest, unittest
• Strong AWS expertise with hands-on experience in:
  • Data Storage: S3, RDS/Aurora, DynamoDB, Redshift
  • Data Processing: Glue (ETL jobs, crawlers, Data Catalog), EMR, Athena
  • Streaming: Kinesis (Data Streams, Firehose, Analytics), MSK (Managed Kafka)
  • Orchestration: Step Functions, EventBridge, Lambda
  • Analytics: QuickSight, Athena, Redshift Spectrum
  • Data Lake: Lake Formation, Glue Data Catalog
  • Infrastructure: CloudFormation, CDK, IAM, VPC, CloudWatch
• Workflow Orchestration:
  • Apache Airflow (strong preference)
• Big Data Technologies:
  • Apache Spark (PySpark) for distributed data processing
  • Experience with EMR, Databricks, or similar platforms
  • Understanding of distributed computing concepts
  • Parquet, Avro, ORC file formats

Architecture & Design

• Solid understanding and implementation knowledge of data modelling (dimensional modelling, star/snowflake schemas)
• Experience with both batch and streaming data processing patterns
• Knowledge of data lake, data warehouse, and lake-house architectures
• Understanding of data partitioning, bucketing, and optimization strategies
• Expertise in designing for data quality, lineage, and governance

DevOps & Best Practices

• Strong experience with CI/CD pipelines for data engineering (GitHub Actions, GitLab CI, Jenkins)
• Infrastructure as Code using Terraform, CloudFormation, or AWS CDK
• Containerization with Docker; experience with ECS/Fargate/Kubernetes is a plus
• Git version control and branching strategies
• Monitoring and observability tools: CloudWatch, Grafana
• Data pipeline testing strategies and frameworks

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field (or equivalent experience)
• Experience in regulated industries (healthcare/pharma, finance, government) with compliance requirements
• Hands-on experience with:
  • Additional AWS services: Glue DataBrew, AppFlow, Data Pipeline, Lambda, SageMaker
  • Streaming platforms: Apache Kafka, Confluent, AWS MSK
  • Data quality tools: Great Expectations, dbt, Monte Carlo, Bigeye
  • Data cataloging: AWS Glue Data Catalog, Alation, Collibra
  • Alternative clouds: GCP (BigQuery, Dataflow), Azure (Synapse, Data Factory)
  • Data orchestration: dbt for transformation workflows
• Experience with clinical data, life sciences, or statistical computing domains (CDISC standards, clinical trials data)
• Knowledge of data mesh or data fabric architectures
• Experience building data platforms for ML/AI workloads
• Familiarity with data governance and metadata management frameworks

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly