Senior Data Engineering Lead

Posted Yesterday
Be an Early Applicant
Pune, Mahārāshtra, IND
In-Office
Senior level
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
The Role
Leads development of production-grade healthcare data platforms and resilient batch, micro-batch, and event-driven pipelines. Builds ingestion frameworks, Spark/PySpark transformations, Delta Lake and Iceberg tables, healthcare data models, quality controls, metadata lineage, privacy-aware PHI processing, and governed data consumption. Owns CI/CD, infrastructure as code, testing, observability, monitoring, incident recovery, and performance optimization across AWS and Databricks.
Summary Generated by Built In
Requisition Number: 2384847
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Primary Responsibilities:
  • Build reliable batch, micro-batch, and event-driven pipelines on AWS and Databricks
  • Develop reusable ingestion frameworks for REST APIs, FHIR Bulk Export, HL7 interfaces, databases, SFTP/file exchange, JSON/NDJSON, CSV, XML, PDFs, and clinical text
  • Implement scalable Spark/PySpark and SQL transformations, including schema inference/evolution, checkpointing, idempotency, retries, backfills, and replay
  • Design and maintain Delta Lake and Apache Iceberg tables, including physical design, partitioning/clustering, compaction, file sizing, incremental reads/writes, performance tuning, and cost optimization
  • Build curated healthcare data models and transformations using FHIR, HL7, OMOP CDM, and clinical terminology mappings
  • Implement data-quality frameworks: schema validation, referential-integrity checks, business-rule testing, anomaly detection, reconciliation, completeness checks, and data-quality observability
  • Build metadata and lineage capture across source systems, pipeline runs, code versions, transformation rules, mappings, and published data products
  • Implement privacy-aware data processing for PHI, including access controls, masking, tokenization/pseudonymization, de-identification, and auditable handling patterns
  • Deliver CI/CD pipelines, automated unit/integration/data tests, Terraform or CloudFormation, containerized services, monitoring, alerting, runbooks, and incident-recovery procedures
  • Integrate and optimize governed data consumption through Databricks SQL, Athena, Snowflake, Trino, or equivalent engines
  • Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Eligibility
To apply to an internal job, employees must meet the following criteria:
  • The candidate should have completed 12 months in the current role
  • The candidate should not be on any active CAP/ PIP
  • The performance review of the candidate must be ME & Above in the last common review

Required Qualifications:
  • Graduate degree or equivalent experience
  • 5+ years of production data-engineering experience
  • Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS
  • Experience with Git, pull requests, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and incident response
  • Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability
  • Production Databricks experience with Auto Loader, Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster/job optimization
  • Advanced Python, SQL, and Apache Spark/PySpark; strong understanding of distributed processing and performance tuning
  • Practical Apache Iceberg knowledge, including tables, catalogs, snapshots, schema/partition evolution, compaction, and interoperability with query engines
  • Healthcare data knowledge: FHIR R4 and/or HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned engineering controls, and de-identification concepts
  • Demonstrated responsible use of AI coding tools and the ability to critically review, test, and productionize generated code

Required Qualifications:
  • Healthcare or regulated-data experience

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.

Skills Required

  • Graduate degree or equivalent experience
  • 5+ years of production data-engineering experience
  • Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS
  • Experience with Git, pull requests, CI/CD, Docker, Terraform or CloudFormation, automated testing, observability, and incident response
  • Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability
  • Production Databricks experience with Auto Loader, Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster/job optimization
  • Advanced Python, SQL, and Apache Spark/PySpark knowledge, including distributed processing and performance tuning
  • Practical Apache Iceberg knowledge, including tables, catalogs, snapshots, schema and partition evolution, compaction, and query-engine interoperability
  • Healthcare data knowledge including FHIR R4 and/or HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned controls, and de-identification
  • Demonstrated responsible use of AI coding tools and ability to review, test, and productionize generated code
  • Healthcare or regulated-data experience
  • Completed 12 months in the current role
  • Not on an active CAP or PIP
  • Performance review of ME or above in the last common review

What the Team is Saying

Optum Compensation & Benefits Highlights

  • Leave & Time Off Breadth — Paid time off is often regarded as generous, with meaningful accrual and flexibility to use it. While practices can vary by team, the overall time-off offering stands out as a relative strength.
  • Retirement Support — A 401(k) plan with employer matching is available and generally well-regarded for supporting long-term savings. It remains valuable, though candidates should verify the current match formula.
  • Parental & Family Support — Programs such as parental leave, family medical leave, and adoption assistance are included as notable elements of the total rewards package. These resources can meaningfully support work-life balance when applicable to the role and location.

Optum Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Eden Prairie, MN
160,000 Employees
Year Founded: 2011

What We Do

Optum, part of the UnitedHealth Group family of businesses, is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together. At Optum, we support your well-being with an understanding team, extensive benefits and rewarding opportunities. By joining us, you’ll have the resources to drive system transformation while we help you take care of your future. We recognize the power of connection to drive change, improve efficiency and make a difference in health care. Join a team where your skills and ideas can make an impact and where collaboration is key to creating technology that produces healthier outcomes.

Gallery

Gallery
Gallery
Gallery

Optum Offices

Hybrid Workspace

Employees engage in a combination of remote and on-site work.

Optum has three workplace models that balance the needs of the business and the responsibilities of each role. These models, core on‑site (5 days/week), hybrid (4 days/week) and telecommute or fully remote, vary by country, role and location.

Typical time on-site: Not Specified
HQEden Prairie, MN
Metro Manila, Philippines
Cebu, Philippines
Davao, Philippines
Ann Arbor, MI
Atlanta, GA
Baltimore, MD
Bengaluru, India
Chennai, India
Dallas, TX
Detroit, MI
Dublin, Ireland
Hartford, CT
Houston, TX
Hyderabad, India
Jacksonville, FL
Las Vegas, NV
Letterkenny, Ireland
Louisville, KY
Madison, WI
Minneapolis, MN
Nashville, TN
New Delhi, India
Philadelphia, PA
Phoenix, AZ
Pune, India
Raleigh, NC
San Diego, CA
Washington, DC
Learn more

Similar Jobs

Optum Logo Optum

Machine Learning Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Pune, Mahārāshtra, IND
160000 Employees

Optum Logo Optum

Senior Software Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Pune, Mahārāshtra, IND
160000 Employees

Optum Logo Optum

Senior ERP Analyst

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Pune, Mahārāshtra, IND
160000 Employees

Optum Logo Optum

Collections Representative - AR Calling - RCM Denial Management

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Pune, Mahārāshtra, IND
160000 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account