The Role
Leads the design, development, optimization, governance, and support of batch and streaming data pipelines using Azure Databricks, PySpark, Delta Lake, and related Azure services. Builds data models, quality frameworks, CI/CD and DataOps processes, and secure governed data solutions. Troubleshoots production issues, drives performance and cost improvements, collaborates with technical and business stakeholders, and mentors junior engineers. Healthcare data, streaming, infrastructure-as-code, certifications, and modernization experience are preferred.
Summary Generated by Built In
Requisition Number: 2394971
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Primary Responsibilities:
Required Qualifications:
Preferred Qualifications:
At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Primary Responsibilities:
- Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
- Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory, Auto Loader and Lakeflow Spark Declarative Pipelines
- Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
- Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
- Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection
- Build data quality, validation and reconciliation frameworks to keep data accurate and reliable
- Tune Spark jobs and Delta tables for performance and cost (partitioning, liquid clustering, Z-ordering, OPTIMIZE/VACUUM, compute sizing)
- Drive CI/CD and DataOps practices using Azure DevOps or GitHub, Declarative Automation Bundles and automated testing
- Troubleshoot production issues, do root cause analysis and put preventive fixes in place
- Work with architects, analysts, data scientists and business stakeholders, and mentor junior engineers
- Use AI-assisted engineering tools to improve development, testing and documentation
- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Required Qualifications:
- B.E./B.Tech./MCA or equivalent degree in Computer Science, IT or Engineering
- 6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks
- Hands-on experience with Delta Lake, Lakehouse architecture and Unity Catalog
- Experience working with file formats such as CSV, JSON, XML, Parquet, Avro and Delta, including schema inference, schema evolution and nested data handling
- Experience with Azure services: ADF, ADLS Gen2, Key Vault, Azure SQL , Power Bi
- Solid knowledge of data modeling, ETL/ELT design, Git, CI/CD and Agile delivery
- Good understanding of views, temporary/global temporary views, materialized views, streaming tables and when to use each
- Solid proficiency in Python, PySpark and SQL, with a solid understanding of Spark internals and performance tuning
- Proven solid problem-solving and communication skills
Preferred Qualifications:
- Databricks Data Engineer Associate/Professional or Microsoft DP-203/DP-700 certification
- Experience with Structured Streaming, Kafka/Event Hubs, Lakeflow Jobs and Databricks SQL
- Infrastructure as Code experience
- Exposure to healthcare data (claims, prior authorization, HL7/FHIR, EDI 837/278) and HIPAA compliance
- Experience with Power BI or AI/BI dashboards
- Experience mentoring or leading a team, or driving cloud migration and modernization
At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.
Skills Required
- B.E., B.Tech., MCA, or equivalent degree in Computer Science, IT, or Engineering
- 6+ years of data engineering experience
- 3+ years of hands-on Azure Databricks experience
- Hands-on experience with Delta Lake, Lakehouse architecture, and Unity Catalog
- Experience with CSV, JSON, XML, Parquet, Avro, and Delta formats, including schema inference, schema evolution, and nested data handling
- Experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure SQL, and Power BI
- Knowledge of data modeling, ETL/ELT design, Git, CI/CD, and Agile delivery
- Understanding of views, temporary and global temporary views, materialized views, and streaming tables
- Proficiency in Python, PySpark, and SQL, with understanding of Spark internals and performance tuning
- Strong problem-solving and communication skills
- Databricks Data Engineer Associate or Professional certification, or Microsoft DP-203 or DP-700 certification
- Experience with Structured Streaming, Kafka or Event Hubs, Lakeflow Jobs, and Databricks SQL
- Infrastructure as Code experience
- Exposure to healthcare data such as claims, prior authorization, HL7/FHIR, or EDI 837/278, and HIPAA compliance
- Experience with Power BI or AI/BI dashboards
- Experience mentoring or leading a team, or driving cloud migration and modernization
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Optum, part of the UnitedHealth Group family of businesses, is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together. At Optum, we support your well-being with an understanding team, extensive benefits and rewarding opportunities. By joining us, you’ll have the resources to drive system transformation while we help you take care of your future. We recognize the power of connection to drive change, improve efficiency and make a difference in health care. Join a team where your skills and ideas can make an impact and where collaboration is key to creating technology that produces healthier outcomes.
Gallery
.jpg)
Optum Offices
Hybrid Workspace
Employees engage in a combination of remote and on-site work.
Optum has three workplace models that balance the needs of the business and the responsibilities of each role. These models, core on‑site (5 days/week), hybrid (4 days/week) and telecommute or fully remote, vary by country, role and location.
Typical time on-site:
Not Specified
