Principal Data Engineer

Posted 4 Hours Ago
Be an Early Applicant
Bengaluru, Bengaluru Urban, Karnataka, IND
In-Office
Senior level
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
The Role
Designs and operates scalable batch and streaming data platforms supporting analytics, machine learning, and generative AI. Builds ingestion, ETL/ELT, orchestration, data quality, observability, governance, and lakehouse solutions. Partners with stakeholders on technical roadmaps, enables AI/ML workflows, improves platform scalability and cost efficiency, reviews engineering work, and mentors engineers.
Summary Generated by Built In
Requisition Number: 2369838
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Functions may include database architecture, engineering, design, optimization, security, and administration; as well as data modeling, big data development, Extract, Transform, and Load (ETL) development, storage engineering, data warehousing, data provisioning and other similar roles. Responsibilities may include Platform-as-a-Service and Cloud solution with a focus on data stores and associated eco systems. Duties may include management of design services, providing sizing and configuration assistance, ensuring strict data quality, and performing needs assessments. Analyzes current business practices, processes and procedures as well as identifying future business opportunities for leveraging data storage and retrieval system capabilities. Manages relationships with software and hardware vendors to understand the potential architectural impact of different vendor strategies and data acquisition. May design schemas, write SQL or other data markup scripting and helps to support development of Analytics and Applications that build on top of data. Selects, develops and evaluates personnel to ensure the efficient operation of the function.
Primary Responsibilities:
  • Design, develop, and maintain scalable data pipelines and data platforms supporting analytics, machine learning, and AI use cases
  • Build and optimize ingestion frameworks for large-scale structured and unstructured data, including streaming and event-driven sources
  • Partner with cross-functional stakeholders to understand evolving data and AI needs and define long-term technical solutions
  • Enable and support machine learning and AI workflows, including feature engineering, data preparation, and model deployment support
  • Drive strategic initiatives around Generative AI, data quality, observability, lineage, and governance
  • Develop and maintain frameworks that support rapid experimentation and deployment of AI/ML solutions
  • Introduce and evolve best practices in data modeling, orchestration, testing, and monitoring
  • Identify and champion opportunities for platform scalability, performance optimization, and cost efficiency
  • Collaborate with product, analytics, and infrastructure teams to deliver high-impact data and AI solutions
  • Build and maintain reusable parsing, enrichment, analytic, and service libraries to accelerate delivery across teams
  • Work comfortably under time-sensitive conditions while ensuring thoroughness
  • Maintain high ethical standards and the ability to remain objective and confidential
  • Reviews the work of others
  • Develops innovative approaches
  • Sought out as expert
  • Serves as a leader/ mentor
  • Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:
  • Bachelor's degree or equivalent experience
  • 5+ years of experience designing, building, and operating production data pipelines and platforms
  • 5+ years of hands-on development with Python (preferred) and/or Java, including code reviews, packaging, and deployment
  • 5+ years of experience with Spark (PySpark) and Databricks (or similar distributed data processing platform)
  • 2+ years of experience leveraging and deploying Generative AI use cases to production environments
  • Cloud experience (AWS, Azure, and/or GCP), including secure handling of sensitive data (PII/PHI) and collaboration with compliance partners
  • Experience building and operating production data platforms and pipelines across batch and streaming workloads
  • Experience with distributed processing and lakehouse/warehouse patterns (eg, Spark/PySpark, Databricks, Snowflake, Microsoft Fabric)
  • Experience building ingestion frameworks for structured and unstructured data, including event/log and semi-structured formats
  • Experience enabling Generative AI solutions in production (eg, RAG-style architectures), including retrieval patterns and evaluation/monitoring practices
  • Experience building ingestion frameworks for structured and unstructured data (e.g., event/log, semi-structured JSON), including parsing and enrichment patterns
  • Experience designing and scaling ELT/ETL frameworks with orchestration tools such as Airflow (or equivalent)
  • Experience implementing data quality, observability, and monitoring practices (e.g., data quality checks, pipeline SLAs/SLOs, alerting)
  • Experience with metadata, lineage, and governance concepts and tooling (e.g., data catalogs, lineage, access controls)
  • Experience with data modeling best practices for analytics and ML use cases
  • Experience with DevOps and CI/CD practices and tools (e.g., GitHub Actions), containerization, and infrastructure-as-code (e.g., Docker, Kubernetes, Terraform)
  • Experience supporting ML/AI workflows (feature engineering, data preparation, and model deployment enablement); exposure to MLOps practices is a plus
  • Solid SQL skills and experience working with data lakes and warehouses (e.g., Databricks, Snowflake)
  • Familiarity with knowledge-centric data approaches (eg, metadata-driven systems, entity resolution, and/or graph concepts) to improve discoverability and downstream analytics
  • Solid data quality, observability, and monitoring mindset (profiling, validation, alerting, and reliability improvements)
  • Comfort with orchestration, CI/CD, containerization, and infrastructure-as-code (eg, Airflow, GitHub Actions, Docker, Terraform, Kubernetes)
  • Demonstrated ability to lead through influence, mentor engineers, and translate ambiguous problems into scalable technical roadmaps
  • Demonstrated ability to partner with cross-functional stakeholders, translate requirements into technical solutions, and lead through influence

Preferred Qualifications:
  • Solid hands-on engineering in Python and SQL; familiarity with JVM languages (Java/Scala) in Spark ecosystems
  • Experience with cloud platforms such as AWS, Azure, or Google Cloud, including managed data services
  • Experience with streaming and event-driven architectures (e.g., Kafka, Kinesis, Event Hubs)
  • Experience with data quality and validation frameworks (e.g., Great Expectations, Deequ) and/or data observability tooling
  • Experience enabling MLOps practices (e.g., feature stores, model registries, experiment tracking, deployment automation)
  • Experience with lakehouse architectures, Delta Lake, and advanced Spark optimization/performance tuning
  • Experience with data visualization tools and libraries
  • Experience with machine learning and predictive analytics
  • Familiarity with security and privacy concepts for data platforms (e.g., least privilege, PII/PHI handling) and working with compliance partners

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.

Skills Required

  • Bachelor's degree or equivalent experience
  • 5+ years designing, building, and operating production data pipelines and platforms
  • 5+ years of hands-on development with Python and/or Java, including code reviews, packaging, and deployment
  • 5+ years of experience with Spark, PySpark, Databricks, or a similar distributed data processing platform
  • 2+ years leveraging and deploying generative AI use cases in production
  • Cloud experience with AWS, Azure, and/or Google Cloud, including secure handling of PII/PHI
  • Experience operating batch and streaming data platforms and pipelines
  • Experience with distributed processing and lakehouse or data warehouse patterns
  • Experience building ingestion frameworks for structured, unstructured, event, log, and semi-structured data
  • Experience enabling production generative AI solutions, including retrieval patterns and evaluation or monitoring practices
  • Experience designing and scaling ETL/ELT frameworks using Airflow or equivalent orchestration tools
  • Experience implementing data quality, observability, monitoring, pipeline SLAs/SLOs, and alerting
  • Experience with metadata, lineage, governance, data catalogs, and access controls
  • Experience with data modeling for analytics and machine learning use cases
  • Experience with DevOps, CI/CD, containerization, and infrastructure-as-code
  • Experience supporting ML/AI workflows, including feature engineering, data preparation, and model deployment enablement
  • Solid SQL skills and experience with data lakes and warehouses
  • Familiarity with metadata-driven systems, entity resolution, or graph concepts
  • Ability to lead through influence, mentor engineers, and translate ambiguous problems into scalable technical roadmaps
  • Ability to partner with cross-functional stakeholders and translate requirements into technical solutions
  • Familiarity with JVM languages such as Java or Scala in Spark ecosystems
  • Experience with managed cloud data services
  • Experience with streaming and event-driven architectures such as Kafka, Kinesis, or Event Hubs
  • Experience with data quality and validation frameworks such as Great Expectations or Deequ
  • Experience enabling MLOps practices, including feature stores, model registries, experiment tracking, or deployment automation
  • Experience with lakehouse architectures, Delta Lake, and advanced Spark optimization
  • Experience with data visualization tools and libraries
  • Experience with machine learning and predictive analytics
  • Familiarity with data platform security, privacy, least privilege, and compliance practices

What the Team is Saying

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Eden Prairie, MN
160,000 Employees
Year Founded: 2011

What We Do

Optum, part of the UnitedHealth Group family of businesses, is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together. At Optum, we support your well-being with an understanding team, extensive benefits and rewarding opportunities. By joining us, you’ll have the resources to drive system transformation while we help you take care of your future. We recognize the power of connection to drive change, improve efficiency and make a difference in health care. Join a team where your skills and ideas can make an impact and where collaboration is key to creating technology that produces healthier outcomes.

Gallery

Gallery
Gallery
Gallery

Optum Offices

Hybrid Workspace

Employees engage in a combination of remote and on-site work.

Optum has three workplace models that balance the needs of the business and the responsibilities of each role. These models, core on‑site (5 days/week), hybrid (4 days/week) and telecommute or fully remote, vary by country, role and location.

Typical time on-site: Not Specified
HQEden Prairie, MN
Metro Manila, Philippines
Cebu, Philippines
Davao, Philippines
Ann Arbor, MI
Atlanta, GA
Baltimore, MD
Bengaluru, India
Chennai, India
Dallas, TX
Detroit, MI
Dublin, Ireland
Hartford, CT
Houston, TX
Hyderabad, India
Jacksonville, FL
Las Vegas, NV
Letterkenny, Ireland
Louisville, KY
Madison, WI
Minneapolis, MN
Nashville, TN
New Delhi, India
Philadelphia, PA
Phoenix, AZ
Pune, India
Raleigh, NC
San Diego, CA
Washington, DC
Learn more

Similar Jobs

Optum Logo Optum

Machine Learning Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Bengaluru, Bengaluru Urban, Karnataka, IND
160000 Employees

Optum Logo Optum

Machine Learning Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Bengaluru, Bengaluru Urban, Karnataka, IND
160000 Employees

Optum Logo Optum

Senior Quality Engineering Lead

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Bengaluru, Bengaluru Urban, Karnataka, IND
160000 Employees

Optum Logo Optum

Software Engineer

Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
In-Office
Bengaluru, Bengaluru Urban, Karnataka, IND
160000 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account