Top Senior Data Engineer Jobs in San Diego, CA

Design and implement data federation and lakehouse architectures, build scalable ETL/ELT pipelines with Python and Spark, optimize performance across federated queries, manage Delta/Iceberg/Hudi tables, and enforce governance, security, and access controls for analytics and AI teams.

Design and develop Big Data applications using Java, Spark, and MapReduce on Cloudera/Hadoop ecosystems. Work with Hive, Impala, YARN, Kafka, and large datasets; write tests (JUnit), perform data analysis, and script in Unix/Python. Lead and manage global technology teams and follow industry best practices.

Design and implement big data solutions (Spark, Hive, Java, CDP). Analyze and consolidate disparate data sources, produce functional specifications, review data models, gather stakeholder requirements, validate implementations, support production deployments, investigate data quality and data lineage, and collaborate with technology leads to ensure data completeness and accuracy.

Design, build, test, and maintain high-performance Python data applications and backend services. Lead technical design, review code, mentor junior engineers, integrate with databases and cloud services, optimize performance, and ensure security, scalability, and reliability.

Design, develop, maintain, and optimize enterprise Tableau dashboards and BI solutions. Integrate and cleanse data from Oracle/Sybase/Big Data, tune SQL, automate Tableau extracts, manage Tableau Server on Linux, support testing, UAT, deployments, and troubleshoot data integrity and performance issues.

Design, develop, and maintain PySpark applications and ETL pipelines to process, transform, and integrate large-scale datasets from SQL, NoSQL, data lakes, and streaming sources. Optimize Spark job performance, implement robust error handling, and collaborate with data analysts, scientists, and architects using orchestration tools like Airflow or Luigi.

Design and build a modern, scalable data platform (warehouse, lakehouse, or hybrid) to support analytics, data science, and near-real-time processing. Lead platform architecture, CDC-based ingestion, ETL/ELT pipelines, data quality/observability, and governance. Partner cross-functionally, provide technical leadership and mentorship, and modernize legacy data processes for low-latency enterprise reporting and ML workloads.

Design, build, and maintain scalable data systems and ETLs on GCP; connect production data to business systems; generate analytics and ML-driven insights for GTM teams; work cross-functionally and introduce modern tools (Airflow, DBT, Presto, Hightouch).

Design, build, and maintain scalable batch and streaming data pipelines, improve data modeling and governance, contribute to enterprise data architecture, and collaborate with cross-functional teams to ensure reliable, high-quality data for analytics and operational use.

Design, build, and own end-to-end ingestion pipelines and dbt transformation layers on Databricks. Lead GCP->AWS/Databricks migration, validate parity, and decommission legacy systems. Implement CDC and batch patterns, ensure data quality and observability, document assets in Unity Catalog, and build reverse ETL integrations to operational systems. Partner with business stakeholders and write architectural decision records to define engineering patterns and acceptance criteria.

Work directly with customers to design, build, and maintain data integrations and pipelines, troubleshoot ingestion/transformation/delivery issues, optimize large-scale data processing, collaborate with data scientists, and guide technical onboarding and long-term success.

Build and maintain production data pipelines and foundations for People Analytics and AI-driven tools. Integrate HR systems (Workday, Greenhouse), design data models, ensure governance for sensitive employee data, support data science and reporting, and deliver dashboards and Streamlit apps. Optimize queries, enable LLM consumption with quality and latency controls, and collaborate cross-functionally to transition AI prototypes to production.

The Senior Data Engineer will design scalable data processing frameworks using Java and Apache Spark, manage data pipelines, and mentor junior engineers.

Design, build, and maintain an internal data platform enabling secure, governed access for engineering and analytics. Partner with architects, implement data pipelines and automations, enforce security and compliance, support platform services, recommend architecture improvements, mentor teams, and drive adoption of scalable, self-service data capabilities.

Own end-to-end client data migrations: reverse-engineer legacy ETL, design and build Airflow DAGs and Spark jobs, validate and reconcile data at scale, document architecture and decisions, and coordinate cross-functional teams through go-live and hypercare.

Lead a team of data engineers to design, build, and maintain scalable data pipelines, a Data Vault, data warehouses and marts. Own backlog, sprint planning, mentoring, stakeholder collaboration, data validation/monitoring, and platform improvements to deliver key metrics and data infrastructure across the business.

Design, deploy, and maintain scalable data infrastructure and orchestration for analytics and ML; write optimized SQL over large warehouses; partner with engineering and product to structure source systems; maintain CDC pipelines; scale reporting and expose analytical datasets enterprise-wide while ensuring data management, security, and compliance.

Design and own end-to-end data platform architecture, build ingestion and transformation pipelines, model multi-tenant datasets, optimize Postgres and ClickHouse performance/cost, implement orchestration and observability, define data quality and reconciliation standards, and mentor data/product engineers.

Design, implement, and manage scalable multi-cloud data platforms and ELT pipelines (AWS/Azure). Build and optimize Snowflake data models, dbt transformations, and Airflow orchestration. Define data engineering standards, enforce governance and quality, collaborate cross-functionally, and mentor global/offshore teams.

Build and maintain automated ETL/ELT data pipelines, improve data quality and monitoring, optimize Postgres and BigQuery schemas and queries, develop internal tooling, support data governance (HIPAA), participate in on-call rotation, and collaborate across engineering teams to scale data systems.

The Data Engineer will design and maintain data pipelines, enhance the data warehouse, support machine learning applications, and collaborate with various teams to leverage data effectively.

The Senior Data Engineer will build and maintain ETL/ELT pipelines, optimize SQL queries, and collaborate on machine learning workflows, driving data engineering best practices.

Design, build, and maintain scalable data lakehouse platforms and ELT pipelines. Improve engineering practices, develop automation/tooling, monitor performance, reduce costs, and support platform modernization across Azure and cloud data technologies.

Design, optimize, and scale Snowflake-based data architecture and pipelines. Implement data observability, operationalize MLOps, build semantic enterprise models, and deploy AI-assisted agents and automation for production analytics and commercial data products. Share cross-functional ownership for pipeline resilience, monitoring, and model lifecycle management.

As a Sr. Data Engineer at Lyra Health, you will build and scale core data infrastructure, design data warehouses, and create efficient data pipelines while collaborating across departments.