The Role
Build scalable production-grade ETL/ELT pipelines on Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, and Workflows. Develop data ingestion, transformations, quality rules, orchestration, monitoring, and automation across Bronze, Silver, and Gold layers. Apply dimensional data modeling, streaming, cloud platforms, CI/CD, governance, security, performance tuning, and lakehouse architecture. Explain technical concepts and architecture decisions using real project examples.
Summary Generated by Built In
DATAECONOMY is one of the fastest-growing Data & Analytics company with global presence. We are well-differentiated and are known for our Thought leadership, out-of-the-box products, cutting-edge solutions, accelerators, innovative use cases, and cost-effective service offerings.
We offer products and solutions in Cloud, Data Engineering, Data Governance, AI/ML, DevOps and Blockchain to large corporates across the globe. Strategic Partners with AWS, Collibra, cloudera, neo4j, DataRobot, Global IDs, tableau, MuleSoft and Talend.
Senior/Lead Data Engineer — Databricks
Raleigh NC
Full-time
- Build scalable, production-grade ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Live Tables, Workflows).
- Ingest structured, semi-structured, and streaming data into Bronze, Silver, and Gold layers.
- Develop optimized transformations, data quality rules, and reusable framework components.
- Implement best practices for job orchestration, monitoring, alerting, and automation.
- Hands-on experience: Spark, Delta Lake, Workflows, Unity Catalog.
- Strong SQL programming and performance tuning skills.
- Experience with cloud environments (AWS/Azure/GCP).
- Experience with modern data lakehouse concepts and distributed systems.
- Strong understanding of Lakeflow Connect, LSDP/Lakehouse, Medallion Architecture, Data Validations, Genie, and Agent Bricks/RAG use cases.
- Should be able to explain these concepts using real project examples and architecture decisions.
Requirements
- Strong Python (PySpark) and SQL programming
- Databricks — Spark, Delta Lake, Workflows, Unity Catalog
- ETL/ELT pipeline development — Medallion Architecture (Bronze/Silver/Gold)
- Delta Live Tables, Auto-Loader, Structured Streaming
- Data modeling — dimensional (star/snowflake), normalization/denormalization
- CI/CD, Git, job orchestration
- Cloud experience — AWS, Azure, or GCP
- 7–10+ years in data engineering
Nice-to-Have Skills
- Lakeflow Connect, LSDP/Lakehouse, Genie, Agent Bricks/RAG use cases
- Data governance, metadata management, Unity Catalog advanced features
- Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cost optimization and performance tuning in cloud environments
- Corporate/enterprise data warehousing background
Skills Required
- 7-10+ years of experience in data engineering
- Strong Python and PySpark programming skills
- Strong SQL programming and performance tuning skills
- Hands-on Databricks experience with Spark, Delta Lake, Workflows, and Unity Catalog
- ETL/ELT pipeline development using Medallion Architecture
- Experience with Delta Live Tables, Auto Loader, and Structured Streaming
- Data modeling experience including dimensional, star, snowflake, normalization, and denormalization models
- Experience with CI/CD, Git, and job orchestration
- Cloud experience with AWS, Azure, or GCP
- Experience with modern data lakehouse concepts and distributed systems
- Knowledge of Lakeflow Connect, LSDP/Lakehouse, Data Validations, Genie, and Agent Bricks/RAG use cases
- Lakeflow Connect, LSDP/Lakehouse, Genie, and Agent Bricks/RAG use cases
- Data governance, metadata management, and advanced Unity Catalog features
- Experience with Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cloud cost optimization and performance tuning
- Corporate or enterprise data warehousing background
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
DATAECONOMY is a global, cloud-first data and AI consultancy delivering enterprise-grade solutions through an innovative intellectual-property suite. Its work spans data and BI platform modernization, self-service AI, data mesh and fabric, master data management, governance, cloud enablement, digital engineering, knowledge graphs, and machine lakes supporting cybersecurity and financial-crime use cases for enterprise clients.
