The Role
Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, Workflows, and Unity Catalog. Ingest batch and streaming data across medallion architecture layers, develop transformations and data quality rules, and implement orchestration, monitoring, alerting, and automation. The role requires cloud data engineering, data modeling, performance tuning, CI/CD, and strong knowledge of lakehouse architecture, governance, and distributed systems.
Summary Generated by Built In
DATAECONOMY is one of the fastest-growing Data & Analytics company with global presence. We are well-differentiated and are known for our Thought leadership, out-of-the-box products, cutting-edge solutions, accelerators, innovative use cases, and cost-effective service offerings.
We offer products and solutions in Cloud, Data Engineering, Data Governance, AI/ML, DevOps and Blockchain to large corporates across the globe. Strategic Partners with AWS, Collibra, cloudera, neo4j, DataRobot, Global IDs, tableau, MuleSoft and Talend.
Senior/Lead Data Engineer — Databricks
Rutherford, NJ/ Jersey City, NJ
Full-time
- Build scalable, production-grade ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Live Tables, Workflows).
- Ingest structured, semi-structured, and streaming data into Bronze, Silver, and Gold layers.
- Develop optimized transformations, data quality rules, and reusable framework components.
- Implement best practices for job orchestration, monitoring, alerting, and automation.
- Hands-on experience: Spark, Delta Lake, Workflows, Unity Catalog.
- Strong SQL programming and performance tuning skills.
- Experience with cloud environments (AWS/Azure/GCP).
- Experience with modern data lakehouse concepts and distributed systems.
- Strong understanding of Lakeflow Connect, LSDP/Lakehouse, Medallion Architecture, Data Validations, Genie, and Agent Bricks/RAG use cases.
- Should be able to explain these concepts using real project examples and architecture decisions.
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Requirements
- Strong Python (PySpark) and SQL programming
- Databricks — Spark, Delta Lake, Workflows, Unity Catalog
- ETL/ELT pipeline development — Medallion Architecture (Bronze/Silver/Gold)
- Delta Live Tables, Auto-Loader, Structured Streaming
- Data modeling — dimensional (star/snowflake), normalization/denormalization
- CI/CD, Git, job orchestration
- Cloud experience — AWS, Azure, or GCP
- 7–10+ years in data engineering
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Nice-to-Have Skills
- Lakeflow Connect, LSDP/Lakehouse, Genie, Agent Bricks/RAG use cases
- Data governance, metadata management, Unity Catalog advanced features
- Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cost optimization and performance tuning in cloud environments
- Corporate/enterprise data warehousing background
Skills Required
- Strong Python and PySpark programming
- Strong SQL programming and performance tuning skills
- Hands-on Databricks experience with Spark, Delta Lake, Workflows, and Unity Catalog
- ETL/ELT pipeline development using Medallion Architecture
- Experience with Delta Live Tables, Auto Loader, and Structured Streaming
- Data modeling experience including dimensional, star, snowflake, normalization, and denormalization models
- CI/CD, Git, and job orchestration experience
- Cloud experience with AWS, Azure, or GCP
- 7-10+ years of data engineering experience
- Knowledge of Lakeflow Connect, LSDP/Lakehouse, Genie, and Agent Bricks/RAG use cases
- Ability to explain Databricks concepts using real project examples and architecture decisions
- Data governance and metadata management experience
- Advanced Unity Catalog features
- Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cloud cost optimization and performance tuning
- Corporate or enterprise data warehousing background
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
DATAECONOMY is a global, cloud-first data and AI consultancy delivering enterprise-grade solutions through an innovative intellectual-property suite. Its work spans data and BI platform modernization, self-service AI, data mesh and fabric, master data management, governance, cloud enablement, digital engineering, knowledge graphs, and machine lakes supporting cybersecurity and financial-crime use cases for enterprise clients.
