Primary Skills
- Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark.
- Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.
- Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.
- Create, monitor, and optimize Delta Live Tables (DLT) pipelines.
- Implement scalable and efficient data ingestion processes using Auto Loader.
- Develop and manage real-time data processing solutions using Structured Streaming.
- Orchestrate, schedule, and monitor data workflows using Databricks Workflows.
- Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.
- Establish and enforce data governance, security, and access controls using Unity Catalog.
- Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.
- Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions. Required Skills (Must Have)
- Databricks Platform
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- PySpark and Apache Spark
- Structured Streaming
- Auto Loader
- SQL
- Lakehouse Data Modeling
- Strong understanding of data engineering best practices and scalable data architectures Preferred Skills (Good to Have)Azure Ecosystem
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Microsoft Purview
- Microsoft Fabric AWS Ecosystem
- AWS Glue
- AWS Lambda
- AWS Step Functions Data Engineering & Integration
- Apache Airflow
- DBT
- Fivetran
- Informatica Streaming & Analytics
- Apache Kafka
- Power BI Data Governance
- Collibra
- Alation GCP
- BigQuery Qualifications
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
- Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
- Strong analytical, troubleshooting, and problem-solving capabilities.
- Experience working in agile and collaborative environments.
- Excellent communication and stakeholder management skills. Preferred Candidate Profile
- Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
- Strong understanding of data governance, security, and compliance frameworks.
- Experience delivering both batch and real-time data processing solutions.
- Ability to work independently while collaborating effectively across global teams. Key Technologies
We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL, with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.
Key ResponsibilitiesDatabricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI
Specialization
- Databricks Engineering: Lead Data Engineer
Job requirements
- Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark.
- Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers.
- Develop and manage Delta Lake solutions with ACID transactions, schema enforcement, and data reliability features.
- Create, monitor, and optimize Delta Live Tables (DLT) pipelines.
- Implement scalable and efficient data ingestion processes using Auto Loader.
- Develop and manage real-time data processing solutions using Structured Streaming.
- Orchestrate, schedule, and monitor data workflows using Databricks Workflows.
- Design and implement Lakehouse data models to support reporting, analytics, and business intelligence requirements.
- Establish and enforce data governance, security, and access controls using Unity Catalog.
- Optimize Spark jobs, SQL queries, and overall platform performance to ensure efficiency and scalability.
- Collaborate with cross-functional teams, including data analysts, architects, and business stakeholders, to deliver high-quality data solutions. Required Skills (Must Have)
- Databricks Platform
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- PySpark and Apache Spark
- Structured Streaming
- Auto Loader
- SQL
- Lakehouse Data Modeling
- Strong understanding of data engineering best practices and scalable data architectures Preferred Skills (Good to Have)Azure Ecosystem
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Microsoft Purview
- Microsoft Fabric AWS Ecosystem
- AWS Glue
- AWS Lambda
- AWS Step Functions Data Engineering & Integration
- Apache Airflow
- DBT
- Fivetran
- Informatica Streaming & Analytics
- Apache Kafka
- Power BI Data Governance
- Collibra
- Alation GCP
- BigQuery Qualifications
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
- Proven experience in designing and implementing cloud-based data engineering solutions and scalable data pipelines.
- Strong analytical, troubleshooting, and problem-solving capabilities.
- Experience working in agile and collaborative environments.
- Excellent communication and stakeholder management skills. Preferred Candidate Profile
- Hands-on experience with modern Lakehouse architectures and enterprise-scale data platforms.
- Strong understanding of data governance, security, and compliance frameworks.
- Experience delivering both batch and real-time data processing solutions.
- Ability to work independently while collaborating effectively across global teams. Key Technologies
We are seeking a highly skilled Databricks Engineer to design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform. The ideal candidate will possess strong expertise in Databricks, PySpark, and SQL, with hands-on experience building batch and real-time data pipelines, implementing Lakehouse architectures, and ensuring data governance and performance optimization.
Key ResponsibilitiesDatabricks | PySpark | Apache Spark | Delta Lake | Delta Live Tables (DLT) | Unity Catalog | Structured Streaming | Auto Loader | SQL | Lakehouse Architecture | Azure | AWS | Airflow | Kafka | Power BI
Skills Required
- Databricks Platform
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows
- PySpark and Apache Spark
- Structured Streaming
- Auto Loader
- SQL
- Lakehouse Data Modeling
- Bachelor's or Master's degree in Computer Science, Data Engineering, IT, or related discipline
- Proven experience designing and implementing cloud-based data engineering solutions and scalable data pipelines
- Strong analytical, troubleshooting, and problem-solving capabilities
- Experience working in agile and collaborative environments
- Excellent communication and stakeholder management skills
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Microsoft Purview
- Microsoft Fabric
- AWS Glue
- AWS Lambda
- AWS Step Functions
- Apache Airflow
- DBT
- Fivetran
- Informatica
- Apache Kafka
- Power BI
- Collibra
- Alation
- BigQuery
Brillio Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Brillio and has not been reviewed or approved by Brillio.
-
Healthcare Strength — Healthcare is considered comprehensive, including medical coverage for employees and dependents alongside life, disability, and accidental death protections. Feedback suggests these protections are a core strength of the package.
-
Leave & Time Off Breadth — Time-off options include paid leave and parental leave, with flexible or ‘flexible PTO’ approaches cited in some contexts. Feedback suggests this breadth helps support work-life balance when team norms permit usage.
-
Wellbeing & Lifestyle Benefits — Wellbeing offerings span counseling, financial-management sessions, fitness programs, and travel insurance, plus region-specific extras like discounted IT hardware and work-from-home essentials. Feedback suggests these add-ons enhance perceived value beyond core insurance.
Brillio Insights
What We Do
Brillio is the leader in global digital business transformation, applying technology with a human touch. We help businesses define internal and external transformation objectives, and translate those objectives into actionable market strategies using proprietary technologies. With 2600+ experts and 13 offices worldwide, Brillio is the ideal partner for enterprises that want to quickly increase their core business productivity, and achieve a competitive edge, with the latest digital solutions.
.png){kind=logo}
.jpeg)
