Primary Skills
Specialization
- Databricks Engineering: Lead Data Engineer
Job requirements
Experience Range: With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark
Key Responsibilities:
1. Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
2. Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
3. Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
4. Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
5. Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
6. Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
7. Monitor and tune performance of data processing jobs for optimal throughput and reliability
8. Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage
Required Skills:
1. Advanced proficiency in Databricks platform
2. Expertise in PySpark for distributed data processing and transformation
3. Strong SQL skills for analytics and data manipulation
4. Experience with data warehousing architectures
5. Hands-on experience designing and maintaining stored procedures
6. Proficiency in Apache Spark for large-scale data processing
Preferred Skills:
1. Experience with Azure Databricks or AWS Glue
2. Familiarity with CI/CD pipelines for Databricks workflows
3. Knowledge of Databricks performance optimization techniques
4. Exposure to data governance and security best practices in cloud environments
5. Experience with real-time data streaming frameworks such as Apache Kafka
Desired Qualifications:
1. Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
2. Certification in Databricks or Apache Spark
3. Certification in SQL or data warehousing technologies
Skills Required
- 5-6 years of experience in data engineering
- Advanced proficiency in Databricks platform
- Expertise in PySpark for distributed data processing
- Strong SQL skills for analytics and data manipulation
- Experience with data warehousing architectures
- Hands-on experience designing and maintaining stored procedures
- Proficiency in Apache Spark for large-scale data processing
- Experience with Azure Databricks or AWS Glue
- Familiarity with CI/CD pipelines for Databricks workflows
- Knowledge of Databricks performance optimization techniques
- Exposure to data governance and security best practices in cloud environments
- Experience with real-time data streaming frameworks such as Apache Kafka
- Bachelor's degree in Computer Science, Information Technology, or related discipline
- Certification in Databricks or Apache Spark
- Certification in SQL or data warehousing technologies
Brillio Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Brillio and has not been reviewed or approved by Brillio.
-
Healthcare Strength — Healthcare is considered comprehensive, including medical coverage for employees and dependents alongside life, disability, and accidental death protections. Feedback suggests these protections are a core strength of the package.
-
Leave & Time Off Breadth — Time-off options include paid leave and parental leave, with flexible or ‘flexible PTO’ approaches cited in some contexts. Feedback suggests this breadth helps support work-life balance when team norms permit usage.
-
Wellbeing & Lifestyle Benefits — Wellbeing offerings span counseling, financial-management sessions, fitness programs, and travel insurance, plus region-specific extras like discounted IT hardware and work-from-home essentials. Feedback suggests these add-ons enhance perceived value beyond core insurance.
Brillio Insights
What We Do
Brillio is the leader in global digital business transformation, applying technology with a human touch. We help businesses define internal and external transformation objectives, and translate those objectives into actionable market strategies using proprietary technologies. With 2600+ experts and 13 offices worldwide, Brillio is the ideal partner for enterprises that want to quickly increase their core business productivity, and achieve a competitive edge, with the latest digital solutions.
