The Role
Develop and optimize Databricks data pipelines using Spark SQL and PySpark. Integrate data from lakes, APIs, and external sources through Unity Catalog; build ETL workflows, data models, views, and stored procedures. Analyze data quality and improve processing performance. Collaborate with front-end teams on reports and dashboards, while using GitLab, CI/CD, and Jira for development and task management.
Summary Generated by Built In
- Develop and optimize data pipelines in Databricks for transforming and processing data from various sources.
- Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.).
- Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures.
- Perform data analysis to identify quality issues, optimize pipelines, and enhance data processing for analytics.
- Collaborate on report generation and dashboard creation with front-end teams.
- Use GitLab for version control, CI/CD automation, and task management (Jira).
Requirements
- Master’s or Bachelor’s degree in Data Engineering or related field.
- 3+ years experience with Databricks and advanced SQL.
- Experience with ETL processes, views, and procedures.
- Strong experience with Databricks, Spark SQL, PySpark, and SQL.
- Expertise in creating and optimizing views and stored procedures in Databricks.
- Experience building ETL workflows and data models.
- Knowledge of cloud platforms (AWS, Azure) and version control tools (Git, GitLab).
- Experience with healthcare or clinical trial data.
- Familiarity with DevOps practices.
Skills Required
- Bachelor’s or master’s degree in Data Engineering or a related field
- 3+ years of experience with Databricks and advanced SQL
- Experience with ETL processes, views, and procedures
- Strong experience with Databricks, Spark SQL, PySpark, and SQL
- Expertise creating and optimizing views and stored procedures in Databricks
- Experience building ETL workflows and data models
- Knowledge of cloud platforms, including AWS and Azure
- Experience with Git and GitLab version control tools
- Experience with healthcare or clinical trial data
- Familiarity with DevOps practices
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Cynosure Corporate Solutions is a human capital services company focused on helping clients align their people strategy with business goals. Its offerings include executive search, recruitment, training, temporary staffing, statutory compliance, and other HR-aligned services. The company says it works to understand client objectives and connect human-resource planning with those objectives, supporting organizations’ workforce needs through tailored talent and employment solutions.
.jpg)
