About This Opportunity
CTI Staffing is partnering with a leading retail organization to find a Senior Databricks Data Engineer for their enterprise data team based in the Greater Boston, MA area.
The team is building out a modern Azure data platform that brings together data from many retail systems. This role matters because reliable, incremental pipelines are what let the business trust its data and act on it faster.
What You’ll Do
- Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
- Build Change Data Capture pipelines for incremental and near real-time processing.
- Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
- Improve existing pipelines by quickly learning current codebases and delivering targeted upgrades.
- Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
- Implement data quality rules and quarantine error records.
- Apply data modeling techniques including SCD Type 1 and Type 2.
- Automate workflows with Databricks Workflows and GitHub-based CI/CD, and work with developers, architects, and product owners in an Agile team.
Requirements
What You Bring
Must-Have:
- Extensive hands-on Databricks experience, able to contribute from day one.
- Proven experience building end-to-end data pipelines with CDC.
- Strong PySpark and SQL skills.
- Experience working in Microsoft Azure.
- Working knowledge of Delta Lake and distributed data processing.
- Experience with Kafka or comparable streaming technologies.
- Hands-on experience with SCD Type 1 and Type 2 data modeling.
- Familiarity with GitHub and CI/CD for version control and deployment.
Nice-to-Have:
- Python experience beyond PySpark.
- Databricks Workflows and Unity Catalog experience.
- Experience with message queues and API-based ingestion.
- Retail, CPG, or supply chain data background.
- Databricks certification.
Technical Environment: Azure, Databricks, PySpark, SQL, Delta Lake, Kafka, Unity Catalog, GitHub, CI/CD, SQL Server, JSON, XML, REST APIs.
Skills Required
- Extensive hands-on Databricks experience
- Experience building end-to-end data pipelines with change data capture
- Strong PySpark and SQL skills
- Experience working with Microsoft Azure
- Working knowledge of Delta Lake and distributed data processing
- Experience with Kafka or comparable streaming technologies
- Hands-on experience with SCD Type 1 and Type 2 data modeling
- Familiarity with GitHub and CI/CD for version control and deployment
- Python experience beyond PySpark
- Experience with Databricks Workflows and Unity Catalog
- Experience with message queues and API-based ingestion
- Retail, CPG, or supply chain data background
- Databricks certification
What We Do
CTI Consulting is a South Florida technology solutions and staffing firm founded in 1998. It provides IT staffing and nearshore staffing services for startups through Fortune 100 companies, alongside digital-product work such as designing, testing, deploying, improving, and updating mobile applications. The company emphasizes close client collaboration, comprehensive candidate evaluation, continuing education, and innovative, scalable software solutions that improve operational efficiency.
