We are seeking a hands-on Data Engineer with strong Databricks and PySpark experience to build scalable data pipelines and analytics applications within a modern data platform.
This role focuses on modernizing legacy ETL and reporting systems (Teradata, Informatica, Tableau) into Databricks-native pipelines, dashboards, and Python-based data applications. The ideal candidate operates across both data engineering and lightweight application development.
Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark on Databricks
- Build analytics data models and transformation workflows for enterprise reporting and analytics
- Migrate legacy ETL workloads from platforms such as Informatica and Teradata to Databricks
- Develop Databricks-native dashboards and analytics applications to replace traditional BI tools
- Build lightweight Python-based data applications (e.g., FastAPI) to expose and interact with data
- Integrate Databricks pipelines with APIs and application services
- Implement Slowly Changing Dimensions (SCD) and dimensional data modeling techniques
- Develop reusable data engineering frameworks and standardized pipelines
- Optimize Spark workloads for performance, scalability, and cost efficiency
- Collaborate with analytics and business teams to deliver user-facing data solutions
- Leverage AI-assisted coding tools (e.g., Copilot, ChatGPT) to improve development productivity
- Contribute to best practices for modern data engineering and analytics application development
Required Experience
- 5+ years of hands-on experience building data pipelines using PySpark in production environments
- Strong experience with the Databricks platform (workspaces, clusters, Jobs & Workflows, Unity Catalog)
- Experience building analytics dashboards within Databricks (Databricks SQL)
- Proven experience designing and building scalable ETL/ELT data pipelines
- Strong Python development skills, including building REST APIs or data services
- Experience building or supporting data-driven applications (not just traditional ETL pipelines)
- Solid understanding of data modeling, including dimensional modeling and transformation patterns
- Experience using AI-assisted development tools (e.g., Copilot, ChatGPT) in engineering workflows
- Exposure to LLM integration or an AI-powered data application
- Familiarity with cloud platforms (AWS, Azure, or GCP)
Preferred Experience
- Experience migrating from legacy platforms: Informatica → Databricks / Spark; Teradata → cloud-native data platforms; Tableau (or similar) → Databricks-native dashboards
- Experience with FastAPI or similar Python frameworks for data applications
- Exposure to CI/CD pipelines for data engineering workflows
- Understanding of microservices architecture and scalable application design
- Experience in Healthcare Payor domain (e.g., claims processing, member data, provider data, eligibility, or billing systems)
Top Skills
What We Do
Doran Jones Inc. (DJI) is the leading financial services Data Engineering and Application Development firm, specializing in Capital Markets, Risk, and Regulatory Compliance. We are US-based, helping our clients fill gaps in capacity and expertise, reduce risk, and accelerate change. Our leaders average over 25 years of experience in Financial Services and Technology. DJI is repeatedly engaged by the largest and most complex clients for our Agile transformation expertise and practical understanding of the critical relationship between data, architecture, and application development. DJI has a mission to place more people from non-traditional backgrounds into sustainable technology careers. Through partnerships with non-profit technology programs in underserved communities and Veteran organizations, candidates transition from tech training programs into real IT careers at DJI. Our unique recruitment policy allows us to create exceptional teams, bringing a broad spectrum of experience to our company and creating anything but a traditional consulting firm. DJI is a wholly-owned subsidiary of McLaren Strategic Ventures, a private equity firm, that invests in start-ups and scale-ups. They provide a full range of highly trusted domain consulting, advisory services, and innovative technologies globally. Doran Jones is able to offer a broad range of technology, deeper AI and machine learning tools, data intelligence, alternative resourcing capabilities, and capital access to invest and digitize at scale. Our clients benefit from this unique portfolio allowing them to accelerate the pace of their own digital transformations through innovative products and services designed around industry-wide issues.
.png)
