About the Role
We are seeking a highly skilled Senior Data Engineer to join our Data Engineering & Platform organization. This role is ideal for someone passionate about building scalable data solutions, enabling cross-functional data delivery, and raising engineering excellence across the organization. You will work on an advanced Databricks-on-AWS data platform and play a key role in transforming our data ecosystem—designing and building resilient pipelines, implementing CI/CD, and contributing to platform modernization efforts.
As a senior member of the team, you will partner with data engineers, analytics teams, product teams, and platform engineering to build high-quality, production-grade data solutions that power enterprise analytics, data science, and operational workloads.
Key Responsibilities:
Data Engineering & Pipeline Development
- Design, build, maintain, and optimize scalable data pipelines and ELT/ETL processes to ingest, process, and store large volumes of data from various sources.
- Architect robust batch and streaming solutions using PySpark, Spark SQL, and Databricks Jobs.
- Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance.
- Ensure reliability, observability, and SLAs through monitoring, alerting, and automated recovery patterns.
- Implement and maintain data integration solutions, including APIs and data connectors, to facilitate data exchange between systems.
Cloud Platform Engineering
- Build and manage data solutions on AWS, leveraging services such as S3, Lambda, Glue, IAM, EC2, EKS, Step Functions, and EventBridge.
- Apply cloud engineering best practices for cost optimization, reliability, and security.
CI/CD, DevOps, and Infrastructure-as-Code
- Implement and maintain CI/CD pipelines using:
- Databricks Asset Bundles
- Terraform (IaC)
- GitHub Actions / Azure DevOps / Jenkins (depending on internal stack)
- Own deployment automation and environment management for data workloads and platform components.
- Contribute to reusable Terraform modules and engineering standards.
Data Governance and Data Quality
- Maintain data quality through validation frameworks (e.g., DQX, Great Expectations, Databricks expectations, custom frameworks).
- Ensure data quality, integrity, and security by implementing best practices for data governance and management.
- Implement governance and lineage through Unity Catalog, cataloging standards, and metadata best practices.
Platform Modernization
- Contribute to initiatives such as:
- Migration of pipelines from legacy integration tools to Lakeflow / Lakehouse-native ingestion
- Improving infrastructure automation
- Enhancing data observability, testing, and deployment workflows
Collaboration & Technical Leadership
- Serve as a technical mentor to junior engineers.
- Partner closely with data analysts, data scientists, and business stakeholders to deliver enterprise data assets.
- Contribute to architecture decisions, design reviews, and platform roadmap conversations.
Required Qualifications
- 7+ years of data engineering experience building production-grade data pipelines.
- Deep expertise in Databricks, Spark, and Delta Lake.
- Hands-on experience with AWS cloud services.
- Strong proficiency in Python, SQL, and distributed computing patterns.
- Experience with Terraform and cloud-native CI/CD.
- Strong understanding of data modeling, orchestration, and data quality frameworks.
- Ability to operate autonomously while collaborating in a team environment.
Additional Qualifications
- Experience with Lakeflow, Databricks Workflows, or ingestion modernization initiatives.
- Knowledge of software engineering best practices:
- test automation
- versioning
- code reviews
- Git workflows
- Familiarity with event-driven architecture and streaming technologies.
- Experience supporting Data Science and ML platform teams.
- P&C Insurance domain knowledge preferred
#LI-RL1
IND1
Click here for some insight into our culture!
The Baldwin Group will not accept unsolicited resumes from any source other than directly from a candidate who applies on our career site. Any unsolicited resumes sent to The Baldwin Group, including unsolicited resumes sent via any source from an Agency, will not be considered and are not subject to any fees for any placement resulting from the receipt of an unsolicited resume.
Similar Jobs
What We Do
BRP is now The Baldwin Group! We’ve updated our name to reflect our unified group of talented teams across the country.
The Baldwin Group is a cohesive group of experts in business insurance, employee benefits, retirement planning, and all areas of private and personal insurance. Since our founding in 2011, we’ve evolved from a local business into a national firm with a vast network of specializations and industry practices for the benefit of our more than two million clients across the country.
In addition, we have built excellent relationships with a wide range of insurance company partners. These relationships, coupled with our entrepreneurial and family-oriented culture, and deep expertise enable us to seamlessly deliver a breadth of innovative solutions to clients.
At The Baldwin Group, we help provide the solutions our clients need to have confidence and gain peace of mind as they pursue what’s possible for themselves, their families, and their businesses. Whether they are renting their first apartment or buying a larger home, opening a small business or taking their company public, we offer solutions to support them on every step of their journey. This has been our story since the beginning—we provide the indispensable expertise and quality insights that give our clients peace of mind to pursue their purpose, passion, and dreams. And that’s what The Baldwin Group will continue to do for years to come: we Protect the Possible℠.
