The Role
Designs and implements Azure and Databricks data architectures, including scalable platforms, ingestion and ELT pipelines, governance, security, data quality, performance optimization, and disaster recovery. Develops PySpark solutions, applies Medallion Architecture, manages Unity Catalog and Delta Live Tables, and partners with stakeholders to deliver analytics solutions for insurance and operational use cases.
Summary Generated by Built In
Overview :
Job Title: Sr. Data Architect – Azure Databricks
Location: REMOTE Hybrid
- Design and implement end-to-end data architecture / data ecosystem using Azure-native services and Databricks
- Build scalable, high-performance data platforms and robust data frameworks
- Partner with stakeholders to translate business requirements into reliable data solutions
- Implement data governance principles using Unity Catalog
- Create and implement data ingestion and data integration patterns using Medallion Architecture
- Apply best practices in data engineering
- Develop PySpark notebooks
- Create procedures for advanced data transformation
- Handle performance tuning in Databricks and Azure SQL DBs
- Architect end-to-end data solutions supporting:
- Yield improvement
- Process control
- Predictive maintenance
- Operational efficiency
- Yield improvement
- Ensure strict governance, security, scalability, and compliance across the platform
- 10+ years of overall experience and 5+ years of architecture experience in data architecture / data engineering roles
- Strong hands-on experience with major enterprise data platforms
- Extensive experience with Azure-native services including:
- Azure Data Lake Storage (ADLS)
- Azure Data Factory (ADF)
- Azure Synapse Analytics
- Azure SQL
- Azure Data Lake Storage (ADLS)
- Strong hands-on experience with Azure Databricks
Should have worked with the latest Databricks capabilities including:
- Unity Catalog:
- Catalog and schema design
- Permissions and security
- Managed and external tables
- Lineage and governance
- Catalog and schema design
- Delta Live Tables (DLT):
- Reliable pipeline development
- CDC implementation
- Data transformations
- Data quality controls
- Scaling and performance tuning
- Reliable pipeline development
- Other modern Databricks capabilities and best practices
- Experience with data ingestion tools such as:
- Fivetran
- HVR or equivalent CDC / replication tools
- Fivetran
- Experience with ELT tools such as:
- DBT
- DBT
- Familiarity with:
- Real-time / streaming data
- Batch processing
- Sensor / OT / ET data flows
- Real-time / streaming data
Requirements
Technical Skills
- Strong expertise in data ingestion, transformation, and orchestration frameworks
- Hands-on experience building insurance-specific data models
- Proficiency in Python / PySpark and Spark SQL
- Strong understanding of streaming vs batch processing
- Experience with Spark job performance tuning
- Ability to optimize:
- Data storage formats
- Partitioning strategies
- Schema design
- Data storage formats
- Experience working with:
- Object stores
- Data lakes
- Cloud compute scaling
- Databricks cluster management
- Object stores
- Deep understanding of:
- Data governance
- Security and compliance
- RBAC and attribute-based access control
- Encryption and audit logs
- Data privacy requirements
- Data governance
- Experience with operational excellence practices:
- Monitoring and observability
- Reliability and metrics
- Backup, failover, and disaster recovery strategies
- Monitoring and observability
- Prior experience working on Insurance P&C cloud-based data and analytics projects
- Strong analytical and problem-solving skills
- Experience defining and enforcing best practices for Databricks workloads and Spark-based pipelines
- Solid understanding of distributed data processing concepts
- Strong communication and stakeholder management skills
- Ability to document architecture decisions, trade-offs, and technical recommendations
Benefits
Diversity Inclusion: Exavalu also promotes flexibility, adapting to the needs of employees, customers, and the business. It might be part-time work, working outside normal 9-5 business hours or working remotely. We also offer a welcome back program to help individuals return to the mainstream after a prolonged absence due to health or family reasons. At Exavalu, we are committed to building a diverse and inclusive workforce. We welcome applications for employment from all qualified candidates, regardless of race, colour, gender, national or ethnic origin, age, disability, religion, sexual orientation, gender identity or any other status protected by applicable law. We nurture a culture that embraces all individuals and promotes diverse perspectives, where you can make an impact and grow your career
Skills Required
- 10+ years of overall professional experience
- 5+ years of architecture experience in data architecture or data engineering roles
- Strong hands-on experience with enterprise data platforms
- Extensive experience with Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, and Azure SQL
- Strong hands-on experience with Azure Databricks
- Experience with Unity Catalog, including catalog and schema design, permissions, security, lineage, and governance
- Experience with Delta Live Tables, including reliable pipelines, CDC, transformations, data quality, scaling, and performance tuning
- Experience with data ingestion tools such as Fivetran and HVR or equivalent CDC and replication tools
- Experience with dbt or equivalent ELT tools
- Proficiency in Python, PySpark, and Spark SQL
- Experience with streaming and batch processing
- Experience with Spark job performance tuning, storage formats, partitioning, and schema design
- Experience with object stores, data lakes, cloud compute scaling, and Databricks cluster management
- Understanding of data governance, security, compliance, RBAC, attribute-based access control, encryption, audit logs, and data privacy
- Experience with monitoring, observability, reliability metrics, backup, failover, and disaster recovery
- Hands-on experience building insurance-specific data models
- Prior experience with Insurance P&C cloud-based data and analytics projects
- Strong analytical, problem-solving, communication, and stakeholder management skills
- Experience defining Databricks and Spark pipeline best practices
- Understanding of distributed data processing concepts
- Ability to document architecture decisions, trade-offs, and technical recommendations
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Exavalu leads the way in business and technology consulting and solution delivery, focusing on specialized digital transformation across the Insurance, Healthcare & Life Sciences, and Nonprofit industries. As an award-winning digital transformation advisor, we take pride in delivering leading-edge solutions to clients worldwide. Led by industry veterans and technology experts, we bring our deep understanding in customer experience, process automation, digital engineering, cloud, data and AI with a very strong industry core foundation. Drawing on our industry insights and technological acumen, we enable clients to drives growth, profitability, and cost optimization. We mitigate risks and elevate customer experience through our comprehensive suite of services, including strategic advisory, expert-led technology [build and maintenance], and best-in-class products and solutions. Our unwavering focus on understanding the industry's hard problems and finding the right solution has garnered recognition as the preferred ally for organizations seeking transformative growth.
