Databricks Senior Architect

• Define and maintain the end-to-end architecture for Primoris’ Databricks Lakehouse, including Medallion layer patterns, ingestion, transformation, serving, and AI/ML/GenAI workloads. 

• Design and implement scalable, secure Databricks patterns (including serverless where appropriate) that power batch, streaming, ML, AI Gateway and GenAI workloads, optimizing for performance, cost, security, and reliability. 

• Architect and implement solutions using the Databricks Medallion architecture, Unity Catalog, and Delta Lake to support both analytical and operational use cases. 

• Partner with Primoris’ enterprise / solution architecture, security, infrastructure, and application teams to align Databricks with broader platform and integration standards. 

Delivery & Operations 

• Assist in architecting, developing, deploying, and migrating priority use cases and workloads into production on the Databricks Lakehouse. 

• Serve as the first point of contact for technical challenges or questions related to development and production workloads (triage, root cause analysis, coordination of escalation as needed). 

• Provide technical leadership and delivery oversight across hybrid teams, including internal employees, consulting partners, and offshore resources, ensuring seamless coordination, clear accountability, and consistent outcomes. 

• Establish and champion engineering standards (CI/CD, automated testing, observability, monitoring/alerting, incident runbooks) to ensure production-grade delivery and operations. 

• Design and review workflows, jobs, and streaming pipelines (e.g., Auto Loader, DLT, structured streaming) for robustness, recoverability, and SLA adherence. 

• Implement Databricks product innovations, private previews, and upgrades following change management and release best practices, in coordination with Primoris IT and partners. 

Data Engineering 

• Build and review multi-language notebooks (SQL, Python / PySpark) with a primary focus on data engineering and data science / AI/ML workflows. 

• Design and guide implementation of streaming and batch data pipelines (Auto Loader, DLT, Structured Streaming) handling structured, semi-structured, and unstructured data. 

Governance, Security & Compliance 

• Implement and oversee Row-Level Security (RLS), column masking, data encryption/decryption, and other data protection controls using Unity Catalog and cloud-native capabilities. 

• Architect and govern enterprise GenAI platforms on Databricks, including Unity Catalog, AI Gateway, Model Serving, and Vector Search, ensuring secure, compliant, observable, and cost-effective AI solutions across the organization. 

• Work with security and compliance stakeholders to define and implement data governance models, access patterns, data classification, auditing, and regulatory/contractual controls as required. 

• Ensure all architectures and implementations align with Primoris’ security, networking, and compliance standards for Azure and Databricks. 

Stakeholder Management & Communication 

• Participate in discovery workshops with business and technical stakeholders to understand current-state architecture, data flows, pain points, and priority use cases. 

• Translate findings into strategic roadmaps, architecture options, and phased delivery plans with clear trade-offs, milestones, and ownership. 

• Provide regular internal and external updates on progress, risks, and blockers to leadership, with clear options, mitigations, and recommendations. 

Coaching, Standards & Reuse 

• Define and evangelize pragmatic engineering and platform standards (coding patterns, data modeling, observability, cost optimization, workspace standards). 

• Mentor engineers through design and code reviews, knowledge-sharing sessions, and pairing on complex tasks. 

• Develop and maintain reusable assets (reference architectures, templates, starter notebooks, Terraform/IaC patterns, runbooks) to accelerate adoption and improve consistency. 

• Demonstrate proactive ownership: surface risks early, drive decisions, and remove blockers to maintain delivery momentum and stakeholder confidence. 

Qualifications 

• Bachelor’s degree in Computer Science, Information Systems, Engineering, Data Science, or equivalent practical experience. 

• 8+ years of hands-on experience in Data, Analytics, or AI roles (e.g., data engineering, data architecture, analytics platform engineering), with 4+ years accountable for technical delivery and architecture/tech-lead responsibilities. 

• Proven experience architecting and implementing Databricks solutions using the Medallion architecture, including ingestion, transformation, serving, and AI/ML workloads. 

• Hands-on expertise implementing Databricks on Microsoft Azure (required), including familiarity with Azure security, networking, identity, and storage concepts. 

• Experience designing and developing large-scale distributed systems and big data solutions using technologies such as Apache Spark™, Hadoop, or Cassandra. 

• Proficiency in Python (including PySpark) and SQL, including performance tuning and best practices for large-scale data processing. 

• Proficiency in designing and implementing Auto Loader, Delta Live Tables (DLT), and streaming solutions in Databricks to handle structured, semi-structured, and unstructured data. 

• Experience with Unity Catalog and related capabilities (data governance, RBAC, lineage, auditing, and secure data sharing) or similar enterprise data governance tools. 

• Demonstrated ability to attribute business value and outcomes to specific project deliverables and technical KPIs. 

• Experience conducting discovery workshops, creating strategic roadmaps, performing business/process analysis, and managing or influencing the delivery of complex data/AI programs or projects. 

• Expertise implementing end-to-end data security controls in Databricks (RLS, column masking, encryption, key management), in alignment with enterprise security policies. 

• Evidence of continuous learning and staying current with the fast-evolving Databricks Lakehouse platform, Azure services, and modern data/AI patterns. 

• Nice to have: 

• One or more Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Data Architect Associate/Professional). 

• Deep hands-on knowledge of Big Data Engineering or cloud DWH technologies (e.g., Synapse, Snowflake, BigQuery, Redshift). 

 Skills and Competencies 

• Technical leadership: Ability to set architectural direction, make trade-off decisions, and guide engineers and partners toward production-ready solutions. 

• Stakeholder management: Ability to manage expectations, influence decisions, and build trusted relationships with both technical and non-technical stakeholders (engineering, IT, security, operations, finance, business leadership). 

• Communication: Excellent written and verbal communication skills; able to explain complex technical topics in clear, outcome-oriented language. 

• Organization & execution: Exceptional organizational and time management skills; able to juggle multiple workstreams and drive them to completion. 

• Attention to detail & quality: High standards for solution robustness, reliability, security, and documentation. 

• Team building & mentoring: Strong leadership and team-building capabilities; experience mentoring engineers, establishing best practices, and raising the technical bar. 

• Adaptability: Ability to adapt to changing priorities, work under pressure, and thrive in a fast-paced, evolving environment. 

• Ownership mindset: Bias toward action, proactive identification of issues, and a strong sense of accountability for outcomes rather than just deliverables.