Data Architect

The Data Architect is responsible for designing, building, and evolving the enterprise data foundation supporting the Firm’s artificial intelligence, analytics, and business applications.  This position translates complex enterprise data requirements into practical, scalable architectures and delivers reliable, governed data that technology teams and business stakeholders can use with confidence.  This is a technical role with primary responsibility for enterprise data modeling, lakehouse architecture, data integration, governance, and data quality.  The Data Architect will design and implement solutions within Azure Databricks, develop data pipelines and models, and establish standards and reusable patterns for the Firm’s growing data environment. The position will partner closely with AI Engineering, DevOps, Information Security, Knowledge Management, and other technology and business stakeholders to ensure data is accessible, secure, traceable, and appropriately governed.

KEY RESPONSIBILITIES

• Lead the design, implementation, and ongoing evolution of the Firm’s Azure Databricks data architecture, including lakehouse layers, storage, data models, integration patterns, and the roadmap from current-state systems to the target architecture.
• Partner with attorneys, practice groups, business teams, and technology stakeholders to identify priority data requirements, define data products, and establish measurable standards for data quality, freshness, availability, and usability.
• Design conceptual, logical, and physical data models for enterprise information, including client, matter, people, document, financial, and operational data, establishing consistent definitions, identifiers, relationships, and standards in partnership with data owners.
• Design, build, and maintain scalable data ingestion and transformation pipelines using Python, SQL, Apache Spark, Delta Lake, and related technologies, selecting appropriate batch, incremental, change-data-capture, or streaming approaches based on business requirements.
• Integrate data from enterprise databases, APIs, files, document repositories, and other systems through supported interfaces, including the development of source-to-target mappings, data contracts, reconciliation processes, and controls for schema changes and deletions.
• Design and administer data governance within Unity Catalog, including catalogs, schemas, ownership structures, access policies, lineage, classification, retention, and audit requirements.
• Partner with Information Security and data owners to design, implement, and validate access controls that appropriately reflect source-system permissions, client and matter restrictions, ethical walls, and other confidentiality requirements.
• Establish and maintain data quality standards, automated validation, monitoring, recovery procedures, and service expectations. Troubleshoot data and pipeline failures and optimize reliability, query performance, compute utilization, storage, and overall platform costs.
• Develop curated datasets and governed data interfaces supporting analytics, AI agents, retrieval-augmented generation, and other AI-enabled workflows while maintaining appropriate source traceability and access controls.
• Design and implement Lakebase PostgreSQL data stores supporting agentic applications, including persistent agent state, checkpoints, and memory, with appropriate user and matter isolation, transactional access patterns, retention, and recovery.
• Establish reusable architectural standards, technical documentation, and engineering patterns and provide technical guidance, design review, code review, and mentorship to AI Engineers and other technical team members.
• Partner with DevOps and other technology teams to support secure environments, automated deployments, development/test/production processes, monitoring, operational readiness, and long-term platform supportability.
• Remain current on developments in data architecture, Azure Databricks, cloud data engineering, AI data infrastructure, governance, and related technologies, recommending enhancements where appropriate.

REQUIRED EDUCATION, KNOWLEDGE & EXPERIENCE

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical discipline, or equivalent combination of education and relevant professional experience.

• Significant experience designing, implementing, and operating enterprise data architectures, with demonstrated ability to evaluate tradeoffs involving integration, governance, security, performance, scalability, and cost.

• Advanced hands-on experience with Azure Databricks, Apache Spark, Delta Lake, and Unity Catalog in production environments.

•  Strong experience designing conceptual, logical, and physical data models, including dimensional modeling, entity relationships, shared business definitions, and historical data management.

•   Advanced proficiency with SQL and Python, including PySpark, and experience developing maintainable production data pipelines using version control, automated testing, and code review practices.

• Demonstrated experience integrating data from multiple enterprise systems, including resolving inconsistent identifiers and definitions and implementing reliable incremental processing, reconciliation, and data validation.

•  Strong understanding of Azure lakehouse architecture, including layered raw, validated, and curated data structures and the appropriate selection of ingestion, transformation, storage, and serving patterns.

• Experience designing and administering Unity Catalog environments, including catalog and schema structures, privileges, managed and external data, lineage, and integration with Azure storage and identity controls.

•  Working knowledge of Azure Data Lake Storage Gen2 and Microsoft Entra ID, including managed identities, secrets management, authentication, authorization, and secure connectivity.

•  Experience designing production-grade data pipelines incorporating orchestration, incremental processing or change data capture, schema evolution, retries, safe reprocessing, monitoring, and automated data quality validation.

• Experience with PostgreSQL data modeling and transactional design, including persistent application or agent state, access controls, retention, and data lifecycle management.

• Demonstrated knowledge of Spark and SQL performance optimization, compute sizing, cost management, environment management, monitoring, and disaster recovery or operational recovery practices.

• Strong understanding of enterprise data governance, security, confidentiality, classification, lineage, retention, auditability, and role-based access controls. Demonstrated ability to translate complex technical concepts and architectural decisions for both technical and non-technical audiences. 

• Strong collaboration, communication, analytical, and problem-solving skills, with the ability to work effectively across technology and business functions.

• Ability to provide technical leadership and mentorship while remaining actively involved in architecture, engineering, development, and implementation.

PREFERRED SKILLS & KNOWLEDGE

• Master’s degree in Computer Science, Data Engineering, Information Systems, or a related discipline.

•    Experience with master and reference data management, entity resolution, data stewardship, and enterprise data governance across complex systems.

• Experience developing governed data foundations for artificial intelligence, machine learning, AI agents, and retrieval-augmented generation (RAG), including document preparation, metadata management, vector search, or knowledge graphs.

•   Experience with Databricks SQL, business intelligence integrations, semantic models, and governed datasets supporting enterprise reporting and analytics.  Experience with infrastructure as code and automated deployment technologies such as Terraform, Azure DevOps, or GitHub Actions.

• Experience modernizing legacy data platforms or migrating enterprise data environments to cloud-based architectures.

•   Experience working in legal services, professional services, financial services, or another environment involving highly sensitive information, complex confidentiality requirements, and sophisticated access controls.

•   Experience in technical consulting or another business-facing technology delivery role requiring direct engagement with business stakeholders and senior leaders.

PHYSICAL REQUIREMENTS

• Ability to sit and stand for extended periods.

• Ability to lift up to 15 pounds.

The expected salary range for this position is $200,000 - $230,000. Final compensation will be determined based on several factors, including but not limited to, relevant experience, qualifications, skill set, and geographic location.

Pillsbury Winthrop Shaw Pittman LLP is an Equal Opportunity Employer.

If you require an accommodation in order to apply for a position, please contact us at [email protected].