Big Data Principal

Senior / Lead Data Engineer

Engineering –Enterprise Data Platform

About the Role

We are looking for a Senior / Lead Data Engineer to join the engineering organisation a multi-year banking transformation programme delivering capabilities across Accounts (Real-Time Ledger), Payments, Foreign Exchange and Client Lifecycle Management, underpinned by an Enterprise Data Platform built on Snowflake.

This is a hands-on build role. Working closely with the Enterprise Data Architect, you will turn the target-state data architecture into a production platform: standing up the platform foundations, engineering data acquisition and transformation, and implementing canonical data models and data products. You will act as the engineering counterpart to the Data Architect – implementing the design, feeding back on buildability, and setting the engineering standards others follow. At Lead level, you will also shape and lead the data engineering team.

Key Responsibilities

Platform foundations

• Build Snowflake environment structures – layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions

• Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails

• Manage the platform as code – Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control

• Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments

Data acquisition and CDC

• Assess source systems and recommend fit-for-purpose acquisition patterns – full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub

• Advise on where change data capture should be applied and owned – log-based CDC at source, application-emitted change events, or warehouse-side change detection – and implement the chosen patterns

• Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills

• Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling

• Configure and optimise managed ELT connectors (e.g. Fivetran), and build file- and API-based ingestion where required

• Design and build a metadata-driven ingestion framework – sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing – so new feeds are onboarded through metadata rather than new code

• Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling

• Prove patterns through technical spikes, and record decisions as Architecture Decision Records

Transformation and modelling

• Design and build dbt projects from the ground up – structure, layering, conventions, macros, packages, tests and documentation

• Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework

• Implement canonical data models – including industry-aligned models such as BIAN-based business objects – from source-to-canonical mapping specifications and governed reference data

• Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers

• Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings

Quality, governance and operations

• Embed data quality as code – tests, freshness checks, reconciliations and data contract enforcement

• Implement Snowflake governance controls – tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR – with lineage that supports BCBS 239

• Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes

Orchestration and scheduling

• Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) – dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs

• Orchestrate ingestion, dbt execution (e.g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications

• Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost

• Build operational visibility over task and Dynamic Table runs – run history, latency, failures and credit consumption – and keep orchestration designs portable to external orchestrators where needed

AI-assisted engineering

• Use Snowflake CoCo (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD

• Apply Snowflake Cortex AI capabilities – AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents – where they add value, and design data products to be AI-ready

Leadership and collaboration

• Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs

• Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts

• Define and own engineering standards, templates and reusable patterns

• Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery

Primary Skills (Essential)

Candidates are expected to have strong, hands-on depth in all of the following:

Secondary Skills (Desirable)

Experience in any of the following will strengthen an application:

Experience & Profile

• Senior: 10+ years in data engineering, including at least 3 years of hands-on Snowflake delivery in production. Lead: 12+ years, including leading a data engineering team or workstream

• Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own

• Has stood up a new data platform or major platform workstream – environments, access control, automation – rather than only extending an existing one

• Has built a dbt project from the ground up, including standards, testing and CI/CD

• Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production

• Strong hands-on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables

• Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade-offs between full, incremental and CDC approaches

• Comfortable reading and implementing architecture artefacts – canonical models, mapping specifications, ADRs – and offering well-reasoned implementation feedback

• Clear communicator, able to work with architects, application teams and business stakeholders

Senior vs Lead – Role Distinction