Principal Data Engineer

End Date

We Support Flexible Working – Click here for more information on flexible working options

Flexible Working Options

Job Description Summary

Job Description

• Job Title: Engineering Leader – Data (Lloyds Technology Centre) 
  Location: Hyderabad, India (Hybrid: at least 2 days/week in office)  
  Experience: 15+ years 

   

  About Lloyds Technology Centre 

  Lloyds Technology Centre is the strategic technology hub for Lloyds Banking Group, enabling digital transformation and innovation across the banking ecosystem. Our mission is to build secure, scalable, and data-driven solutions that empower millions of customers and ensure compliance with financial regulations. 

  Role Overview 

  Own the delivery and operation of data products and platforms across multiple squads, with end‑to‑end accountability for outcomes, quality, and reliability. Establish reusable, metadata‑driven engineering patterns; elevate semantic models, KPI/metrics, and BI consumption; and build an AI‑ready foundation (knowledge layers, knowledge graphs, semantic models) that accelerates analytics and machine learning use‑cases across the organisation. 

  Key Responsibilities 

  • Delivery ownership: Define roadmaps, OKRs, and release plans; oversee scope, estimation, risk, and stakeholder communication; ensure on‑time, on‑budget, quality delivery across squads. 

  • Team ownership & leadership: Build and lead high‑performing teams (hiring, coaching, performance management); set coding standards, DoR/DoD, working agreements, and succession plans. 

  • Engineering patterns at scale: Establish factory‑mode delivery via inner‑sourced, reusable frameworks (ingestion, transform, quality, lineage, observability) and golden paths with strong documentation 

  • Semantic layer & metrics: Define and govern enterprise semantic models, KPI/metric definitions, conformed dimensions, metric stores, and query‑ready views to enable consistent BI and self‑serve analytics. 

  • AI readiness & knowledge layers: Shape data for ML/GenAI—ontologies, knowledge graphs, feature/embedding strategies, and patterns that make the platform AI‑ready by design. 

  • Data modelling & performance: Guide dimensional (star/snowflake) and Data Vault 2.0 modelling; set standards for physical design on cloud warehouses (partitioning, clustering, caching, workload mgmt.) 

  • Streaming & real‑time: Oversee event pipelines (Kafka/Pub/Sub/Kinesis + Flink/Spark) with exactly‑once semantics, replay, SLAs/SLOs, and resiliency patterns. 

  • Quality, metadata & lineage: Make quality the default—data contracts, DQ rules/tests, reconciliation—and automate capture/propagation of technical/business metadata and end‑to‑end lineage 

  • Observability & FinOps: Implement platform and pipeline telemetry (logs, metrics, tracing) and proactive cost/performance guardrails for cloud DW/compute; continuously optimise jobs/queries.  

  • Security & compliance: Champion IAM, least‑privilege patterns, encryption, secrets mgmt., and auditability; embed privacy‑by‑design and regulatory controls into pipelines and platforms. 

  • Vendor & partner management: Govern partner delivery (e.g., TCS/HCL), enforce standards, and ensure reusable IP is contributed back to inner‑source repos. 

   

  Required Skills 

  • Leadership & ownership: Proven record of owning complex data platform deliveries, leading multiple squads, and managing outcomes/SLAs in production. 

  • Semantic & BI expertise: Hands‑on experience defining semantic layers, KPI/metric logic, metric stores, and consumption models that scale across BI tools. 

  • AI‑ready design: Practical understanding of AI/ML data needs—feature engineering, data products for ML, knowledge graphs/ontologies, vector/embedding patterns—and how to operationalise them on cloud platforms.  

  • Modelling depth: Dimensional (star/snowflake), Data Vault 2.0, SCDs, schema evolution/versioning; ability to set and enforce modelling standards. 

  • Cloud data platforms: Strong experience with one or more of BigQuery, Snowflake, Redshift, Synapse/Databricks SQL; clarity on cloud DW vs traditional MPP trade‑offs and performance levers. 

  • Streaming systems: Kafka/Pub/Sub/Kinesis and Spark/Flink; event schema management (Avro/Protobuf), idempotency, back‑pressure, state mgmt. 

  • Software‑engineering mindset: Git‑based workflows, code reviews, automated testing, CI/CD, artifact versioning; inner‑sourcing culture and documentation excellence.  

  • IaC & runtime: Terraform/CloudFormation, Docker/Kubernetes; sizing, autoscaling, job concurrency, and reliability engineering. 

  • Data quality & governance: Data contracts; testing frameworks (e.g., Great Expectations/dbt tests); catalogue/lineage tooling; policy enforcement via pipelines.  

  • FinOps & observability: Cost modelling for cloud DW/compute, workload tuning, budgets/alerts; platform and pipeline telemetry with actionable SLOs.