Distinguished Data Engineer

Job Description

Title                  Distinguished Data Engineer

Department       AMP

Location           Bengaluru

Reports To        Research & Sustainable Investing Data Engineering Lead

Level                 5

We share a commitment to making things better for clients and each other. We continually explore new technology and different ways of working to put our clients first. Bring your boldest ideas to our Research & Sustainable Investing Technology team and feel like you are making progress.

About your team

AMP Delivery is responsible for the design and delivery of all changes in business process and/or technology solutions that support the growth for Fidelity’s Global Investment Solutions & Services business. We partner with Investment Management, Asset Management Operations and Distribution teams across London, Hong Kong, Tokyo, Toronto, Australia, Singapore, and China.  

The Research & Sustainability team delivers strategic initiatives that enhance investment decision-making through modern research workflows, investment data platforms, sustainability capabilities, advanced analytics, and AI-powered solutions. We are increasingly leveraging Large Language Models (LLMs), frontier AI models, and agentic workflows to transform how investment professionals discover insights, conduct research, and make investment decisions across Equities, Fixed Income, and Multi-Asset. 

About your role

This is a specialist, hands-on AI data engineering role within the AMP - Research & Sustainable Investing team. Reporting to the Data Engineering Lead, you will build the trusted data foundations that power machine learning, Large Language Model (LLM), retrieval-augmented generation (RAG), search and agentic AI workflows for investment research and sustainability use cases.

You will engineer structured, semi-structured and unstructured data through the full AI data lifecycle - ingestion, transformation, enrichment, feature and evaluation dataset creation, embedding generation, vector indexing, retrieval and governed delivery. You will work primarily across Snowflake, AWS and Kafka while integrating with enterprise sources including Oracle and Microsoft SQL Server.

You will work closely with AI/ML engineers, data engineers, research analysts, sustainability specialists and investment teams to make AI data accurate, discoverable, traceable, secure and production-ready. The role directly contributes to Accelerated Innovation, Cost Optimisation, Risk Mitigation and Business Enablement.

Key Responsibilities

• Design, build, test, deploy and operate production-grade data pipelines for structured, semi-structured and unstructured research, sustainability and investment data.
• Build AI-ready datasets for machine learning and generative AI, including feature, training and evaluation datasets, embeddings, vector indexes and retrieval-augmented generation workflows.               
• Engineer retrieval data flows that curate, enrich, index and serve trusted content for semantic search, vector search and LLM-based applications, with appropriate metadata and provenance.
• Help to develop metadata and knowledge structures, including taxonomies, ontologies, entity resolution and knowledge graphs where appropriate, to improve data discovery, retrieval and contextual understanding.
• Build reusable data services and APIs that expose governed investment data to AI/ML models, applications, analytics and agentic workflows.
• Apply data quality, lineage, data contracts, security, privacy, entitlements, auditability and provenance controls across data pipelines and AI-ready data products.
• Use Snowflake, AWS, Kafka and enterprise data platforms to ingest and prepare data, integrating safely with Oracle and Microsoft SQL Server where source or legacy data is required.
• Apply strong software and data engineering practices to data pipelines, including Python/SQL development, Git, CI/CD, automated testing, observability, monitoring, performance tuning and secure production support.
• Partner with AI/ML engineers and investment stakeholders to translate AI use cases into reusable data products, trusted retrieval layers and measurable production outcomes.
• Investigate complex data, embedding, indexing, retrieval and pipeline issues, perform root-cause analysis and implement sustainable fixes with the Data Engineering Lead and wider team.

About You

Core Technical Skills

• Data Engineering: Strong hands-on experience preparing structured, semi-structured and unstructured data for machine learning and generative AI, including feature, training and evaluation datasets.
• Embeddings, Vector Search and RAG: Practical experience building embedding pipelines, vector indexes, retrieval workflows and RAG data foundations for applications.
• Evaluation Data: Experience creating and curating evaluation datasets and applying data-quality, relevance, traceability and provenance checks to support reliable retrieval and model evaluation.
• Data Pipelines: Strong experience designing production-grade ingestion, transformation, enrichment and delivery pipelines using batch, near-real-time, streaming, API and event-based patterns.
• Programming: Strong proficiency in Python, Spark and SQL, with the ability to build maintainable, tested production code for data preparation, embeddings, indexing, retrieval and data services.
• Data Platforms and Integration: Experience with Snowflake, AWS and Kafka, plus enterprise relational sources such as Oracle or Microsoft SQL Server; able to integrate modern data and retrieval workloads with existing platforms.
• Data Services and Orchestration: Experience building reusable APIs and data services and using orchestration tools such as Airflow or Control-M to operate dependable production data workflows.
• Engineering Practices: Experience with Git, CI/CD, automated testing, infrastructure as code, data observability, monitoring, performance tuning and secure production engineering.
• Governance and Responsible AI: Good understanding of data quality, lineage, data contracts, access control, privacy, entitlements, auditability, provenance and responsible AI controls across governed data products and AI-ready datasets.
• Data Modelling and Semantic Layers: Good understanding of canonical and dimensional modelling, semantic layers and governed business concepts that can be consumed consistently by analytics and AI-enabled solutions.

Professional Experience

• Demonstrated hands-on experience delivering production data engineering solutions for AI/ML use cases, particularly data pipelines and data foundations supporting machine learning, LLMs, search, retrieval and generative AI.
• Experience working closely with AI/ML engineers, data engineers, architects, analysts and business stakeholders to move data and retrieval workflows from experimentation into secure, reliable and supportable production services.
• Ability to own assigned data engineering components from design through build, testing, deployment, monitoring and production support.                       
• Strong communication skills, with the ability to explain data engineering, retrieval, metadata, data quality and governance topics clearly and translate investment and AI use cases into practical engineering solutions.

Key Soft Skills

• Technical Ownership: Takes accountability for the quality, security, resilience, provenance and maintainability of data products and AI-ready data components from build through production support.
• Problem-Solving: Applies strong analytical judgement to ambiguous data, retrieval and integration problems, investigates root causes and drives issues to sustainable resolution.
• Collaboration: Works effectively with AI/ML engineers, data engineers and investment stakeholders, contributes constructively to reviews and shares knowledge across the team.
• Communication: Explains complex data engineering, retrieval and governance topics clearly to technical and non-technical audiences and documents decisions and solutions effectively.
• Learning and Adaptability: Keeps current with evolving data engineering and AI-enablement approaches, experiments responsibly and applies new techniques where they create measurable value.

Feel rewarded

For starters, we'll offer you a comprehensive benefits package. We'll value your wellbeing and support your development. And we'll be as flexible as we can about where and when you work - finding a balance that works for all of us. It's all part of our commitment to making you feel motivated by the work you do and happy to be part of our team. For more about our work, our approach to dynamic working and how you could build your future here, visit careers.fidelityinternational.com.