The Role
Build and maintain data foundations for AI agents, analytics, scorecards, and dashboards. Responsibilities include developing ingestion pipelines and enterprise connectors, modeling raw-to-mart layers, managing metadata and vector indexes, implementing data quality and lineage controls, and supporting governed data stores. The role also covers API integrations, incremental ingestion, security controls, testing, release activities, and data validation. Preferred experience includes RAG preparation, unstructured content processing, knowledge graphs, LLM applications, and regulated life sciences data.
Summary Generated by Built In
Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation.
Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks.
Key responsibilities- Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata.
- Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs.
- Model and build raw-to-mart data layers that serve analytics and AI use cases.
- Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG.
- Set up and manage vector databases and knowledge repositories used by LLM agents.
- Implement data quality rules, profiling, scoring outputs, and exception handling.
- Register lineage and maintain source registries, version tracking, and refresh controls.
- Design data stores for signals, findings, audit trails, and user feedback loops.
- Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling.
- Support SIT/UAT data validation, defect fixes, and production release activities.
- Strong Python and SQL; solid data modeling (dimensional and normalized).
- Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services.
- Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar).
- API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns.
- Data quality frameworks and testing practices for pipelines.
- Version control (Git) and CI/CD for data workloads.
- RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar).
- Unstructured content processing: text extraction, OCR, document parsing.
- Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases).
- Experience supporting LLM or agentic applications with grounded, traceable data.
- Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling.
- Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake).
Skills Required
- 3 to 6 years of experience in data engineering
- Strong Python and SQL skills
- Solid dimensional and normalized data modeling experience
- Hands-on experience with Databricks, Snowflake, or Azure/AWS data services
- Experience with pipeline orchestration and transformation using Spark, Airflow, ADF, dbt, or similar tools
- Experience with REST API integration, JSON handling, and incremental or CDC ingestion patterns
- Experience with data quality frameworks and pipeline testing
- Experience with Git and CI/CD for data workloads
- Data foundation architecture and connector framework ownership experience for Senior Data Engineer applicants
- RAG data preparation, chunking, embeddings, and vector databases
- Unstructured content processing, text extraction, OCR, or document parsing
- Metadata management, data catalogs, ontologies, knowledge graphs, or graph databases
- Experience supporting LLM or agentic applications with grounded, traceable data
- Life sciences data exposure and regulated-data handling
- Cloud certification in Azure Data Engineering, Databricks, AWS, or Snowflake
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Techdome is a technology consultancy and IT solutions provider founded in 2020. It helps businesses modernize operations and navigate digital transformation through scalable, end-to-end services spanning planning, building, designing, developing, and launching technology solutions, alongside AI, cloud computing, automation, data engineering, machine learning, custom large language models, and cybersecurity. Its mission is to make complex technology accessible and support business growth across diverse industries and markets.
