The Role
Build and maintain Azure-based data pipelines for governance and analytics. Develop Databricks ETL/ELT workflows using PySpark, SQL, and Delta Lake; automate data quality, lineage, monitoring, access auditing, cost analysis, retention tracking, policy enforcement, and metadata integration with Microsoft Purview. Maintain curated governance metric tables, optimize performance, troubleshoot pipeline issues, and collaborate with engineering, QA, and reporting teams while meeting security and privacy standards.
Summary Generated by Built In
Job title : Data Engineer
Experience: 6+ years
Location: Chennai
Shift: US Eastern Time ( 5:00 PM – 2:00 AM )
Requirements
We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated
data pipelines supporting data governance and analytics initiatives. This role will focus on
building production-ready workflows for ingestion, transformation, quality checks, lineage
capture, access auditing, cost usage analysis, retention tracking, and metadata integration,
primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.
Experience: 6+ years in data engineering, with strong Azure and Databricks experience
Pipeline Development – Design, build, and deploy robust ETL/ELT pipelines in Databricks
(PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational
metadata from multiple sources landed in Databricks.
Granular Data Quality Capture – Implement profiling logic to capture issue-level metadata
(source table, column, timestamp, severity, rule type) to support drill-down from dashboards
into specific records and enable targeted remediation.
Governance Metrics Automation – Develop data pipelines to generate metrics for
dashboards covering data quality, lineage, job monitoring, access & permissions, query cost,
usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and
governance KPIs.
Microsoft Purview Integration – Automate asset onboarding, metadata enrichment,
classification tagging, and lineage extraction for integration into governance reporting.
Data Retention & Policy Enforcement – Implement logic for retention tracking and policy
compliance monitoring (masking, RLS, exceptions).
Job & Query Monitoring – Build pipelines to track job performance, SLA adherence, and
query costs for cost and performance optimization.
Metadata Storage & Optimization – Maintain curated Delta tables for governance metrics,
structured for efficient dashboard consumption.
Testing & Troubleshooting – Monitor pipeline execution, optimize performance, and resolve
issues quickly.
Collaboration – Work closely with the lead engineer, QA, and reporting teams to validate
metrics and resolve data quality issues.
Security & Compliance – Ensure all pipelines meet organizational governance, privacy, and
security standards.
Skills Required
- 6+ years of experience in data engineering
- Strong experience with Azure
- Strong experience with Databricks
- Experience developing ETL/ELT pipelines using PySpark, SQL, and Delta Lake
- Experience with data governance, data quality, lineage, metadata, retention, and policy compliance
- Experience integrating Microsoft Purview for asset onboarding, metadata enrichment, classification, and lineage
- Experience with pipeline monitoring, performance optimization, and troubleshooting
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Verifitech provides AI-powered background verification and screening solutions for enterprises. Its platform supports employment background screening, credit verification, exit interview checks, drug checks, and verification services for housekeeping staff. The company uses modern technology to deliver fast, accurate, and compliant background verification reports, serving organizations across sectors including information technology, banking, logistics, finance, manufacturing, and automobiles.
