Position: Data Quality Engineer (Databricks DQX)
Location: Windrose Ave., Plano, TX 75024
Job Type- 12-months contract
Job Description:
Design and run the data quality layer of our new Databricks lakehouse using Databricks Labs DQX. You will turn business rules from the legacy SQL platform into automated checks, quarantine bad records before they reach Gold tables, and give the business visibility into data health.
Key responsibilities:
Profile source and Bronze data with the DQX profiler and turn the output into reviewed quality rules.
Define DQX checks (completeness, uniqueness, validity, ranges, referential and custom SQL checks) in YAML/JSON config or code.
Set criticality per rule (error vs warn) and implement split/quarantine patterns for invalid rows.
Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines for batch and streaming data.
Store check definitions and results in Delta tables under Unity Catalog; build DQ dashboards and alerts.
Translate legacy SQL validation logic (constraints, stored procedure checks) into equivalent DQX rules.
Define DQ thresholds that act as quality gates for layer promotion (Bronze → Silver → Gold).
Version-control rules and deploy them through CI/CD.
Must-have skills :
3+ years in data engineering or data quality roles.
Hands-on PySpark and Spark SQL on Databricks.
Practical experience with DQX, or strong experience with a comparable framework (Great Expectations, Soda, Deequ, DLT expectations) and willingness to ramp up on DQX quickly.
Delta Lake and Unity Catalog fundamentals.
Understanding of data quality dimensions and how to measure them.
Git-based workflow and CI/CD basics.
Nice-to-have skills :
Azure Data Factory orchestration of Databricks jobs.
Databricks SQL dashboards, Lakehouse Monitoring or Power BI for DQ reporting.
Experience with SQL Server to cloud migrations.
Databricks Certified Data Engineer Associate/Professional.
Responsibilities:
DQX checks live on every Silver and Gold table in scope.
Invalid records quarantined with a documented triage process.
ADQ dashboard the business uses to track trends and SLA breaches.
All your information will be kept confidential according to EEO guidelines.
Skills Required
- 3+ years of experience in data engineering or data quality roles
- Hands-on experience with PySpark and Spark SQL on Databricks
- Practical experience with Databricks DQX or a comparable data quality framework, with willingness to learn DQX
- Fundamentals of Delta Lake and Unity Catalog
- Understanding of data quality dimensions and measurement
- Git-based workflow and CI/CD basics
- Experience with Azure Data Factory orchestration of Databricks jobs
- Experience with Databricks SQL dashboards, Lakehouse Monitoring, or Power BI for data quality reporting
- Experience with SQL Server to cloud migrations
- Databricks Certified Data Engineer Associate or Professional certification
What We Do
BuzzClan is a technology consulting firm collaborating to provide software, advisory, implementation, transformation and staffing services. We provide systems integration, IT consulting, managed services and workforce solutions to public and private sector clients. BuzzClan is a certified partner for most Tier-1 Cloud, Software & Hardware Providers, including Oracle, Amazon, Microsoft & Saviynt. BuzzClan is committed to providing substantive business value on each and every client engagement. We do this through a combination of industry-specific business expertise, technical skills, proven project management methods and our “onsite - off-site - offshore” delivery model. We strive to work in partnership with our customers to build high-performance teams and create business solutions that will last. Vertically Integrated solutions company Full ownership of the solution The combined experience of more than 250+ years Various OEM Certifications such as OCP, PMP, AWS, MS Azure, etc. Onsite presence across the North American(US & Canada) clients Near-Shore & Offshore presence
