Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.
- Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
- Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
- Implement robust checks for Data Quality Dimensions
- Build scalable data testing frameworks using python, pyspark and good knowledge of automation tools like playwright & pyspark based automation testing.
- Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
- Implement data quality-as-code practices using tools like Soda, dbt etc.
- Validate Spark transformations, Delta Lake tables, and streaming pipelines
- Perform large-scale data validation using PySpark and SQL
- Optimize validation logic for high-volume datasets
- Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
- Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
- Define and implement data quality SLAs and KPIs
- Build dashboards to track data health and pipeline reliability
- Proactively identify anomalies and data drifts
- Implement alerting mechanisms for data failures
- Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
- Translate business requirements into technical reporting solutions.
- Identify bottlenecks and improve report/query performance.
- Implement best practices for dashboard design and scalability.
- Establish BI standards, frameworks, and best practices.
Required Qualifications
- 8-10 years experience in enterprise data modeling and data architecture roles.
- Strong hands-on experience with Azure Data Factory & Pyspark in large-scale environments is preferred.
- Experience with Databricks (Delta Lake, Spark, Unity Catalog).
- Advanced SQL and strong understanding of pyspark scripts & using pyspark for Data validation.
- Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems).
- Strong understanding of data governance, data quality, metadata, and lineage.
- Excellent communication skills across business and technical audiences.
Core Technology Stack
- Analytics Platform: Databricks (Delta Lake, Unity Catalog), Microsoft PowerBI
- Languages: SQL, Spark SQL, Python / PySpark
- Governance/Data Quality: Unity Catalog/Informatica DG/DQ
#LI-KS1
About the TeamAt Wesco, we build, connect, power and protect the world. As a leading provider of business-to-business distribution, logistics services and supply chain solutions, we create a world that you can depend on. Skills Required
- 8-10 years experience in enterprise data modeling and data architecture roles
- Strong hands-on experience with Azure Data Factory & PySpark in large-scale environments
- Experience with Databricks (Delta Lake, Spark, Unity Catalog)
- Advanced SQL and strong understanding of PySpark scripts and using PySpark for data validation
- Experience integrating major enterprise applications (ERP, CRM, OMS, MDM, AR systems)
- Strong understanding of data governance, data quality, metadata, and lineage
- Experience integrating automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
- Experience building scalable BI dashboards and reports using Power BI, Tableau, or Looker
- Experience implementing data quality-as-code practices and familiarity with tools like Soda and dbt
- Excellent communication skills across business and technical audiences
WESCO International Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about WESCO International and has not been reviewed or approved by WESCO International.
-
Retirement Support — Feedback suggests the 401(k) with company match is a relative strong point and is consistently part of the package. Retirement offerings are frequently cited as a solid element within the total rewards.
-
Leave & Time Off Breadth — Feedback suggests PTO policies are favorable, including self‑managed or generous banks in some roles along with paid holidays. Paid parental leave and related time‑off options add to the perceived strength of leave benefits.
-
Flexible Benefits — Feedback suggests employees have meaningful choice through medical plan options with HSA/FSA alongside ancillary programs like EAP, discounts, and supplemental coverages. The breadth of selectable add‑ons supports tailoring benefits to individual needs.
WESCO International Insights
What We Do
At Wesco, we believe life should run smoothly. As a leading provider of business-to-business distribution, logistics services and supply chain solutions, we create a world that you can depend on. Harnessing 100 years of ingenuity and expertise, we increase profitability, improve productivity and mitigate risk for approximately 150,000 customers worldwide. With nearly 1.5 million products and locations in more than 50 countries, Wesco is your partner in progress. Our company’s greatest asset is our people. From our corporate and field offices to our distribution sites, Wesco employs over 20,000 professionals around the globe. We’re committed to fostering diversity and inclusion across our workforce by embracing the unique perspectives, authenticity, and individuality our team members contribute to the company. Headquartered in Pittsburgh, Wesco is a publicly traded (NYSE: WCC) FORTUNE 500® company with 2022 net sales of $21.4 billion.
