W2PE - Engineer

Trace3 is seeking a Fabric Data Engineer to accelerate the enterprise Data Modernization program by building and refactoring scalable, metadata-driven ETL pipelines in Microsoft Fabric. This role will design and implement medallion architecture (Bronze, Silver, Gold) using parameterized Python notebooks to move source data from landing through governed, trusted analytical layers, reducing dependence on legacy ETL platforms and enabling faster, more reliable speed-to-insight for enterprise reporting. The Fabric Data Engineer will work hands-on to modernize existing pipelines, apply reusable and parallelized data-processing patterns, and support the broader Fabric medallion implementation already underway as part of Phase 1C of the program. This is a delivery-focused engineering role with an explicit expectation of transferring architecture, code, and operating knowledge to the current data-engineering team as the program matures.

Fabric Pipeline & Metadata-Driven Engineering

• Design, build, and maintain metadata-driven data pipelines in Microsoft Fabric that are reusable across source systems and entities rather than built one-off per pipeline.
• Develop and maintain a metadata/configuration-driven framework that governs ingestion, transformation, and load behavior across the pipeline estate.
• Standardize ingestion patterns using parameterized notebooks to minimize duplicate logic and simplify ongoing maintenance.

Medallion Architecture Implementation

• Build and refactor Bronze, Silver, and Gold layers in the Fabric Lakehouse/Warehouse, ensuring each layer meets defined validation and quality gates before promotion.
• Preserve source fidelity in the landing zone and implement conformed, deduplicated, and historized data in Silver in support of downstream Gold transformations.
• Apply warehouse-side transformation logic and support native-service patterns (e.g., Dataflows Gen2, Fabric Data Factory) as part of the broader platform modernization effort.

Python Notebook Development & Parallel Processing

• Develop Python notebooks (including PySpark-based processing) to support scalable, parallel data-processing pipelines that improve data-processing efficiency and reduce time to insight.
• Build restartable, incremental, and schema-aware processing logic that scales across growing data volumes and source counts.

Data Quality, Validation & Operational Controls

• Implement data validation and quality checks at each layer (landing, Bronze, Silver, Gold) rather than treating validation as an end-stage activity.
• Maintain operational evidence for pipeline runs, notebook executions, merges, and quality results to support troubleshooting and auditability.
• Support performance tuning, refresh-frequency optimization, and capacity-efficient pipeline design within Fabric.

Collaboration, Documentation & Knowledge Transfer

• Partner with the Director, Data and Analytics Enablement and the current data-engineering team to align pipeline design with governance, metadata, and certified-dataset standards.
• Document architecture, code, and runbooks, and actively transfer operating knowledge so the current data-engineering team can independently operate, troubleshoot, and extend delivered pipelines.
• Collaborate with reporting, governance, and platform stakeholders to ensure pipeline outputs support trusted, consistent enterprise reporting.

• 3+ years of hands-on data engineering experience building production ETL/ELT pipelines.
• Demonstrated experience designing and implementing metadata-driven data pipelines (parameterized, configuration-based pipeline patterns rather than one-off builds).
• Strong proficiency in Python, including experience building notebook-based data-processing pipelines (PySpark or equivalent) for parallel/distributed processing.
• Hands-on experience with Microsoft Fabric and/or Databricks, including notebooks, Lakehouse/Warehouse objects, and pipeline orchestration.
• Practical experience implementing medallion (Bronze/Silver/Gold) data architecture, including data validation, deduplication, and historization patterns.
• Working knowledge of SQL and relational data warehousing concepts.
• Experience with cloud-based analytics platforms (Azure preferred).
• Strong troubleshooting skills and ability to document and communicate technical designs to both technical and non-technical stakeholders.

It’s a Plus If You Have:

• Experience migrating pipelines from Azure Data Factory or similar legacy ETL platforms into Fabric-native services.
• Familiarity with Power BI semantic models and downstream reporting consumption patterns.
• Microsoft Fabric certifications (e.g., DP-600, DP-700) or equivalent Databricks certifications.
• Experience operating in a consulting or client-delivery environment.

While performing the standard duties of this job, the employee can expect light physical exertion with long periods of sitting, occasional standing and walking; limited bending, crouching, stooping, stretching, reaching, or similar activities; occasional lifting/moving of items up to 25 pounds. Hearing and speech communication in person and over the telephone is essential to fulfilling the duties of this position. Specific vision abilities required to read printed materials and a computer screen, including close vision and the ability to adjust focus.