Data Engineer

About Division:
empowOR was ‘born’ out of the complex needs of multi-service agencies who are trying to:

• De-silo their data and approach
• Use data and technology to improve efficiencies in operations
• Coordinate services
• Use data for funder reporting & strategic analysis

The Position: empowOR and CADS data — the CADS refactor, the Fabric lakehouses and warehouses, and the pipelines and transforms that move and reshape customer data — is architected, built, and kept performant, so that linking, reporting, and integrations run reliably and scale.

Key Responsibilities

Enterprise Data Platform:

• Ensure Data Platform is scalable , secure and well supported, CADS is owned end to end — refactored onto a Fabric lakehouse architecture that scales, then supported and maintained.
• Own CADS data flow in full — meaning supporting the new infrastructure that utilizes MS Fabric, SPLINK (python based deduplication code), data flow and data syncs across fabric and a small DB for UI interactions
• Build and maintain the ingest and transform path — likely using SQL MERGE INTO / COALESCE — with the API acting as an ingest gateway rather than doing transactional per-record updates and utilizing queue tables and pipelines to process data
• Support migration of data
• Ensure SSNs and other sensitive fields are hashed or encrypted before storage in Fabric, per compliance requirements
• Participate in CADS Team / CADS Refactor and Architecture sessions

Data Pipelines & Integrations:

• Design, build, and maintain data pipelines and transforms across Microsoft Fabric and Azure Data Factory (ADF)
• Diagnose and resolve complex dataflow and transform issues, including the recurring ADF/SQL internal errors, and own the architecture-level fixes
• Serve as the senior escalation point for CADS, and pipeline issues raised by the Tech 2 — Fabric & Integrations specialist and support
• Write efficient SQL, troubleshoot poorly performing queries, and lead performance tuning across the data platform
• Gain familiarity and assist in support of internal data transformation code (called CDTF--core data transform framework)

Enterprise Data Warehouse & Semantic Models:   

• Write SQL to build data warehouse tables that have been designed
• Suggest improvements to data warehouse infrastructure
• Support the internal (semantic) model in PowerBI and measure updates (shared with the Reporting & Integration Manager)
• Build new data sets and pipelines that support Data Autonomy and downstream reporting
• Keep the model and measures correct so that Power BI / Report Builder outputs built downstream are accurate

Reporting Infrastructure & Business Intelligence:

• Build new Report Builder data sets
• Serve as the escalation point for Report Builder issues
• Engineer report queuing solutions
• Maintain mirrored databases for reporting
• Build and support data empowORment reporting offerings
• Make SQL improvements to reporting queries and structures as needed

Data Migration & Integration:  

• Support new data source integrations into CADS
• support, maintain and improve data integrations into and out of empowOR
• Design and lead data migration work
• Attend reporting & Integration Team meetings (as relevant to the warehouse, semantic model, and pipelines)

Release Management:

• Write QA plans for and QA architecture- and pipeline-level changes across CADS, transforms, the warehouse, and reporting structure
• Recreate the scenarios that cause an issue, test those and ancillary functions in the QA portal, leave notes in the task discussion, and follow the QA pass/fail process

Strategic Initiatives:

• Participate in strategic-plan projects as needed, including building in-house capability (e.g., updating the CADS UI / Breck model) where it reduces reliance on outside consultants

Required Skills & Knowledge

• 3–4 years of relevant experience required.
• Strong hands-on experience with Microsoft Fabric — lakehouse, warehouse, and semantic model
• Strong hands-on experience with Azure Data Factory (ADF) — building pipelines, not only monitoring them
• Demonstrated experience diagnosing and resolving data flow and transform issues, and designing transform/pipeline architecture (medallion and ETL/ELT patterns, MERGE/upsert logic)
• Advanced SQL — query authoring, performance tuning, and troubleshooting poorly performing queries
• Experience with data integration patterns — ingest APIs/gateways, SFTP, scheduled jobs, and file-based imports (parquet/JSON)
• Understanding of data security and compliance handling, such as hashing or encrypting sensitive fields (e.g., SSNs)
• Scripting experience in Python or a similar language; familiarity with entity-linking / record-matching concepts (e.g., Splink) a plus
• Comfortable working cross-functionally with engineering, DevOps, and the reporting team
• Familiarity with nonprofit-focused software and agency data needs a plus
  Work Shift
  4PM – 1 AM OR
  5PM – 2 AM Pakistan Time