Job Summary
Seeking an experienced Data Engineer to develop and optimize data pipelines, lakehouse solutions, and data warehouse workloads using Azure Databricks and Azure Synapse Analytics. The role focuses on batch and streaming data integration, data modeling, orchestration, performance tuning, and secure data governance for sensitive health data in compliance with HIPAA and agency policies.
Key Responsibilities
Build and maintain batch and streaming pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.
Implement medallion architecture (Bronze/Silver/Gold) and incremental data ingestion.
Develop and optimize data models, SQL queries, and warehouse workloads in Azure Synapse Analytics.
Orchestrate workflows using Azure Data Factory and Databricks Workflows.
Implement data governance, security, and access controls using Unity Catalog, Entra ID, and Key Vault.
Manage CI/CD, source control, monitoring, troubleshooting, and pipeline performance optimization.
Collaborate with architects, analysts, data scientists, and project managers on requirements and technical solutions.
Maintain technical documentation and support data quality, governance, and operational improvements.
Required Skills & Qualifications
4+ years of relevant hands-on data engineering experience or equivalent education and experience.
Strong experience with PySpark, Spark SQL, Delta Lake, Databricks notebooks, and medallion architecture.
Hands-on experience with Unity Catalog, Delta Lake transactions, schema evolution, and Spark performance tuning.
Experience with Azure Synapse Analytics, MPP architecture, dimensional modeling, star schemas, SCD Type 1/2, and SQL optimization.
Proficiency in Azure Data Factory, Databricks Workflows, ADLS Gen2, and pipeline orchestration.
Experience with Azure DevOps/GitHub, CI/CD, managed identities, security controls, and Azure monitoring tools.
Knowledge of Microsoft lakehouse patterns and Synapse–Databricks integration.
Strong written and verbal communication skills.
Preferred Qualifications
Databricks Certified Data Engineer Associate or Professional.
Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.
Experience with Power BI connectivity to Synapse, including Import vs. DirectQuery and dashboard performance optimization.
Skills Required
- 4+ years of relevant hands-on data engineering experience or equivalent education and experience
- Strong experience with PySpark, Spark SQL, Delta Lake, Databricks notebooks, and medallion architecture
- Hands-on experience with Unity Catalog, Delta Lake transactions, schema evolution, and Spark performance tuning
- Experience with Azure Synapse Analytics, MPP architecture, dimensional modeling, star schemas, SCD Type 1/2, and SQL optimization
- Proficiency in Azure Data Factory, Databricks Workflows, ADLS Gen2, and pipeline orchestration
- Experience with Azure DevOps or GitHub, CI/CD, managed identities, security controls, and Azure monitoring tools
- Knowledge of Microsoft lakehouse patterns and Synapse-Databricks integration
- Strong written and verbal communication skills
- Databricks Certified Data Engineer Associate or Professional
- Microsoft Certified Azure Data Engineer Associate DP-203
- Experience with Power BI connectivity to Synapse, including Import versus DirectQuery and dashboard performance optimization
What We Do
Info Origin is an AI-first technology company. We design and deploy intelligent systems that help enterprises operate with clarity, speed, and scale. From advanced analytics to autonomous workflows, our work is grounded in real-world impact. Our approach brings together data, engineering, and AI to modernize applications, unlock insight, and orchestrate complex operations with precision. Our products and platforms extend this capability: AgentFarm — multi-agent, enterprise-grade, policy-governed AI workforce that enables structured execution, resilient workflows, and full auditability, with parallel reasoning, tool orchestration, and real-time observability. Contract Insights — AI-driven clause extraction, NL search multi-document comparison engine for legal and commercial workflows. CDMS — intelligent document creation, collaboration, and compliance, powered by AI-based automation. Info QA — codeless test automation with end-to-end traceability. We partner with organizations to translate AI ambition into measurable outcomes - securely, responsibly, and at scale.
