Sr. Databricks Azure Data Engineer

About Fusemachines

Fusemachines is a leading AI strategy, talent, and education services provider. Founded by Sameer Maskey Ph.D., Adjunct Associate Professor at Columbia University, Fusemachines has a core mission of democratizing AI. With a presence in 4 countries (Nepal, the United States, Canada, and the Dominican Republic) and more than 450 full-time employees, Fusemachines brings global AI expertise to transform companies worldwide. Founded in 2013, Fusemachines is a global provider of enterprise AI products and services, on a mission to democratize AI. Leveraging proprietary AI Studio and AI Engines, the company helps drive the clients’ AI Enterprise Transformation, regardless of where they are in their Digital AI journeys. With offices in North America, Asia, and Latin America, Fusemachines provides a suite of enterprise AI offerings and specialty services that allow organizations of any size to implement and scale AI. Fusemachines serves companies in industries such as retail,  manufacturing, and government.

Fusemachines continues to actively pursue the mission of democratizing AI for the masses by providing high-quality AI education in underserved communities and helping organizations achieve their full potential with AI.

Type: Remote, Full-time

Job Description:

This is a role responsible for designing, building, and maintaining the automated testing frameworks and Python SDKs required for data integration, parity verification, and user accessibility in the Industrial IoT / Smart Buildings domain.

We are seeking a Senior Data Engineer with strong Python and PySpark skills, and proven implementation of automated Data Quality frameworks and CI/CD pipelines, delivering Data and Analytics products using Agile methodology. The ideal candidate will possess strong technical, analytical, and interpersonal skills.

Qualification / Skill Set Requirement:

• 5+ years of hands-on data engineering experience with deep expertise in the Azure ecosystem.
• Expert Python skills (building Libraries/SDKs, Py4J).
• Strong experience with PySpark and DataFrames.
• Experience with Automated Testing frameworks (PyTest, JUnit) for data pipelines.
• Experience with Data Quality tools (Great Expectations, Deequ).
• Deep understanding of Apache Spark Internals (Catalyst Optimizer, Logical Plans).
• Experience with Databricks and Delta Lake optimization.
• Solid understanding of SDLC and Agile methodologies with hands-on experience in Azure DevOps, GitHub, CI/CD, and artifact management.
• Skilled in data modeling, data design, and data warehousing solutions on Azure Databricks.
• Knowledge of data governance, and security best practices within Azure (AD, NSG, encryption, compliance).
• Certifications preferred: Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional and Azure Solutions Architect Expert (nice to have).

Responsibilities

• Implement scalable and efficient data solutions on Spark and Flink.
• Develop a Python SDK that wraps core libraries for use in Jupyter Notebooks by data scientist and analyst..
• Build automated Parity Test Suite: CI/CD pipeline that runs queries against "Golden Datasets" in both Spark and Flink and asserts equality.
• Implement Data Quality (DQ) Rules configuration: Defining thresholds for "Completeness" and "Validity" that can be enforced.
• Manage and optimize Azure and Databricks resources, for performance, reliability, and cost-efficiency.
• Transform, clean, and prepare data using SQL, Python and Java.
• Monitor and fine-tune workloads and pipelines for optimal performance and reliability.
• Maintain CI/CD pipelines.
• Maintain clear documentation of solutions, configurations, and workflows.
• Actively participate in Agile team activities and continuous improvement initiatives.
• Promote and enforce data engineering best practices, including data governance, security, and data quality.

Fusemachines is an Equal Opportunities Employer, committed to diversity and inclusion. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or any other characteristic protected by applicable federal, state, or local laws.