Senior Data Engineer- Remote, India

Responsibilities

• Understand the values and vision of the organization
• Protect the Intellectual Property
• Adhere to all the policies and procedures
• Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.
• Build and optimize data architectures and data models (Lakehouse / medallion, dimensional) for efficient data storage and retrieval.
• Develop ETL/ELT processes to transform and load data from various sources into data warehouses and data lakes.
• Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL and Delta Lake, orchestrated with Databricks Workflows.
• Integrate data from enterprise source systems including SAP (ABAP/CDS extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.
• Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB) and Azure DevOps (Git repositories, YAML build and release pipelines), promoting code across dev, QA and production.
• Implement data quality, validation, freshness and reconciliation checks with pipeline observability.
• Ensure data integrity, quality, and security across all data systems.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
• Document data processes, architectures, and data flow diagrams.

Essential Skills

Job 

• 7 - 8 years of hands-on data engineering experience building and running production data pipelines at scale.
• Strong expertise in Azure and Azure data services (ADLS Gen2, Azure Databricks, Azure DevOps).
• Deep hands-on experience with Databricks: PySpark, Spark SQL, Delta Lake, Lakehouse / medallion architecture and Databricks Workflows.
• CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure DevOps (Git, YAML build/release pipelines, multi-environment promotion). (Must-have)
• Strong Snowflake experience (data modeling, performance tuning, loading and optimization).
• Proficiency in SQL and Python.
• Experience integrating data from SAP and other enterprise ERP / source systems into a data lake or warehouse.
• Solid data modeling (dimensional, star/snowflake, Lakehouse) and ETL/ELT design.
• Building data-quality, validation, reconciliation and pipeline monitoring / observability.

Personal 

• Excellent communication and interpersonal skills, with the ability to engage with all levels of employees and management. 
• Collaborative approach to effectively present and advocate for quick design solutions. 
• Stay updated on the latest design trends, tools, and technologies, bringing innovative ideas to enhance the product experience.
• A proactive approach to problem solving, with a focus on delivering exceptional customer satisfaction.

Certifications

• At least one current certification is required; multiple is a strong plus:
• Databricks Certified Data Engineer Associate or Professional. (Required)
• Microsoft Certified: Azure Data Engineer Associate (DP-203), or Azure Fundamentals (DP-900 / AZ-900).
• SnowPro Core or SnowPro Advanced: Data Engineer (Snowflake).

Preferred Skills

Job

• Familiarity with SAP finance / ERP data domains (Accounts Receivable, invoice-to-pay, bank statements).
• Streaming and event-driven pipelines (Apache Kafka / Azure Event Hubs).
• Workflow orchestration (Apache Airflow, Databricks Workflows).
• Databricks Unity Catalog and Delta Live Tables.
• Infrastructure as Code (Terraform) and containerization (Docker).
• Data governance, lineage and cost/performance optimization on Databricks and Snowflake.
• Exposure to BI / visualization tools (Power BI, Tableau).

Personal 

• Demonstrate proactive thinking
• Strong communication and collaboration skills.
• Should have strong interpersonal relations, expert business acumen and mentoring skills
• Strong problem-solving skills with attention to detail.
• Have the ability to work under stringent deadlines and demanding client conditions.
• Strong analytical and problem-solving skills.
• Ability to work independently and as part of a team. 

Other Relevant Information

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 7 - 8 years of experience in data engineering & architecture, including hands-on Databricks and Azure DevOps CI/CD.
• This role offers the flexibility of working remotely in India.