Senior Data Engineer

Posted 5 Days Ago
Be an Early Applicant
Coimbatore South, Coimbatore, Tamil Nadu, IND
In-Office
Senior level
Information Technology • Software • Analytics
The Role
Designs and maintains scalable Azure-based ETL/ELT pipelines using Data Factory, Databricks, Python, SQL, PySpark, and Spark. Integrates diverse data sources, develops data models and warehouse structures, implements CDC and batch processing, and optimizes queries and distributed workloads. Manages orchestration, monitoring, error handling, CI/CD, documentation, and production support using Airflow, Git, GitHub, and Azure DevOps. Collaborates with cross-functional teams to troubleshoot issues and deliver reliable data solutions.
Summary Generated by Built In

  • Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
  • Develop data ingestion and transformation workflows using Python, SQL, and PySpark.
  • Work with Apache Spark and PySpark for large-scale distributed data processing.
  • Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.
  • Integrate data from multiple sources, including relational databases, APIs, and cloud storage.
  • Work with Azure Data Lake Storage Gen2 and Azure Blob Storage.
  • Develop and optimize data models, fact tables, dimension tables, and data warehouse structures.
  • Perform data cleansing, validation, deduplication, and aggregation.
  • Implement and manage workflows using Azure Data Factory and Apache Airflow.
  • Optimize SQL queries, Spark jobs, and data pipelines for better performance.
  • Work with Azure Synapse Analytics and relational databases.
  • Implement error handling, logging, monitoring, and alerting for production pipelines.
  • Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
  • Troubleshoot production issues, perform root cause analysis, and provide production support.
  • Maintain technical documentation and collaborate with cross-functional teams.


Requirements

Programming Languages

  • Python, SQL, PySpark.
  • Pandas, NumPy, OOP concepts, exception handling, and API integration.

Big Data Technologies

  • Apache Spark and PySpark.
  • Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions.
  • Partitioning, caching, and performance optimization.

Azure Cloud Technologies

  • Azure Data Factory (ADF).
  • Azure Databricks.
  • Azure Data Lake Storage Gen2 (ADLS Gen2).
  • Azure Blob Storage.
  • Azure Synapse Analytics.
  • Azure Key Vault and Azure Monitor.

ETL & Data Engineering

  • ETL/ELT pipeline development.
  • Data ingestion, transformation, and migration.
  • Change Data Capture (CDC).
  • Incremental loading and full loading.
  • Batch and distributed processing.

Database Technologies

  • Microsoft SQL Server, PostgreSQL, MySQL, and Oracle.
  • Complex SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

  • Dimensional and relational data modeling.
  • Fact and dimension tables.
  • Data normalization and denormalization.
  • Schema design and data warehouse development.

Workflow Orchestration & DevOps

  • Azure Data Factory and Apache Airflow.
  • Git, GitHub, Azure DevOps, and CI/CD pipelines.

File Formats

  • CSV, JSON, XML, Parquet, and Delta Lake.
  • 4–5 years of relevant experience in Data Engineering.
  • Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.
  • Experience developing and optimizing scalable data pipelines.
  • Good understanding of cloud-based data engineering and data warehousing concepts.
  • Strong analytical, debugging, and problem-solving skills.
  • Good communication and collaboration skills.


Skills Required

  • 4-5 years of relevant Data Engineering experience
  • Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL
  • Experience developing and optimizing scalable data pipelines
  • Understanding of cloud-based data engineering and data warehousing concepts
  • Experience with Python, SQL, PySpark, Pandas, NumPy, object-oriented programming, exception handling, and API integration
  • Experience with Apache Spark, Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations, window functions, partitioning, caching, and performance optimization
  • Experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2, Azure Blob Storage, Azure Synapse Analytics, Azure Key Vault, and Azure Monitor
  • Experience with ETL/ELT, data ingestion, transformation, migration, CDC, incremental loading, full loading, batch processing, and distributed processing
  • Experience with Microsoft SQL Server, PostgreSQL, MySQL, Oracle, complex SQL queries, indexing, stored procedures, and query optimization
  • Experience with dimensional and relational data modeling, fact and dimension tables, normalization, denormalization, schema design, and data warehouse development
  • Experience with Azure Data Factory, Apache Airflow, Git, GitHub, Azure DevOps, and CI/CD pipelines
  • Experience working with CSV, JSON, XML, Parquet, and Delta Lake file formats
  • Strong analytical, debugging, and problem-solving skills
  • Good communication and collaboration skills
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
174 Employees
Year Founded: 2020

What We Do

InFynd is a cloud-based B2B data and prospecting platform that helps sales, marketing, and recruiting teams identify and engage potential customers. It provides contact information, company profiles, buyer-intent data, social media handles, and verified business email addresses and phone numbers. The platform supports target identification, company research, lead generation, and more effective sales conversion through accurate, GDPR-compliant business intelligence.

Similar Jobs

Kyndryl Logo Kyndryl

Senior Data Engineer

Cloud • Information Technology • Consulting
In-Office
2 Locations
46070 Employees

Kumaran Systems Logo Kumaran Systems

Senior Data Engineer

Information Technology • Consulting
In-Office
Chennai, Tamil Nadu, IND
1078 Employees

ShyftLabs Logo ShyftLabs

Senior Data Engineer

Blockchain • Database • Analytics
Hybrid
Coimbatore, Tamil Nadu, IND
110 Employees

Nexaminds Logo Nexaminds

Senior Data Engineer

Artificial Intelligence • Cloud • Information Technology • Machine Learning • Natural Language Processing • Consulting
In-Office
Coimbatore, Tamil Nadu, IND
80 Employees

Similar Companies Hiring

Revel Thumbnail
Aerospace • Hardware • Robotics • Software
Marina Del Rey, California
70 Employees
Blee Thumbnail
Artificial Intelligence • Marketing Tech • Software
New York, New York
33 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account