The purpose of this role is to maintain, improve, clean and manipulate data in the business’s operational and analytics databases. The Data Engineer works with the business’s software engineers, data analytics teams, data scientists and data warehouse engineers in order to understand and aid in the implementation of database requirements, analyse performance, and troubleshoot any existent issues.
Job Description:
Responsibilities:
• Design, develop, and maintain scalable data pipelines and systems for data processing.
• Utilize Hadoop and related technologies to manage large-scale data processing.
• Perform data ingestion using Kafka & spark, Sqoop and various file formats and process data into Hive using Beeline/spark.
• Develop and maintain shell scripts for automation of data processing tasks.
• Implement full and incremental data loading strategies to ensure data consistency and availability.
• Orchestrate and monitor workflows using Apache Airflow.
• Ensure code quality and version control using Git.
• Troubleshoot and resolve data-related issues in a timely manner.
• Stay up-to-date with the latest industry trends and technologies to continuously improve our data infrastructure.
Requirements:
• Proven experience as a Data Engineer (ETL, data warehousing).
• Strong knowledge of Hadoop and its ecosystem (HDFS, YARN, MapReduce, Tez and spark).
• Proficiency in Kafka & spark, Sqoop and Hive.
• Experience with shell scripting for automation.
• Expertise in full and incremental data loading techniques.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively in a team environment and communicate effectively with stakeholders.
Good to have:
• Understanding of PySpark and its application in real-time data processing.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Proficiency with Git for version control
• Experience on Postgres SQL or SQL server or MSBI
Location:
Bengaluru
Brand:
Merkle
Time Type:
Full time
Contract Type:
Permanent
Similar Jobs
What We Do
We are dentsu. We team together to help brands predict and plan for disruptive future opportunities and create new paths to growth in the sustainable economy. We know people better than anyone else and we use those insights to connect brand, content, commerce and experience, underpinned by modern creativity. We are the network designed for what’s next
