Sr Developer - Data Engineering - Chennai

Key Responsibilities: 

• Develop, test, and deploy data processing applications using Apache Spark and Scala. 

• Optimize and tune Spark applications for better performance on large-scale data sets. 

• Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions. 

• Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions. 

• Design and implement high-performance data processing and analytics solutions. 

• Ensure data integrity, accuracy, and security across all processing tasks. 

• Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies. 

• Implement version control and CI/CD pipelines for Spark applications. 

Required Skills & Experience: 

• Minimum 8 years of experience in application development. 

• Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing. 

• Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop. 

• Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi. 

• Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data. 

• Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL. 

• Knowledge of data warehousing concepts, dimensional modeling, and data lakes. 

• Ability to troubleshoot and optimize Spark and Cloudera platform performance. 

• Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).