The Role
Design and architect large-scale distributed big data solutions and ETL pipelines using Java, Scala, and Spark. Build and optimize Spark applications on Cloudera CDH (HDFS, Hive, Impala, HBase, Kafka). Ensure data integrity, troubleshoot performance issues, and implement version control and CI/CD for Spark workloads while collaborating with cross-functional teams.
Summary Generated by Built In
Key Responsibilities:
- Architect and design large-scale, distributed big data solutions using Java and big data technologies to handle high-volume data processing and analytics.
- Optimize and tune Spark applications for better performance on large-scale data sets.
- Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions.
- Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions.
- Design and implement high-performance data processing and analytics solutions.
- Ensure data integrity, accuracy, and security across all processing tasks.
- Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies.
- Implement version control and CI/CD pipelines for Spark applications.
Required Skills & Experience:
- Minimum 15+ years of experience in application development.
- Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing.
- Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop.
- Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi.
- Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data.
- Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL.
- Knowledge of data warehousing concepts, dimensional modeling, and data lakes.
- Ability to troubleshoot and optimize Spark and Cloudera platform performance.
- Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).
Skills Required
- Minimum 15+ years of experience in application development
- Hands-on experience with Apache Spark, Scala, and Spark SQL
- Experience architecting and designing large-scale distributed big data solutions
- Hands-on experience with Cloudera Hadoop (CDH) components: HDFS, Hive, Impala, HBase, Kafka, Sqoop
- Familiarity with Big Data tools: Flume, Oozie, NiFi
- Experience building and optimizing ETL pipelines using Spark for structured and unstructured data
- Experience with SQL and NoSQL databases (Hive, HBase, PostgreSQL)
- Knowledge of data warehousing concepts, dimensional modeling, and data lakes
- Ability to troubleshoot and optimize Spark and Cloudera platform performance
- Familiarity with version control (Git) and CI/CD tools (Jenkins, GitLab)
- Experience using Java for big data application development
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Photon.com has emerged as one of the world’s largest and fastest-growing Digital Agencies. We work with 40% of the Fortune 100 on their Digital initiatives and are known for our ability to integrate Strategy Consulting, Creative Design, and Technology at scale. Please visit www.photon.com to learn more about us, how we work, and our customer case studies. Digital Transformation Starts Here.
