Sr Data Engineer - Apache DolphinScheduler - Chennai

Reposted 3 Months Ago
Be an Early Applicant
2 Locations
In-Office or Remote
Senior level
Agency • Information Technology
The Role
Design, implement, and optimize Apache DolphinScheduler workflows and ETL pipelines. Integrate with Hadoop, Spark, Flink, Hive, and Kafka. Develop custom plugins, troubleshoot performance and scalability issues, monitor job execution and resource allocation, and maintain CI/CD, version control, and infrastructure automation while collaborating with data engineers and DevOps teams.
Summary Generated by Built In

Key Responsibilities:

  • Develop, configure, and optimize Apache DolphinScheduler workflows for data processing and automation.
  • Design and implement ETL pipelines, job scheduling, and workflow orchestration solutions.
  • Troubleshoot and resolve performance, scalability, and reliability issues in DolphinScheduler.
  • Integrate DolphinScheduler with big data tools such as Hadoop, Spark, Flink, Hive, and Kafka.
  • Work closely with data engineers, DevOps, and software teams to enhance workflow automation.
  • Develop custom plugins and extensions for DolphinScheduler as needed.
  • Monitor and optimize job execution, resource allocation, and workflow efficiency.
  • Maintain best practices for CI/CD, version control, and infrastructure automation.

Required Skills & Qualifications:

  • Strong experience with Apache DolphinScheduler in a production environment.
  • Proficiency in Java, Python, or Scala for workflow scripting and automation.
  • Experience with big data technologies like Hadoop, Spark, Hive, Flink, and Kafka.
  • Understanding of workflow orchestration, DAG execution, and scheduling strategies.
  • Strong problem-solving skills and ability to work in a fast-paced environment.

Skills Required

  • Strong experience with Apache DolphinScheduler in a production environment
  • Proficiency in Java, Python, or Scala for workflow scripting and automation
  • Experience with big data technologies such as Hadoop, Spark, Hive, Flink, and Kafka
  • Understanding of workflow orchestration, DAG execution, and scheduling strategies
  • Maintain best practices for CI/CD, version control, and infrastructure automation
  • Strong problem-solving skills and ability to work in a fast-paced environment
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: London
5,017 Employees
Year Founded: 2007

What We Do

Photon.com has emerged as one of the world’s largest and fastest-growing Digital Agencies. We work with 40% of the Fortune 100 on their Digital initiatives and are known for our ability to integrate Strategy Consulting, Creative Design, and Technology at scale. Please visit www.photon.com to learn more about us, how we work, and our customer case studies. Digital Transformation Starts Here.

Similar Jobs

Remote or Hybrid
2 Locations
289097 Employees

Rubrik Logo Rubrik

Development Manager

Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Cybersecurity • Data Privacy
Remote
India
3000 Employees

Pfizer Logo Pfizer

GSC Quality Management System Lead

Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
In-Office or Remote
9 Locations
121990 Employees
215K-358K Annually

CSC Logo CSC

Accountant

Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Remote or Hybrid
2 Locations
8500 Employees

Similar Companies Hiring

NODA AI Thumbnail
Artificial Intelligence • Information Technology • Software • Cybersecurity • Defense • Defense Technology
Austin, TX
54 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account