The Role
Develop and maintain data pipelines, ELT processes, workflow orchestration, custom data connectors, and CI/CD pipelines using Airflow, Python, PySpark, Hive/Trino, SQL, and Snowflake. Collaborate with cross-functional teams, monitor and troubleshoot data workflows, validate data quality, document technical processes, and support data governance.
Summary Generated by Built In
Must Needed Skills: Apache Airflow, Python, PySpark ,Hive(Trino), SQL and Snowflake
- Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark ,Hive(Trino) and Snowflake to ensure the efficient and reliable delivery of data.
- Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
- Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
- Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
- Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
- Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
- Stay up-to-date with emerging technologies and best practices in data engineering.
- Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.
Skills Required
- Apache Airflow
- Python
- PySpark
- Hive or Trino
- SQL
- Snowflake
Similar Jobs
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Photon.com has emerged as one of the world’s largest and fastest-growing Digital Agencies. We work with 40% of the Fortune 100 on their Digital initiatives and are known for our ability to integrate Strategy Consulting, Creative Design, and Technology at scale. Please visit www.photon.com to learn more about us, how we work, and our customer case studies. Digital Transformation Starts Here.
